* move dashboard tests to own workflow
* * do not crash dashboard app if credentials not available
* do not sort columns in dataset browser
* try sleep in e2e tests
* disable python 3.14 tests for now
* disable mac e2e tests for dashboard
clean up step conditions
* basic runner docs
* fix extension
* removed unintended file
* extra section on current.runner(), linted snippets
* fix typo
* fix example
* refactor: dlt_plus.PipelineRunner
* dlt_plus.runner()
* proper keywords and an extra note
* better text
* change requests
* fix link
* fix snippets
* fix title
* updates, removed runner page
* changed order of clean folder and trace-saving sections
* full config in python api example
* forgotten comma
* better project runner vs pipeline runner description
* typo
* more tabs
* missing link
* improved docs
* another update
* observability docs
* mention custom callbacks in runner
* broken link fix
* i give up on this link
* change requests
* another final attempt at this broken link
* remove uneeded file
* fix forwarding of pipelines dir to marimo app
* disable state sync and display all schemas and remote state and schemas in pipeline overview
* add support for multiple schemas
* fix e2e tests, further updates pending
* use dropdown instead of multiselect for schema selection
add multi schema pipeline to fixtures
* add last run info in pipeline overview
add buttons to open pipeline folder and local data folder if present
* fix loads browser to select correct schema
* allow to start dashboard for a pipeline that is not there yet and add helpful error message in this case
* nicer last run time formatting
show pipeline error screen also when manually chnaing the pipeline name in the url
* move buttons to top, add refresh buttons to sections
* use raw query when constructing queries
* lazy load remote state tab
* fix traces and trace typing (mostly)
* add exception traces to ui
* add file watcher
* remove test code
* add source and resource state viewer to data panel
* update existing unit tests
* add unit test for new utils
* make marimo dashboard the default app for pipeline show
* update docs
* update existing e2e tests for new yaml based rendering of state
* move streamlit app down in sidebar
* grammar fixes for dashboard strings
* open duckdb in readme mode in datapanel in dashboard
* remove old tests
re-enable dashboard main command
* add missing args to dashboard command
* small fixes to e2e tests
* add tests for exceptions
* re-organize e2e tests into invidual tests
* add basic schema selection checks
* improve dashboard help and dashboard docs page
* short some strings in testing to make selecting predictable
* merge devel
* typo
---------
Co-authored-by: djudjuu <djudju@proton.me>
* Update playground.py
Replace REST demo with DummyJSON users (CORS-safe for Playground)
* update with actual use case
* feat: improve playground example with real API data
- Switch from dummyjson to JSONPlaceholder users API
- Add dev_mode=True for better development experience
- Fix function parameter (remove unused dlt parameter)
- Update table references from 'items' to 'users' consistently
- Fix test assertion to expect 10 users instead of 50
- Add write_disposition='replace' to prevent data duplication
- Improve data display with proper return statements
* feat(playground): make pipeline idempotent with refresh=True
* fix: remove problematic sqlite3 micropip install causing linting failure
* lint: ruff fix & format playground notebook
* Playground: updated assert
* feat: add improved Marimo with better optimization and formatting
* code edit : removed argument refresh to avoid users confusion
* Add pkey, disabled_algorithms, transport_factory and auth_strategy parameters to paramiko.client.connect. Also update filesystem docs for SFTP creds
* Move paramiko imports after the pytest.skip
---------
Co-authored-by: Ayush Patel <Ayush.Patel@imc.com>
* remove transformation code and tests that now live in dlt_plus
* move lineage code and tests into dataset folder scope
* start fixing model item format tests
* revert model item format tests back to version before last big change (with some updates)
* disable transformations snippets linting and testing for now
* remove uneeded test
* bumps to version 1.15.0
* handled duckdb 1.3.2 in iceberg scanner and bumps dev version - seems to work with adlfs
* binds old dev duckb on windows until segfault is fixed
* test fixes, docs update
* replace arrow2 with arrow backend for connectorx
* updated docs/
* updated minimal deps
* update docs and pyproject.toml deps
* updated minimal deps to support 3.9
* converts +00:00 to UTC right after handover from connectorx
* fixes examples connectorx lint
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* fixes sqlalchemy destination to work with mssql
* do not generate ; in merge jobs
* fixes engine version type
* demonstrates plugging TypeMapper into sqlalchemy destination
* excludes temp files from snippet linting
* adds precision to _dlt_load_id and _dlt_id columns
* adds json field support for mssql
* fallback for alembic migrations when dialect not supported ie trino
* normalizes use of ; to separate queries
* adds type mappers for mysql, mssql and trino
* fixes type mapper import
* updates destination caps from explicit destination params at the end to overwrite adjustments
* normalizes ; usage, forward trackebacks when handling database exception
* fixes sqlalchemy merge eq condition and tests
* fixes clickhouse temporary table engine
* synth unpickle synthesizes on any error
* fixes duckdb with table scanners accessing self.execute... in open_connection
* fixes synpase json column fallback for index
* moves adding _dlt_load_id to arrow table after it is merged and normalized
* fixes more tests
* moves _dlt_load_id add in arrow extractor after normalization, before table merrging
* tests run context plug passthrough
* fixes BIGQUERY numeric creation
* fixes databricks PRIMARY KEY injection in tests
* dbml WIP
* dbml exporter
* full reference support; Schema.to_dbml()
* revert uv.lock changes
* fixed condition for _dlt tables ref
* rename _dbml.py to private module; use json encoder
* use TStoredSchema as entrypoint
* implementation completed
* added documentation
* added CLI support
* support unknown data type
* please the linter gods
* minified the image
* enables dbml for schema export
* image link to bucket; renamed constant
* updated docs linting
* include recommended VSCode extension
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* fix: prevent DuplicateSchema error when using public schema in Redshift
- Override has_dataset() method in RedshiftSqlClient to return True for 'public' schema
- This prevents dlt from trying to CREATE SCHEMA public when it already exists
- Add test to verify the fix works correctly
- Fixes issue where pipeline fails with 'Schema "public" already exists' error
Closes#2770
* Fix flake8 error
* Fix flake8 error
* Fix black errors
* adjust to new flow
* adjust to new flow
* adjust to new flow
* fixes links in docs
* Update instructions
* improves restapi-cursor workflow and leaves several todos
* Minor cleaning
* Rename the page, add redirect
* Reverse redirect
---------
Co-authored-by: Adrian <Adrian>
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com>
* sets request and response validation in aws credentials
* removes init files from dlt tables, makes folders only when schema is updated
* fixes tests
* fixes table data exists/not exists check in filesystem
* unifies when data tables and dlt tables in filesystem are considered empty vs not existing
* fixes synth unpickler to handle enums
