* feat: Support OAuth and base GCP credentials for BigQuery destination
Fixes#3380
- Add Union type to allow GcpOAuthCredentials and GcpCredentials
- Maintains backward compatibility with GcpServiceAccountCredentials
- Enables OAuth authentication for Workload Identity Federation
- Add tests for OAuth credentials acceptance
This change allows BigQuery destination to work with OAuth tokens
from GitHub Actions Workload Identity Federation and other OAuth flows,
without breaking existing service account authentication.
* remove baseclass from union
* fix tests
* lazy import google library
* type update
---------
Co-authored-by: djudjuu <djudju@proton.me>
Add support for column comments in Snowflake adapter, following the same
pattern as the Databricks adapter.
Changes:
- Add escape_snowflake_literal() function for proper SQL escaping
- Add COLUMN_COMMENT_HINT constant for Snowflake-specific hints
- Override _get_column_def_sql() to append COMMENT clause
- Support both generic "description" field and Snowflake-specific hint
- Add tests for column comments including special character escaping
Fixes#3312
Co-authored-by: Somasundaram Sekar <somasundaramsekar.1986@gmail.com>
* feat: implement advanced Iceberg partitioning with explicit ordering
- Add support for advanced partition transforms (year, month, day, hour, bucket, truncate)
- Implement explicit partition ordering via index property
- Add custom partition naming support
- Implement priority system: advanced partitioning overrides legacy partition: True
- Add comprehensive validation for partition specifications
- Add graceful error handling for PyIceberg limitations
- Add performance optimization with early exit for non-partitioned schemas
- Update schema typing to support dict/list partition syntax
- Add pyiceberg-core>=0.6.0 dependency for advanced transforms
- Add comprehensive test suite with 22+ test cases covering all scenarios
Backward compatible: existing partition: True syntax continues to work
Resolves partition ordering limitations in Iceberg table format
* Port iceberg_partition and build_iceberg_partition_spec to dlt core
* update type hint in IcebergLoadFilesystemJob
* Add tests for Iceberg advanced partitioning; remove unused partition extraction code
* Add docs for iceberg_adapter
---------
Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com>
* merge resource hints before extract for all backends
* check load package directly
* better type check
* log if unsupported hints
* better log message
* do not use ensure_table_schema_columns
* test for desired behavior
* refactor
* clarified test assertions
* lint
* bumps to version 1.20.0
* update the hub reference docs, add CI check
* use dependency specifier in hub for plugin version check
* minimum dlt runtime cli check
* rollaback to old fsspec min version
* fixes test_hub ci workflow
* fixes flaky test
* bumps hub extra
* updates cli docs linting
* fixes docs lock
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
Co-authored-by: ivasio <ivan@dlthub.com>
* [fix/3358] add pagination stopping to JSONResponseCursorPaginator
* [fix/3358] add some tests when there are more pages
* [fix/3358] fix naming
* [fix/3374] make stop_after_empty_page robust to data = None
* [fix/3358] align has more handling with RangePaginator and add test cases
* Compile path in __init__
short-circuit on empty page before touching has_more
---------
Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com>
* allows load jobs to separately set failed message and exception to be re-raised
* allows to cancel normalize via flag in load package, returns metrics when failed
* corrects cleaning of current load id, re-raises job exceptions in load, passes load and job ids in exception chain
* adds warnings on pending and partially loaded packages in pipeline failed exception
* creates schema when package is created
* makes internal pipeline load storage readonly
* fixes test
* fixes utime on windows
* review code reorg
* adds hub extra
* makes hub module more user friendly when hub not installed
* test and lint fixes
* adds plugin version check util function
* basic cell appearing if installed
* use data quality cell
* show raw data too
* adds dlt-runtime to hub extra, minimal import tests
* bumps to dlthub 0.20.0 alpha
* lists pipelines with cli using the same functions as dashboard, dlt pipeline will list pipelines by default
* adds configured propfiles method on context so only profiles with configs or pipelines are listed
* adds list of locations that contained actual configs to provider interface
* improves workspace and profile commands
* test fixes
* fixes tests
* update text
* adds quality widget as python functions
* adds data_quality as module to hub
* adds hub extra to docs deps
* fixes dashboard imports
* bumps to alpha x.20.0a1
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* adds hub extra
* makes hub module more user friendly when hub not installed
* test and lint fixes
* adds plugin version check util function
* adds dlt-runtime to hub extra, minimal import tests
* bumps to dlthub 0.20.0 alpha
* lists pipelines with cli using the same functions as dashboard, dlt pipeline will list pipelines by default
* adds configured propfiles method on context so only profiles with configs or pipelines are listed
* adds list of locations that contained actual configs to provider interface
* improves workspace and profile commands
* test fixes
* fixes tests
* a tracker that sends pipeline trace, schemas and trace to a bucket is activated when RUN_ID and workspace context are present
* a sync step is executed under the conditions above when workspace dashboard starts
* improves deployment packager (hash computation)
* implement RunContext.reset_config, call it in PluggableRunContext.reload_providers
* fix _config access
* reiinitialize RunContext._runtime_config on access
* adjust the test to .runtime_config being always available
* fixes dlthub tests
---------
Co-authored-by: ivasio <ivan@dlthub.com>
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* extracts adbc parquet load job with file format selector
* ports postgres parquet job to base job
* implements mssql adbc job
* adds pickle test for all destination caps
* adds dbc to adbc group, updates test workflow
* fixes sqlglot from find
* fixes docs
* adds sqlalchemy adbc docs
* adds support from sqllite and mysql in sqlalchemy
* fixes and tests str annotation resolving
* allows to disable adbc and does that in tests
* fixes imports
* docs lock bump
* fixes globalns extraction
* clarifies how adbc drivers are installed, implements fallback for postgres
* improves dashboard multi schema test
* fixes followup jobs
* fixes connection string escaping
* Update docs/website/docs/dlt-ecosystem/destinations/sqlalchemy.md
Co-authored-by: djudjuu <djudju@proton.me>
* removes code dedup
* fixes columns that receive None, simple and nested values
---------
Co-authored-by: djudjuu <djudju@proton.me>
* add support for snowflake clustering key modifications
* add cluster column order test case
* update snowflake cluster hint docs
* switch to reading snowflake cluster hints from table schema
* improves dashboard multi schema test
* closes and waits for sections in multi-schema test
* removes command line snippet with generic text in exceptions
* disables transformers pokeapi test
* child table column removed from parent
* A utility functin that checks whether a column has seen-null-first set
* Improved comments and docstrings, separate method in worker
* null column not inferred if exists as compound
* Column level x-normalizer cleaning moved outside of worker
* Test for empty column becoming compound
* Test clean_seen_null_first_hint
* split home and workspace render methods
* header row dry-er
* catch-all errors in home()-cell
* local try-catch for broken traces
* e2e test for broken trace
* removes this
* shows navigation on pipeline attach error
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* makes trace backward compat with 1.17.0 and earlier
* skips trace if any error in unpickle
* always saves merged pipeline trace to have consistent pipeline.last_trace property
* tests for past traces, broken traces and other improvements
* Initial commit
* Html cleaned
* Summary moved to home section, migration badge added
* Load package status badges improved
* Test getting steps data, migrations count
* Various tests
* Fix in test
* Styles moved, improved ui
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* make arrow_stream default return_type for connectorx backend
* formatting
* bump connectorx version
* return to arrow by default, keep arrow_stream support, add info message
* document arrow_stream cornercases in the docs
* add the test for connectorx arrow_stream return type
* fix formatting
* fix test typo
* fix the tests
* fix package version check, return original version constraint
* adds utils function to losless cast date64 to timestamp[us]
* cast date64 to timestamp for connectorx, update test
---------
Co-authored-by: ivasio <ivan@dlthub.com>
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
Added a dropdown for profile selection in the dashboard interface and updated the layout to display profile and workspace information inline with pipeline selection.
* adds option in load that prevents draining pool on signal
* adds runtime pipeline option to not intercept signals
* refactors signal module
* tests new cases
* describes signal handling in running in prod docs
* bumps dlt to 1.18.0
* fixes tests forked
* removes logging and buffered console output from signals
* adds retry count to load job metrics, generates started_at in init of runnable load job
* allows to update existing metrics in load step
* finalized jobs require start and finish dates
* generates metrics in each job state and in each completed loop, does not complete package if pool drained but jobs left, adds detailed tests for metrics
* fixes remote metrics
* replaces event with package bound semaphore to complete load jobs early
* fixes dashboard to on windows
* improves signals docs
* renames delayed_signals to intercepted_signals
