* feat: Support OAuth and base GCP credentials for BigQuery destination
Fixes#3380
- Add Union type to allow GcpOAuthCredentials and GcpCredentials
- Maintains backward compatibility with GcpServiceAccountCredentials
- Enables OAuth authentication for Workload Identity Federation
- Add tests for OAuth credentials acceptance
This change allows BigQuery destination to work with OAuth tokens
from GitHub Actions Workload Identity Federation and other OAuth flows,
without breaking existing service account authentication.
* remove baseclass from union
* fix tests
* lazy import google library
* type update
---------
Co-authored-by: djudjuu <djudju@proton.me>
Add support for column comments in Snowflake adapter, following the same
pattern as the Databricks adapter.
Changes:
- Add escape_snowflake_literal() function for proper SQL escaping
- Add COLUMN_COMMENT_HINT constant for Snowflake-specific hints
- Override _get_column_def_sql() to append COMMENT clause
- Support both generic "description" field and Snowflake-specific hint
- Add tests for column comments including special character escaping
Fixes#3312
Co-authored-by: Somasundaram Sekar <somasundaramsekar.1986@gmail.com>
* feat: implement advanced Iceberg partitioning with explicit ordering
- Add support for advanced partition transforms (year, month, day, hour, bucket, truncate)
- Implement explicit partition ordering via index property
- Add custom partition naming support
- Implement priority system: advanced partitioning overrides legacy partition: True
- Add comprehensive validation for partition specifications
- Add graceful error handling for PyIceberg limitations
- Add performance optimization with early exit for non-partitioned schemas
- Update schema typing to support dict/list partition syntax
- Add pyiceberg-core>=0.6.0 dependency for advanced transforms
- Add comprehensive test suite with 22+ test cases covering all scenarios
Backward compatible: existing partition: True syntax continues to work
Resolves partition ordering limitations in Iceberg table format
* Port iceberg_partition and build_iceberg_partition_spec to dlt core
* update type hint in IcebergLoadFilesystemJob
* Add tests for Iceberg advanced partitioning; remove unused partition extraction code
* Add docs for iceberg_adapter
---------
Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com>
* merge resource hints before extract for all backends
* check load package directly
* better type check
* log if unsupported hints
* better log message
* do not use ensure_table_schema_columns
* test for desired behavior
* refactor
* clarified test assertions
* lint
* bumps to version 1.20.0
* use dependency specifier in hub for plugin version check
* minimum dlt runtime cli check
* rollaback to old fsspec min version
* fixes test_hub ci workflow
* fixes flaky test
* bumps to version 1.20.0
* update the hub reference docs, add CI check
* use dependency specifier in hub for plugin version check
* minimum dlt runtime cli check
* rollaback to old fsspec min version
* fixes test_hub ci workflow
* fixes flaky test
* bumps hub extra
* updates cli docs linting
* fixes docs lock
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
Co-authored-by: ivasio <ivan@dlthub.com>
* working copy of docs
* added diagram; wip
* checkpoint
* Misc docusaurus fixes
* Remove placeholder text and whitespace
* Move images to the gcp bucket
* add data quality section
* fixed linting
* Escape curly braces
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com>
* [fix/3358] add pagination stopping to JSONResponseCursorPaginator
* [fix/3358] add some tests when there are more pages
* [fix/3358] fix naming
* [fix/3374] make stop_after_empty_page robust to data = None
* [fix/3358] align has more handling with RangePaginator and add test cases
* Compile path in __init__
short-circuit on empty page before touching has_more
---------
Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com>
* allows load jobs to separately set failed message and exception to be re-raised
* allows to cancel normalize via flag in load package, returns metrics when failed
* corrects cleaning of current load id, re-raises job exceptions in load, passes load and job ids in exception chain
* adds warnings on pending and partially loaded packages in pipeline failed exception
* creates schema when package is created
* makes internal pipeline load storage readonly
* fixes test
* fixes utime on windows
* review code reorg
* adds hub extra
* makes hub module more user friendly when hub not installed
* test and lint fixes
* adds plugin version check util function
* basic cell appearing if installed
* use data quality cell
* show raw data too
* adds dlt-runtime to hub extra, minimal import tests
* bumps to dlthub 0.20.0 alpha
* lists pipelines with cli using the same functions as dashboard, dlt pipeline will list pipelines by default
* adds configured propfiles method on context so only profiles with configs or pipelines are listed
* adds list of locations that contained actual configs to provider interface
* improves workspace and profile commands
* test fixes
* fixes tests
* update text
* adds quality widget as python functions
* adds data_quality as module to hub
* adds hub extra to docs deps
* fixes dashboard imports
* bumps to alpha x.20.0a1
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* adds hub extra
* makes hub module more user friendly when hub not installed
* test and lint fixes
* adds plugin version check util function
* adds dlt-runtime to hub extra, minimal import tests
* bumps to dlthub 0.20.0 alpha
* lists pipelines with cli using the same functions as dashboard, dlt pipeline will list pipelines by default
* adds configured propfiles method on context so only profiles with configs or pipelines are listed
* adds list of locations that contained actual configs to provider interface
* improves workspace and profile commands
* test fixes
* fixes tests
* a tracker that sends pipeline trace, schemas and trace to a bucket is activated when RUN_ID and workspace context are present
* a sync step is executed under the conditions above when workspace dashboard starts
* improves deployment packager (hash computation)
* implement RunContext.reset_config, call it in PluggableRunContext.reload_providers
* fix _config access
* reiinitialize RunContext._runtime_config on access
* adjust the test to .runtime_config being always available
* fixes dlthub tests
---------
Co-authored-by: ivasio <ivan@dlthub.com>
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* fixes historic builds
* fix broken link
* constrain docs build env to python 3.10
* switch snippets testing to python 3.10
* allows python up to py3.12 in docs project
---------
Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
* extracts adbc parquet load job with file format selector
* ports postgres parquet job to base job
* implements mssql adbc job
* adds pickle test for all destination caps
* adds dbc to adbc group, updates test workflow
* fixes sqlglot from find
* fixes docs
* adds sqlalchemy adbc docs
* adds support from sqllite and mysql in sqlalchemy
* fixes and tests str annotation resolving
* allows to disable adbc and does that in tests
* fixes imports
* docs lock bump
* fixes globalns extraction
* clarifies how adbc drivers are installed, implements fallback for postgres
* improves dashboard multi schema test
* fixes followup jobs
* fixes connection string escaping
* Update docs/website/docs/dlt-ecosystem/destinations/sqlalchemy.md
Co-authored-by: djudjuu <djudju@proton.me>
* removes code dedup
* fixes columns that receive None, simple and nested values
---------
Co-authored-by: djudjuu <djudju@proton.me>
* add support for snowflake clustering key modifications
* add cluster column order test case
* update snowflake cluster hint docs
* switch to reading snowflake cluster hints from table schema
* improves dashboard multi schema test
* closes and waits for sections in multi-schema test
* removes command line snippet with generic text in exceptions
* disables transformers pokeapi test
