mirror of
https://github.com/dlt-hub/dlt.git
synced 2025-12-17 19:31:30 +00:00
51516b11a1
* copies rest_api source code and test suite, adjusts imports * integrates rest_client/conftest.pi into rest_api/conftest.py. Fixes incompatibilities except for POST request (/search/posts) * integrates POST search test * do no longer skip test with typed dict config * reuses tests/sources/helpers/rest_client/conftest.py in tests/sources/rest_api * checks off TODO * formats rest_api code according to dlt-core rules * fixes typing errors and graphlib import error * moves latest changes from rest_api into core (687e7ddab3a95fa621584741af543e561147ebe3). Formats and lints entire rest API starts to reorganize test suite * modularizes rest_api test suite * formats code and imports * updates signature of Paginator.update_state() * moves source test suite after duckdb is installed * end-to-end test rest_api_source on all destinations. Removes redundant helpers from test/utils.py * adds example rest_api_pipeline.py, corrects sample rest_api_pipeline docs on secrets * loads latest 30 days of issues instead of fixed date * refactors types * tests example rest_api pipelines, adds filesystem configs to load tests * fix inheritance of incremental args, make typed_dict detection work with typing extensions dicts * type incremental cursor_path as str * refactors intersection of TResourceHints and ResourceBase into TResourceHintsBase * uses str instead of generic TCursorValue * configures github access token for CI * copies sql source and tests * adjusts import paths * workaround for UUID type missing in sqlalchemy < 2.0 * extracts load tests to tests/load. Adds necessary test utility functions * formats code * corrects example postgres credentials for the test suite * formats imports, removes duplicate definition * conditionally skips test for range type detection * fixes side effects of tests modifying os.environ. * fixes lint errors * moves tests to right places, runs on all destinations where applicable moves filesystem source with tests and examples rearranges old sources.filesystem adds copy sig for transformers fixes Windows tests moves source test suite after duckdb is installed Revert "attempt to make duckdb a minimal dependency by removing it from extras" This reverts commit6b7e6705ad. attempt to make duckdb a minimal dependency by removing it from extras formats code updates signature of Paginator.update_state() formats imports modularizes rest_api test suite adds new files from 687e7ddab3a95fa621584741af543e561147ebe3, starts to reorganize test suite moves latest changes from rest_api into core (687e7ddab3a95fa621584741af543e561147ebe3). Formats and lints entire rest API fixes last type errors fixes more type errors and formats code fixes graphlib import error fixes more type errors fixes type errors except for test_configurations.py fixes typing errors where optional field was required formats rest_api code according to dlt-core rules checks off TODO reuses tests/sources/helpers/rest_client/conftest.py in tests/sources/rest_api do no longer skip test with typed dict config integrates POST search test integrates rest_client/conftest.pi into rest_api/conftest.py. Fixes incompatibilities except for POST request (/search/posts) copies rest_api source code and test suite, adjusts imports * post rebase fixes and formatting * first simple version of init command that can use core sources * update tests for core sources * improve tests a bit more * move init / generic source to core * detect explicit repo url in init command * update output and clean up structure in init command a bit * fix tests * add option for omitting core sources and reverting to the old behavior * add core sources to the dlt init -l list * add init template files to build * remove one unneded file * revert common tests file * move sources tests to dedicated file * remove destination tests for now, revert later * upgrade sqlalchemy for local source tests * create sql_database extra * fix bug in transform * set up timezone fixtures properly, still does not work right * fallback to timezone on duckdb with timestamp * separate common from load tests properly * update duckdb timezone test * add sql_alchemy dependency to last part of common tests * updates imports * add sql_database_pipeline file, update dlt init commands, add basic tests for sql_database_pipeline * only import sqlalchemy in tests if present * fix linter errors * bump connectorx for python 3.12 support * move sql_alchemy shims to shims file and use the original file for the same dependency system as with other libs * Fix linter errors (reverts back to wilis version from a few commits ago) * exclude connectorx from python 3.8 * make rest api example pipeline also work without a token * remove secrets from local sources tests * change test setup to work with both sqlalchemy versions * adds secrets to a part of common tests * make sql database pipeline tests succeed on both sqlalchemy versions * add excel dependenices to common tests * fix bug in schema inference of sql_alchemy backed sources * fix tests running for sql alchemy 1.4 * add concept of single file templates in the core * update tests and fix some * add some example pipelines * fixes some issues * sort source names * fix unsupported columns * fix all sql database tests for sqlalchemy 2.0 * fix some tests for sqlalchemy 1.4 * deselect connectorx incremental tests on sqlalchemy 1.4 * fixes some more tests * some cleanup * fix bug in init script * Revert "remove destination tests for now, revert later" This reverts commit47e1933975. * exclude sources load tests from destination workflows * fix openpyxl install * disable requests tests for now * fix commen tests * add dataframe example pipeline clean up other examples a bit * add intro examples * update cleaning scripts for athena and redshift * make timezone tests slightly more strict * reorders sql_database import to get user friendly dependency error --------- Co-authored-by: dave <shrps@posteo.net> Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>