Compare commits

..

41 Commits

Author SHA1 Message Date
Michelle Ark
60f87411d5 Merge branch 'feature/decouple-adapters-from-core' into macro-resolver-remove-lazy-loading 2023-12-07 08:27:45 +09:00
Michelle Ark
eb96e3deec Add RelationConfig Protocol for use in Relation.create_from (#9210)
* move relation contract to dbt.adapters

* changelog entry

* first pass: clean up relation.create_from

* type ignores

* type ignore

* changelog entry

* update RelationConfig variable names
2023-12-06 10:46:45 -08:00
Michelle Ark
f68af070f3 remove manifest from adapter.execute_macro, replace with MacroResolver + remove lazy loading 2023-12-06 23:27:42 +09:00
Michelle Ark
7ad1accf2b Merge branch 'feature/decouple-adapters-from-core' into relation-create-from-refactor 2023-12-06 11:19:16 +09:00
Michelle Ark
ed8f5d38e4 Remove ResultNode usage from connections (#9211) 2023-12-06 11:04:26 +09:00
Michelle Ark
7ad6aa18da update RelationConfig variable names 2023-12-06 11:02:39 +09:00
Michelle Ark
6796edd66e Merge branch 'feature/decouple-adapters-from-core' into relation-create-from-refactor 2023-12-06 10:58:41 +09:00
Michelle Ark
e01eb30884 Remove usage of dbt.contracts.relation in dbt/adapters (#9207) 2023-12-06 10:34:56 +09:00
Michelle Ark
ba53f053fd changelog entry 2023-12-05 17:07:32 +09:00
Michelle Ark
b8de881ed3 type ignore 2023-12-05 17:06:49 +09:00
Michelle Ark
d7d5e2335c type ignores 2023-12-05 15:41:13 +09:00
Michelle Ark
160d0db238 first pass: clean up relation.create_from 2023-12-05 15:03:16 +09:00
Michelle Ark
2cee8652a6 changelog entry 2023-12-05 12:06:06 +09:00
Michelle Ark
7f777f8a42 move relation contract to dbt.adapters 2023-12-05 12:03:16 +09:00
Michelle Ark
00f49206e9 Decouple macro generator from adapters (#9149) 2023-12-05 10:51:51 +09:00
Michelle Ark
1bca662883 first pass: adapter migration script (#9160) 2023-12-05 08:41:11 +09:00
colin-rogers-dbt
41ac915949 Refactor EventManager setup and interaction (#9180)
* moving types_pb2.py to common/events

* move event manager setup back to core, remove ref to global EVENT_MANAGER and clean up event manager functions

* move invocation_id from events to first class common concept

* move lowercase utils to common

* move lowercase utils to common

* ref CAPTURE_STREAM through method

* add changie
2023-12-01 08:48:19 -08:00
Mila Page
373125ecb8 Move the semver package to common and alter references. (#9166)
* Move the semver package to common and alter references.

* Alter leftover references to dbt.semver, this time using from syntax.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-11-29 11:03:13 -08:00
Michelle Ark
294ad82e50 delete accidentally merged types_pb2.py 2023-11-28 20:20:59 -05:00
Michelle Ark
12bd1e87fb Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-28 20:08:33 -05:00
colin-rogers-dbt
8bad75c65b Move adapter logger to adapters (#9165)
* moving types_pb2.py to common/events

* Move AdapterLogger to adapter folder

* add changie
2023-11-28 15:05:30 -08:00
Michelle Ark
220f56d8d2 remove adapter.get_compiler (#9134) 2023-11-28 09:27:00 -05:00
Michelle Ark
615ad1fe2d move include/global_project to adapters (#8930) 2023-11-27 18:14:02 -05:00
Peter Webb
2ab0f7b26b Decouple adapter constraints from core (#9054)
* Move constraints to dbt.common

* Move constraints to contracts folder, per review

* Add a changelog entry.
2023-11-21 12:16:39 -05:00
Michelle Ark
e56a5dae8b Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific (#9051) 2023-11-20 17:26:11 -05:00
Michelle Ark
1d0a3e92c8 Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-16 11:28:51 -05:00
colin-rogers-dbt
51b94b26cc Refactor Base Exceptions (#8989)
* moving types_pb2.py to common/events

* Refactor Base Exceptions

* update make_log_dir_if_missing to handle str

* move remaining adapters exception imports to common/adapters
---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2023-11-10 09:54:36 -05:00
Michelle Ark
4ee950427a Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-07 11:38:24 -05:00
Michelle Ark
c4ff280436 remove dbt.flags.LOG_CACHE_EVENTS usage in dbt/adapters (#8933) 2023-11-01 18:44:59 -04:00
Michelle Ark
1260782bd2 remove dbt.flags.MP_CONTEXT usage in dbt/adapters (#8931) 2023-11-01 16:13:39 -04:00
Michelle Ark
333120b111 Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-01 12:00:17 -04:00
Michelle Ark
af916666a2 move agate_helper into common (#8911)
* move agate_helper into common

* add changie

---------

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
2023-10-26 16:22:04 -04:00
Michelle Ark
7de8930d1d move agate_helper unit tests under tests/unit/common 2023-10-26 16:19:13 -04:00
Michelle Ark
200bcdcd9f Merge branch 'move-agate-client-to-common' into feature/decouple-adapters-from-core 2023-10-26 16:18:30 -04:00
Michelle Ark
b9a603e3aa remove usage of dbt.config.PartialProject from dbt/adapters (#8909)
* remove usage of dbt.config.PartialProject from dbt/adapters

* add changie

---------

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
2023-10-26 16:04:39 -04:00
colin-rogers-dbt
1a825484fb Add utils module (#8910)
* moving types_pb2.py to common/events

* split out utils into core/common/adapters

* add changie
2023-10-26 15:59:07 -04:00
Michelle Ark
f44d704801 move agate_helper into common 2023-10-25 17:05:08 -04:00
Michelle Ark
dbd02e54c2 move types_pb2.py from events to common/events 2023-10-25 15:46:41 -04:00
Michelle Ark
a89642a6f9 fix import 2023-10-25 15:04:17 -04:00
colin-rogers-dbt
c141148616 Move events to common (#8676)
* Move events to common

* More Type Annotations (#8536)

* Extend use of type annotations in the events module.

* Add return type of None to more __init__ definitions.

* Still more type annotations adding -> None to __init__

* Tweak per review

* Allow adapters to include python package logging in dbt logs (#8643)

* add set_package_log_level functionality

* set package handler

* set package handler

* add logging about stting up logging

* test event log handler

* add event log handler

* add event log level

* rename package and add unit tests

* revert logfile config change

* cleanup and add code comments

* add changie

* swap function for dict

* add additional unit tests

* fix unit test

* update README and protos

* fix formatting

* update precommit

---------

Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
2023-10-25 14:57:02 -04:00
Michelle Ark
469a9aca06 remove dbt.contracts.connection imports from adapter module 2023-10-25 14:56:55 -04:00
1177 changed files with 43946 additions and 71819 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.10.0a1
current_version = 1.8.0a1
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
@@ -35,3 +35,13 @@ first_value = 1
[bumpversion:file:core/setup.py]
[bumpversion:file:core/dbt/version.py]
[bumpversion:file:plugins/postgres/setup.py]
[bumpversion:file:plugins/postgres/dbt/adapters/postgres/__version__.py]
[bumpversion:file:docker/Dockerfile]
[bumpversion:file:tests/adapter/setup.py]
[bumpversion:file:tests/adapter/dbt/tests/adapter/__version__.py]

View File

@@ -1,6 +1,6 @@
# dbt Core Changelog
- This file provides a full account of all changes to `dbt-core`
- This file provides a full account of all changes to `dbt-core` and `dbt-postgres`
- Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Remove adapter.get_compiler interface
time: 2023-11-27T11:47:57.443202-05:00
custom:
Author: michelleark
Issue: "9148"

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Move AdapterLogger to adapters folder
time: 2023-11-28T13:43:56.853925-08:00
custom:
Author: colin-rogers-dbt
Issue: "9151"

View File

@@ -0,0 +1,7 @@
kind: Breaking Changes
body: move event manager setup back to core, remove ref to global EVENT_MANAGER and
clean up event manager functions
time: 2023-11-30T13:53:48.645192-08:00
custom:
Author: colin-rogers-dbt
Issue: "9150"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Begin using DSI 0.4.x
time: 2023-10-31T13:19:54.750009-07:00
custom:
Author: QMalcolm peterallenwebb
PR: "8892"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Update typing-extensions version to >=4.4
time: 2023-11-06T13:00:51.062386-08:00
custom:
Author: tlento
PR: "9012"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Upgrading dbt-semantic-interfaces to 0.8.3 for custom grain support in offset windows
time: 2024-11-12T16:38:15.351519-05:00
custom:
Author: WilliamDee
Issue: None

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Bump codecov/codecov-action from 4 to 5"
time: 2024-11-18T00:11:13.00000Z
custom:
Author: dependabot[bot]
Issue: 11009

View File

@@ -0,0 +1,6 @@
kind: Docs
body: fix get_custom_database docstring
time: 2023-11-06T12:31:57.525711Z
custom:
Author: LeoTheGriff
Issue: "9003"

View File

@@ -0,0 +1,6 @@
kind: Features
body: 'Allow adapters to include package logs in dbt standard logging '
time: 2023-09-15T12:37:33.862862-07:00
custom:
Author: colin-rogers-dbt
Issue: "7859"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add drop_schema_named macro
time: 2023-10-17T14:36:20.612289-07:00
custom:
Author: colin-rogers-dbt
Issue: "8025"

View File

@@ -0,0 +1,6 @@
kind: Features
body: migrate utils to common and adapters folders
time: 2023-10-26T11:08:21.458709-07:00
custom:
Author: colin-rogers-dbt
Issue: "8924"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Move Agate helper client into common
time: 2023-10-26T12:35:56.538587-07:00
custom:
Author: MichelleArk
Issue: "8926"

View File

@@ -0,0 +1,6 @@
kind: Features
body: remove usage of dbt.config.PartialProject from dbt/adapters
time: 2023-10-26T12:39:13.904116-07:00
custom:
Author: MichelleArk
Issue: "8928"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add exports to SavedQuery spec
time: 2023-10-31T13:20:22.448158-07:00
custom:
Author: QMalcolm peterallenwebb
Issue: "8892"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Remove legacy logger
time: 2023-11-07T13:56:35.186648-08:00
custom:
Author: colin-rogers-dbt
Issue: "8027"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support setting export configs hierarchically via saved query and project configs
time: 2023-11-10T15:42:55.042317-08:00
custom:
Author: QMalcolm
Issue: "8956"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add new hard_deletes="new_record" mode for snapshots.
time: 2024-11-04T12:00:53.95191-05:00
custom:
Author: peterallenwebb
Issue: "10235"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add `batch` context object to model jinja context
time: 2024-11-21T12:56:30.715473-06:00
custom:
Author: QMalcolm
Issue: "11025"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Ensure pre/post hooks only run on first/last batch respectively for microbatch
model batches
time: 2024-12-06T19:53:08.928793-06:00
custom:
Author: MichelleArk QMalcolm
Issue: 11094 11104

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support "tags" in Saved Queries
time: 2024-12-16T09:54:35.327675-08:00
custom:
Author: theyostalservice
Issue: "11155"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Calculate source freshness via a SQL query
time: 2024-12-17T17:16:31.841076-08:00
custom:
Author: ChenyuLInx
Issue: "8797"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add freshness definition on model for adaptive job
time: 2024-12-18T17:07:29.55754-08:00
custom:
Author: ChenyuLInx
Issue: "11123"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Meta config for dimensions measures and entities
time: 2025-01-06T13:28:29.176439-06:00
custom:
Author: DevonFulcher
Issue: None

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add doc_blocks to manifest for nodes and columns
time: 2025-01-22T17:03:28.866522Z
custom:
Author: aranke
Issue: 11000 11001

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: For packages installed with tarball method, fetch metadata to resolve nested dependencies
time: 2023-10-13T13:09:43.188308-04:00
custom:
Author: adamlopez
Issue: "8621"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix partial parsing not working for semantic model change
time: 2023-10-16T16:39:53.05058-07:00
custom:
Author: ChenyuLInx
Issue: "8859"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Handle unknown `type_code` for model contracts
time: 2023-10-24T11:01:51.980781-06:00
custom:
Author: dbeatty10
Issue: 8877 8353

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add back contract enforcement for temporary tables on postgres
time: 2023-10-24T14:55:04.051683-05:00
custom:
Author: emmyoop
Issue: "8857"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Rework get_catalog implementation to retain previous adapter interface semantics
time: 2023-10-24T15:54:00.628086-04:00
custom:
Author: peterallenwebb
Issue: "8846"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add version to fqn when version==0
time: 2023-10-26T00:25:36.259356-05:00
custom:
Author: aranke
Issue: "8836"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix cased comparison in catalog-retrieval function.
time: 2023-10-30T09:37:34.258612-04:00
custom:
Author: peterallenwebb
Issue: "8939"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Catalog queries now assign the correct type to materialized views
time: 2023-10-31T00:53:45.486203-04:00
custom:
Author: mikealfare
Issue: "8864"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix compilation exception running empty seed file and support new Integer agate data_type
time: 2023-10-31T14:48:37.774871-04:00
custom:
Author: gshank
Issue: "8895"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Make relation filtering None-tolerant for maximal flexibility across adapters.
time: 2023-11-01T15:58:24.552054-04:00
custom:
Author: peterallenwebb
Issue: "8974"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Update run_results.json from previous versions of dbt to support deferral and
rerun from failure
time: 2023-11-06T15:59:33.677915-05:00
custom:
Author: jtcohen6 peterallenwebb
Issue: "9010"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix git repository with subdirectory for Deps
time: 2023-11-07T09:23:58.214271-08:00
custom:
Author: ChenyuLInx
Issue: "9000"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Use MANIFEST.in to recursively include all jinja templates; fixes issue where
some templates were not included in the distribution
time: 2023-11-07T09:41:30.121733-05:00
custom:
Author: mikealfare
Issue: "9016"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix formatting of tarball information in packages-lock.yml
time: 2023-11-13T11:49:56.437007-08:00
custom:
Author: ChenyuLInx QMalcolm
Issue: "9062"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'deps: Lock git packages to commit SHA during resolution'
time: 2023-11-27T15:43:10.122069+01:00
custom:
Author: jtcohen6
Issue: "9050"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'deps: Use PackageRenderer to read package-lock.json'
time: 2023-11-27T15:43:47.842423+01:00
custom:
Author: jtcohen6
Issue: "9127"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Get sources working again in dbt docs generate'
time: 2023-11-28T15:52:25.738256Z
custom:
Author: aranke
Issue: "9119"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: dbt retry does not respect --threads
time: 2024-08-22T12:21:32.358066+05:30
custom:
Author: donjin-master
Issue: "10584"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: update adapter version messages
time: 2024-10-25T10:43:39.274723-05:00
custom:
Author: dave-connors-3
Issue: "10230"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Catch DbtRuntimeError for hooks
time: 2024-11-21T18:17:39.753235Z
custom:
Author: aranke
Issue: "11012"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Access DBUG flag more consistently with the rest of the codebase in ManifestLoader
time: 2024-11-28T16:29:36.236729+01:00
custom:
Author: Threynaud
Issue: "11068"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Improve the performance characteristics of add_test_edges()
time: 2024-12-04T10:04:29.096231-05:00
custom:
Author: peterallenwebb
Issue: "10950"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Implement partial parsing for singular data test configs in yaml files
time: 2024-12-05T14:53:07.295536-05:00
custom:
Author: gshank
Issue: "10801"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix debug log messages for microbatch batch execution information
time: 2024-12-09T11:38:06.972743-06:00
custom:
Author: MichelleArk QMalcolm
Issue: "11111"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix running of extra "last" batch when there is only one batch
time: 2024-12-09T13:33:17.253326-06:00
custom:
Author: QMalcolm
Issue: "11112"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix interpretation of `PartialSuccess` to result in non-zero exit code
time: 2024-12-09T15:07:11.391313-06:00
custom:
Author: QMalcolm
Issue: "11114"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Warn about invalid usages of `concurrent_batches` config
time: 2024-12-12T11:36:11.451962-06:00
custom:
Author: QMalcolm
Issue: "11122"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Error writing generic test at run time
time: 2024-12-16T13:46:45.936573-05:00
custom:
Author: gshank
Issue: "11110"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Run check_modified_contract for state:modified
time: 2024-12-17T15:48:48.053054-05:00
custom:
Author: gshank
Issue: "11034"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix unrendered_config for tests from dbt_project.yml
time: 2024-12-18T11:26:40.270022-05:00
custom:
Author: gshank
Issue: "11146"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Make partial parsing reparse referencing nodes of newly versioned models.
time: 2025-01-02T14:05:43.629959-05:00
custom:
Author: d-cole
Issue: "8872"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Ensure warning about microbatch lacking filter inputs is always fired
time: 2025-01-07T17:37:19.373261-06:00
custom:
Author: QMalcolm
Issue: "11159"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix microbatch dbt list --output json
time: 2025-01-09T12:33:09.958795+01:00
custom:
Author: internetcoffeephone
Issue: 10556 11098

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix for custom fields in generic test config for not_null and unique tests
time: 2025-01-10T15:58:24.479245-05:00
custom:
Author: gshank
Issue: "11208"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Loosen validation on freshness to accomodate previously wrong but harmless config.
time: 2025-01-28T13:55:09.318833-08:00
custom:
Author: ChenyuLInx peterallenwebb
Issue: "11123"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Added more type annotations.
time: 2023-08-31T16:44:35.737954-04:00
custom:
Author: peterallenwebb
Issue: "8537"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.include.global_project in dbt/adapters
time: 2023-10-26T18:49:53.36449-04:00
custom:
Author: michelleark
Issue: "8925"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add a no-op runner for Saved Qeury
time: 2023-10-27T14:00:48.4755-07:00
custom:
Author: ChenyuLInx
Issue: "8893"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: remove dbt.flags.MP_CONTEXT usage in dbt/adapters
time: 2023-11-01T10:27:58.790153-04:00
custom:
Author: michelleark
Issue: "8967"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: 'Remove usage of dbt.flags.LOG_CACHE_EVENTS in dbt/adapters'
time: 2023-11-01T17:31:24.974093-04:00
custom:
Author: michelleark
Issue: "8969"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move CatalogRelationTypes test case to the shared test suite to be reused by
adapter maintainers
time: 2023-11-03T19:52:22.694394-04:00
custom:
Author: mikealfare
Issue: "8952"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Treat SystemExit as an interrupt if raised during node execution.
time: 2023-11-06T08:04:22.022179-05:00
custom:
Author: benmosher
Issue: n/a

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Removing unused 'documentable'
time: 2023-11-06T10:57:30.694056-08:00
custom:
Author: QMalcolm
Issue: "8871"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove use of dbt/core exceptions in dbt/adapter
time: 2023-11-07T13:57:28.683727-08:00
custom:
Author: colin-rogers-dbt MichelleArk
Issue: "8920"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Cache dbt plugin modules to improve integration test performance
time: 2023-11-07T19:15:46.170151-05:00
custom:
Author: peterallenwebb
Issue: "9029"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Fix test_current_timestamp_matches_utc test; allow for MacOS runner system clock
variance
time: 2023-11-11T17:53:50.098843-05:00
custom:
Author: mikealfare
Issue: "9057"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific
event types and protos
time: 2023-11-16T17:42:51.005023-05:00
custom:
Author: michelleark
Issue: 8927 8918

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Clean up unused adaptor folders
time: 2023-11-20T13:47:35.923794-08:00
custom:
Author: ChenyuLInx
Issue: "9123"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move column constraints into common/contracts, removing another dependency of
adapters on core.
time: 2023-11-20T18:32:14.859503-05:00
custom:
Author: peterallenwebb
Issue: "9024"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move dbt.semver to dbt.common.semver and update references.
time: 2023-11-28T17:07:32.172421-08:00
custom:
Author: versusfacit
Issue: "9039"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move lowercase utils method to common
time: 2023-11-30T13:54:32.561673-08:00
custom:
Author: colin-rogers-dbt
Issue: "9180"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usages of dbt.clients.jinja in dbt/adapters
time: 2023-12-05T09:35:44.845352+09:00
custom:
Author: michelleark
Issue: "9205"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.contracts in dbt/adapters
time: 2023-12-05T12:05:59.936775+09:00
custom:
Author: michelleark
Issue: "9208"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.contracts.graph.nodes.ResultNode in dbt/adapters
time: 2023-12-05T16:58:12.932172+09:00
custom:
Author: michelleark
Issue: "9214"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Introduce RelationConfig Protocol, consolidate Relation.create_from
time: 2023-12-05T17:07:25.33861+09:00
custom:
Author: michelleark
Issue: "9215"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Create a no-op exposure runner
time: 2024-12-02T16:47:15.766574Z
custom:
Author: aranke
Issue: ' '

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Improve selection peformance by optimizing the select_children() and select_parents()
functions.
time: 2024-12-05T14:31:44.584216-05:00
custom:
Author: peterallenwebb
Issue: "11099"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Change exception type from DbtInternalException to UndefinedMacroError when
macro not found in 'run operation' command
time: 2025-01-07T12:39:55.234321-05:00
custom:
Author: michelleark
Issue: "11192"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Create LogNodeResult event
time: 2025-01-07T20:58:38.821036Z
custom:
Author: aranke
Issue: ' '

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Fix error counts for exposures
time: 2025-01-10T20:20:57.01632Z
custom:
Author: aranke
Issue: ' '

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Misc fixes for group info in logging
time: 2025-01-17T15:22:15.497485Z
custom:
Author: aranke
Issue: '11218'

View File

@@ -31,7 +31,43 @@ kinds:
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
@@ -70,10 +106,18 @@ footerFormat: |
{{- $changeList := splitList " " $change.Custom.Author }}
{{- $IssueList := list }}
{{- $changeLink := $change.Kind }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }}
{{- $changes := splitList " " $change.Custom.PR }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- else }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- end }}
{{- /* check if this contributor has other changes associated with them already */}}
{{- if hasKey $contributorDict $author }}

View File

@@ -7,7 +7,6 @@ ignore =
W503 # makes Flake8 work like black
W504
E203 # makes Flake8 work like black
E704 # makes Flake8 work like black
E741
E501 # long line checking is done in black
exclude = test/

25
.github/CODEOWNERS vendored
View File

@@ -13,6 +13,31 @@
# the core team as a whole will be assigned
* @dbt-labs/core-team
### ADAPTERS
# Adapter interface ("base" + "sql" adapter defaults, cache)
/core/dbt/adapters @dbt-labs/core-adapters
# Global project (default macros + materializations), starter project
/core/dbt/include @dbt-labs/core-adapters
# Postgres plugin
/plugins/ @dbt-labs/core-adapters
/plugins/postgres/setup.py @dbt-labs/core-adapters
# Functional tests for adapter plugins
/tests/adapter @dbt-labs/core-adapters
### TESTS
# Overlapping ownership for vast majority of unit + functional tests
# Perf regression testing framework
# This excludes the test project files itself since those aren't specific
# framework changes (excluded by not setting an owner next to it- no owner)
/performance @nathaniel-may
/performance/projects
### ARTIFACTS
/schemas/dbt @dbt-labs/cloud-artifacts

View File

@@ -1,18 +0,0 @@
name: 📄 Code docs
description: Report an issue for markdown files within this repo, such as README, ARCHITECTURE, etc.
title: "[Code docs] <title>"
labels: ["triage"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this code docs issue!
- type: textarea
attributes:
label: Please describe the issue and your proposals.
description: |
Links? References? Anything that will give us more context about the issue you are encountering!
Tip: You can attach images by clicking this area to highlight it and then dragging files in.
validations:
required: false

View File

@@ -1,8 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Documentation
url: https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose
about: Problems and issues with dbt product documentation hosted on docs.getdbt.com. Issues for markdown files within this repo, such as README, should be opened using the "Code docs" template.
- name: Ask the community for help
url: https://github.com/dbt-labs/docs.getdbt.com/discussions
about: Need help troubleshooting? Check out our guide on how to ask

View File

@@ -30,16 +30,6 @@ body:
What is the definition of done for this ticket? Include any relevant edge cases and/or test cases
validations:
required: true
- type: textarea
attributes:
label: Suggested Tests
description: |
Provide scenarios to test. Link to existing similar tests if appropriate.
placeholder: |
1. Test with no version specified in the schema file and use selection logic on a versioned model for a specific version. Expect pass.
2. Test with a version specified in the schema file that is no valid. Expect ParsingError.
validations:
required: true
- type: textarea
attributes:
label: Impact to Other Teams
@@ -62,6 +52,7 @@ body:
attributes:
label: Context
description: |
Provide the "why", motivation, and alternative approaches considered -- linking to previous refinement issues, spikes and documentation as appropriate
Provide the "why", motivation, and alternative approaches considered -- linking to previous refinement issues, spikes, Notion docs as appropriate
validations:
validations:
required: false

3
.github/_README.md vendored
View File

@@ -47,8 +47,7 @@ ___
### How to re-run jobs
- From the UI you can rerun from failure
- You can retrigger the cla check by commenting on the PR with `@cla-bot check`
- Some actions cannot be rerun in the GitHub UI. Namely the snyk checks and the cla check. Snyk checks are rerun by closing and reopening the PR. You can retrigger the cla check by commenting on the PR with `@cla-bot check`
___

View File

@@ -1,21 +1,20 @@
name: "GitHub package `latest` tag wrangler for containers"
description: "Determines if the published image should include `latest` tags"
name: "Github package 'latest' tag wrangler for containers"
description: "Determines wether or not a given dbt container should be given a bare 'latest' tag (I.E. dbt-core:latest)"
inputs:
package_name:
description: "Package being published (i.e. `dbt-core`, `dbt-redshift`, etc.)"
description: "Package to check (I.E. dbt-core, dbt-redshift, etc)"
required: true
new_version:
description: "SemVer of the package being published (i.e. 1.7.2, 1.8.0a1, etc.)"
description: "Semver of the container being built (I.E. 1.0.4)"
required: true
github_token:
description: "Auth token for GitHub (must have view packages scope)"
gh_token:
description: "Auth token for github (must have view packages scope)"
required: true
outputs:
tags:
description: "A list of tags to associate with this version"
latest:
description: "Wether or not built container should be tagged latest (bool)"
minor_latest:
description: "Wether or not built container should be tagged minor.latest (bool)"
runs:
using: "docker"
image: "Dockerfile"

View File

@@ -1,71 +1,98 @@
import os
from packaging.version import Version, parse
import requests
import sys
from typing import List
def main():
package_name: str = os.environ["INPUT_PACKAGE_NAME"]
new_version: Version = parse(os.environ["INPUT_NEW_VERSION"])
github_token: str = os.environ["INPUT_GITHUB_TOKEN"]
response = _package_metadata(package_name, github_token)
published_versions = _published_versions(response)
new_version_tags = _new_version_tags(new_version, published_versions)
_register_tags(new_version_tags, package_name)
def _package_metadata(package_name: str, github_token: str) -> requests.Response:
url = f"https://api.github.com/orgs/dbt-labs/packages/container/{package_name}/versions"
return requests.get(url, auth=("", github_token))
def _published_versions(response: requests.Response) -> List[Version]:
package_metadata = response.json()
return [
parse(tag)
for version in package_metadata
for tag in version["metadata"]["container"]["tags"]
if "latest" not in tag
]
def _new_version_tags(new_version: Version, published_versions: List[Version]) -> List[str]:
# the package version is always a tag
tags = [str(new_version)]
# pre-releases don't get tagged with `latest`
if new_version.is_prerelease:
return tags
if new_version > max(published_versions):
tags.append("latest")
published_patches = [
version
for version in published_versions
if version.major == new_version.major and version.minor == new_version.minor
]
if new_version > max(published_patches):
tags.append(f"{new_version.major}.{new_version.minor}.latest")
return tags
def _register_tags(tags: List[str], package_name: str) -> None:
fully_qualified_tags = ",".join([f"ghcr.io/dbt-labs/{package_name}:{tag}" for tag in tags])
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"fully_qualified_tags={fully_qualified_tags}")
def _validate_response(response: requests.Response) -> None:
message = response["message"]
if response.status_code != 200:
print(f"Call to GitHub API failed: {response.status_code} - {message}")
sys.exit(1)
import requests
from distutils.util import strtobool
from typing import Union
from packaging.version import parse, Version
if __name__ == "__main__":
main()
# get inputs
package = os.environ["INPUT_PACKAGE"]
new_version = parse(os.environ["INPUT_NEW_VERSION"])
gh_token = os.environ["INPUT_GH_TOKEN"]
halt_on_missing = strtobool(os.environ.get("INPUT_HALT_ON_MISSING", "False"))
# get package metadata from github
package_request = requests.get(
f"https://api.github.com/orgs/dbt-labs/packages/container/{package}/versions",
auth=("", gh_token),
)
package_meta = package_request.json()
# Log info if we don't get a 200
if package_request.status_code != 200:
print(f"Call to GH API failed: {package_request.status_code} {package_meta['message']}")
# Make an early exit if there is no matching package in github
if package_request.status_code == 404:
if halt_on_missing:
sys.exit(1)
# everything is the latest if the package doesn't exist
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write("latest=True")
gh_output.write("minor_latest=True")
sys.exit(0)
# TODO: verify package meta is "correct"
# https://github.com/dbt-labs/dbt-core/issues/4640
# map versions and tags
version_tag_map = {
version["id"]: version["metadata"]["container"]["tags"] for version in package_meta
}
# is pre-release
pre_rel = True if any(x in str(new_version) for x in ["a", "b", "rc"]) else False
# semver of current latest
for version, tags in version_tag_map.items():
if "latest" in tags:
# N.B. This seems counterintuitive, but we expect any version tagged
# 'latest' to have exactly three associated tags:
# latest, major.minor.latest, and major.minor.patch.
# Subtracting everything that contains the string 'latest' gets us
# the major.minor.patch which is what's needed for comparison.
current_latest = parse([tag for tag in tags if "latest" not in tag][0])
else:
current_latest = False
# semver of current_minor_latest
for version, tags in version_tag_map.items():
if f"{new_version.major}.{new_version.minor}.latest" in tags:
# Similar to above, only now we expect exactly two tags:
# major.minor.patch and major.minor.latest
current_minor_latest = parse([tag for tag in tags if "latest" not in tag][0])
else:
current_minor_latest = False
def is_latest(
pre_rel: bool, new_version: Version, remote_latest: Union[bool, Version]
) -> bool:
"""Determine if a given contaier should be tagged 'latest' based on:
- it's pre-release status
- it's version
- the version of a previously identified container tagged 'latest'
:param pre_rel: Wether or not the version of the new container is a pre-release
:param new_version: The version of the new container
:param remote_latest: The version of the previously identified container that's
already tagged latest or False
"""
# is a pre-release = not latest
if pre_rel:
return False
# + no latest tag found = is latest
if not remote_latest:
return True
# + if remote version is lower than current = is latest, else not latest
return True if remote_latest <= new_version else False
latest = is_latest(pre_rel, new_version, current_latest)
minor_latest = is_latest(pre_rel, new_version, current_minor_latest)
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"latest={latest}")
gh_output.write(f"minor_latest={minor_latest}")

View File

@@ -5,15 +5,6 @@ runs:
steps:
- shell: bash
run: |
sudo apt-get --purge remove postgresql postgresql-*
sudo apt update -y
sudo apt install gnupg2 wget vim -y
sudo sh -c 'echo "deb https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/postgresql.gpg
sudo apt update -y
sudo apt install postgresql-16
sudo apt-get -y install postgresql postgresql-contrib
sudo systemctl start postgresql
sudo systemctl enable postgresql
sudo systemctl start postgresql.service
pg_isready
sudo -u postgres bash ${{ github.action_path }}/setup_db.sh

View File

@@ -5,9 +5,7 @@ runs:
steps:
- shell: bash
run: |
brew install postgresql@16
brew link postgresql@16 --force
brew services start postgresql@16
brew services start postgresql
echo "Check PostgreSQL service is running"
i=10
COMMAND='pg_isready'

View File

@@ -5,22 +5,8 @@ runs:
steps:
- shell: pwsh
run: |
Write-Host -Object "Installing PostgreSQL 16 as windows service..."
$installerArgs = @("--install_runtimes 0", "--superpassword root", "--enable_acledit 1", "--unattendedmodeui none", "--mode unattended")
$filePath = Invoke-DownloadWithRetry -Url "https://get.enterprisedb.com/postgresql/postgresql-16.1-1-windows-x64.exe" -Path "$env:PGROOT/postgresql-16.1-1-windows-x64.exe"
Start-Process -FilePath $filePath -ArgumentList $installerArgs -Wait -PassThru
Write-Host -Object "Validating PostgreSQL 16 Install..."
Get-Service -Name postgresql*
$pgReady = Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
$exitCode = $pgReady.ExitCode
if ($exitCode -ne 0) {
Write-Host -Object "PostgreSQL is not ready. Exitcode: $exitCode"
exit $exitCode
}
Write-Host -Object "Starting PostgreSQL 16 Service..."
$pgService = Get-Service -Name postgresql-x64-16
$pgService = Get-Service -Name postgresql*
Set-Service -InputObject $pgService -Status running -StartupType automatic
Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
$env:Path += ";$env:PGBIN"
bash ${{ github.action_path }}/setup_db.sh

View File

@@ -11,6 +11,11 @@ updates:
schedule:
interval: "daily"
rebase-strategy: "disabled"
- package-ecosystem: "pip"
directory: "/plugins/postgres"
schedule:
interval: "daily"
rebase-strategy: "disabled"
# docker dependencies
- package-ecosystem: "docker"

Some files were not shown because too many files have changed in this diff Show More