Compare commits

..

124 Commits

Author SHA1 Message Date
Emily Rockman
c429d2d136 fix change kind (#9964) 2024-04-17 15:40:16 -05:00
Emily Rockman
f815b03178 bump sqlparse to 0.5 (#9951)
* bump sqlparse

* changelog
# Conflicts:
#	core/setup.py
2024-04-17 15:40:02 -05:00
Emily Rockman
a5c6d54baa [1.7] Fix Workflow Deprecations (#9799) (#9911) 2024-04-17 07:46:55 -05:00
Emily Rockman
677e2bc9d1 be less explicit (#9936) (#9944) 2024-04-15 12:26:54 -05:00
Emily Rockman
83a9f88c03 update to wrk for all versions (#9916) (#9917)
# Conflicts:
#	.github/workflows/main.yml
2024-04-15 08:00:03 -05:00
FishtownBuildBot
885d5f0701 [Automated] Merged prep-release/1.5.11_8469709936 into target 1.5.latest during release process 2024-03-28 11:02:02 -05:00
Github Build Bot
61db9208b0 Bumping version to 1.5.11 and generate changelog 2024-03-28 15:32:42 +00:00
Quigley Malcolm
afb675c767 Restrict protobuf to 4.* versions (#9630) (#9708)
Protobuf v5 has breaking changes. Here we are limiting the protobuf
dependency to one major version, 4, so that we don't have to patch
over handling 2 different major versions of protobuf.
2024-02-29 10:42:19 -08:00
FishtownBuildBot
a404cf6073 [Automated] Merged prep-release/1.5.10_8087144220 into target 1.5.latest during release process 2024-02-28 13:43:18 -08:00
Github Build Bot
b6f8b08a78 Bumping version to 1.5.10 and generate changelog 2024-02-28 21:14:22 +00:00
Quigley Malcolm
c94f132177 [Backport 1.5.latest] Restrict to protobuf 4 (#9676)
* Restrict protobuf to version 4.

* Restrict protobuf to major version 4.

---------

Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
2024-02-26 15:35:28 -08:00
Quigley Malcolm
b2270fa38a [Backport 1.5.latest] Upgrade Jinja2 dependency version specification to address CVE-2024-22195 (#9670)
* Upgrade Jinja2 dependency version specification to address CVE-2024-22195 (#9638)

CVE-2024-22195 identified an issue in Jinja2 versions <= 3.1.2. As such
we've gone and changed our dependency requirement specification to be
3.1.3 or greater (but less than 4).

Note: Preivously we were using the `~=` version specifier. However due
to some issues with the `~=` we've moved to using `>=` in combination
with `<`. This gives us the same range that `~=` gave us, but avoids
a pip resolution issue when multiple packages in an environment use `~=`
for the same dependency.
2024-02-26 15:09:30 -08:00
github-actions[bot]
dc3e667792 [Backport 1.5.latest] Pin pytest in dev-requirements.txt (#9476)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2024-01-30 16:45:31 +00:00
github-actions[bot]
5bbdab618d test pinning ddtrace (#9090) (#9094)
(cherry picked from commit 3902137dfc)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-11-15 17:04:59 -06:00
FishtownBuildBot
2708be3a82 [Automated] Merged prep-release/1.5.9_6709967767 into target 1.5.latest during release process 2023-10-31 13:45:40 -04:00
Github Build Bot
470fb5ede0 Bumping version to 1.5.9 and generate changelog 2023-10-31 17:06:11 +00:00
github-actions[bot]
4e250ad941 Contract enforcement on temporary tables (#8889) (#8904)
* add test

* fix test

* first pass with constraint error

* add back column checks for temp tables

* changelog

* Update .changes/unreleased/Fixes-20231024-145504.yaml

(cherry picked from commit 98310b6612)

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-10-26 10:00:46 -05:00
FishtownBuildBot
01b78980f1 [Automated] Merged prep-release/1.5.8_6485826531 into target 1.5.latest during release process 2023-10-11 13:54:42 -04:00
Github Build Bot
5262489f24 Bumping version to 1.5.8 and generate changelog 2023-10-11 17:15:53 +00:00
Jeremy Cohen
d81da57731 [Backport 1.5.latest] Fix #6497: Support global flags passed in after subcommands (#8734)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-10-11 16:18:15 +01:00
Michelle Ark
3c8cf858a2 [Backport 1.5.latest] respect project root when loading seeds (#8762) (#8805)
(cherry picked from commit 964e0e4e8a)
2023-10-11 15:55:24 +01:00
github-actions[bot]
543f87e17f [Backport 1.5.latest] Fix #8022: Foreign key constraint on incremental model results in Database Error (#8807)
(cherry picked from commit 6461f5aacf)

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-10-11 14:48:07 +01:00
github-actions[bot]
885a8a9652 Fix uncaught exception for group updates (#8792) (#8813)
* add test

* write test

* fix test

* updating test

* add clean

* cleanup

* more tests, fix comment

* add new test, move fixtures

(cherry picked from commit 4f9bd0cb38)

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-10-10 16:23:51 -05:00
Emily Rockman
6510a8b810 support doc blocks (#8771) (#8785) 2023-10-10 09:58:20 -05:00
FishtownBuildBot
c85aee1616 [Automated] Merged prep-release/1.5.7_6344197726 into target 1.5.latest during release process 2023-09-28 17:32:56 -05:00
Github Build Bot
41395c1bfb Bumping version to 1.5.7 and generate changelog 2023-09-28 21:42:48 +00:00
github-actions[bot]
6fb39bf35b Fix: avoid double-rendering sql_header in dbt show (#8740) (#8742)
(cherry picked from commit 408a78985a)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-09-28 14:19:06 -05:00
Emily Rockman
13816dc4df update changelog type (#8735) 2023-09-28 08:21:01 -07:00
Michelle Ark
4875c31bbd Inline limit in SQL sent from dbt show (#8641) (#8731) 2023-09-27 20:36:16 +01:00
Emily Rockman
76a276ca58 Support quoted parameter list for MultiOption cli options (#8665) (#8703)
* allow multioption to be quoted

* changelog

* fix test

* remove list format

* fix tests

* fix list object

* review arg change

* fix quotes

* Update .changes/unreleased/Features-20230918-150855.yaml

* add types

* convert list to set in test

* make mypy happy

* mroe mypy happiness

* more mypy happiness

* last mypy change

* add node to test
2023-09-26 09:59:04 -05:00
Emily Rockman
3862ef956a [BACKPORT] Split integration tests into parallel groups / jobs (#6346) (#8603)
* Split integration tests into parallel groups / jobs (#6346)

# Conflicts:
#	.github/workflows/main.yml

* rename job

---------

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-09-11 16:09:41 -05:00
Emily Rockman
00a5b3a9c8 Support dbt-cloud config dict in dbt_project.yml (#8527) (#8556)
* first pass at adding dbt-cloud config

* changelog

* fix test, add direct validation
2023-09-05 14:35:01 -05:00
Gerda Shank
3be943edb0 Fix snapshot success message to display "INSERT 0 1" (for example) instead of success (#8524) (#8531) 2023-08-31 14:55:44 -04:00
FishtownBuildBot
42addafa07 [Automated] Merged prep-release/1.5.6_5929068153 into target 1.5.latest during release process 2023-08-21 12:15:20 -05:00
Github Build Bot
46a90cab4d Bumping version to 1.5.6 and generate changelog 2023-08-21 16:41:55 +00:00
github-actions[bot]
96a222387a revert python version for docker images (#8445) (#8447)
* revert python version for docker images

* add comment to not update python version, update changelog

(cherry picked from commit f485c13035)

Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
2023-08-18 10:59:10 -05:00
FishtownBuildBot
fe2a5ba326 [Automated] Merged prep-release/1.5.5_5894165573 into target 1.5.latest during release process 2023-08-17 13:24:47 -05:00
Github Build Bot
bdc3adcabe Bumping version to 1.5.5 and generate changelog 2023-08-17 17:45:21 +00:00
github-actions[bot]
85c77acd80 Fix using project-dir with list command and path selector (#8388) (#8428)
(cherry picked from commit 048553ddc3)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-08-16 18:10:23 -04:00
Emily Rockman
8394bbd278 pin upper bound for sqlparse (#8236) (#8239)
* pin upper bound for sqlparse

* changelog
# Conflicts:
#	core/setup.py

Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
2023-08-15 17:56:46 -07:00
Emily Rockman
dd7a3293da loosen the click pin (#8232) (#8252)
* loosen the click pin

* changelog

* separate out sqlparse pin

* remove changelog

* add ignores

Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
2023-08-15 14:00:19 -07:00
Gerda Shank
4b21acdaad [Backport 1.5.latest] Ensure that target_schema from snapshot config is promoted to node level (#8117) (#8231)
(cherry-picked from fe9c875d32)
2023-08-15 13:36:38 -04:00
github-actions[bot]
5fc6d4d273 add a node status (#8174) (#8179)
(cherry picked from commit 0d645c227f)

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-08-15 11:44:54 +08:00
github-actions[bot]
a84883818f add env vars to tox.ini (#8365) (#8369)
* add env vars to tox.ini

* revert test

(cherry picked from commit b7aee3f5a4)

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-08-11 15:41:15 -05:00
github-actions[bot]
045a6d8412 add param to control maxBytes for single dbt.log file (#8200) (#8241)
* add param to control maxBytes for single dbt.log file

* nits

* nits

* Update core/dbt/cli/params.py

Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>

---------

Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
(cherry picked from commit f392add4b8)

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-08-01 13:12:38 -07:00
FishtownBuildBot
ed92f1369c [Automated] Merged prep-release/1.5.4_5692874947 into target 1.5.latest during release process 2023-07-28 09:54:48 -05:00
Github Build Bot
1be5a0cae8 Bumping version to 1.5.4 and generate changelog 2023-07-28 14:11:49 +00:00
FishtownBuildBot
a658072f77 [Automated] Merged prep-release/1.5.4rc1_5683294297 into target 1.5.latest during release process 2023-07-27 12:42:26 -05:00
Github Build Bot
7083c0e14f Bumping version to 1.5.4rc1 and generate changelog 2023-07-27 16:54:16 +00:00
Gerda Shank
a98b69c77c [Backport 1.5.latest] Initialize sqlparse lexer and tweak order of setting compilation fields (#8215) (#8221)
(cherry picked from commit fdeccfaf24)
2023-07-27 11:42:05 -04:00
Gerda Shank
fac2f62ea8 [Backport 1.5.latest] Rearrange pp_versioned_models test (#8157)
* Rearrange pp_versioned_models test (#8150)

(cherry picked from commit a32713198b)

* Do the equivalent of rename_if_exists
2023-07-26 12:06:49 -04:00
Quigley Malcolm
d39c53d160 [CT-2594] Backport #8180 to 1.5.latest (#8198)
* Add test ensuring `warn_error_options` is dictified in `invocation_args_dict` of contexts

* Add dictification specific to `warn_error_options` in `args_to_dict`

* Changie doc for serialization changes of warn_error_options
2023-07-24 16:00:55 -07:00
github-actions[bot]
d26028876f [Backport 1.5.latest] Hotfix for 372: Use JSONEncoder in json.dumps (#8163)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-07-21 16:06:38 -05:00
Emily Rockman
0d8f6d0784 add env vars for datadog ci visibility (#8097) (#8107)
* add env vars for datadog ci visibility

* modify pytest command for tracing

* fix posargs

* move env vars to job that needs them

* add test repeater to DD

* swap flags
# Conflicts:
#	.github/workflows/test-repeater.yml
2023-07-18 08:17:21 -05:00
FishtownBuildBot
efdbb2b9e2 [Automated] Merged prep-release/1.5.3_5578590246 into target 1.5.latest during release process 2023-07-17 12:57:43 -05:00
Github Build Bot
4bd3a40e4c Bumping version to 1.5.3 and generate changelog 2023-07-17 17:17:22 +00:00
Michelle Ark
774e905e26 Backport 8072 1.5.latest3 (#8084)
* Always build contract checksum for versioned model

* changelog entry
2023-07-13 12:14:43 +02:00
github-actions[bot]
2d7582997a Nicer error message for contracted model missing 'columns' (#8024) (#8041)
(cherry picked from commit 746ca7d149)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-07-11 20:20:08 +02:00
Emily Rockman
f466740cc5 pin click (#8050) (#8052)
* pin click

* changelog
# Conflicts:
#	core/setup.py
2023-07-07 12:38:17 -05:00
github-actions[bot]
b3975dbd96 allow on_schema_change: fail for incremental models with contracts (#8006) (#8037)
(cherry picked from commit 7ea51df6ae)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-07-06 11:53:34 +02:00
Jeremy Cohen
3e5e693442 fire proper event for inline query error (#7960) (#8021) 2023-07-05 08:36:14 -07:00
Gerda Shank
a935df97ca [BACKPORT to 1.5.latest] Remove pin of sqlparse to below 0.4.4 (#8008)
* Remove pin of sqlparse to below 0.4.4

* Changie
2023-06-30 13:54:37 -04:00
Gerda Shank
98fcb4ac55 Use events.contextvar because of multiprocessing unable to pickle ContextVar (#7949) (#7981)
Cherry-picked from commit 2e7c96841

* Add task contextvars to events/contextvars.py

* Use events.contextvars instead of task.contextvars

* Changie
2023-06-30 08:50:03 -04:00
Jeremy Cohen
b1874006e6 Add target_path to more cli commands that use it (#7647) (#7961)
Co-authored-by: Daniel Reeves <31971762+dwreeves@users.noreply.github.com>
2023-06-28 11:27:34 +02:00
FishtownBuildBot
66dd15b29d [Automated] Merged prep-release/1.5.2_5347682028 into target 1.5.latest during release process 2023-06-22 11:43:53 -05:00
Github Build Bot
39702dc4b7 Bumping version to 1.5.2 and generate changelog 2023-06-22 15:57:55 +00:00
FishtownBuildBot
36e6c67d47 [Automated] Merged prep-release/1.5.2rc2_5291645992 into target 1.5.latest during release process 2023-06-16 11:05:56 -05:00
Github Build Bot
043e511ec0 Bumping version to 1.5.2rc2 and generate changelog 2023-06-16 15:31:40 +00:00
colin-rogers-dbt
9776e7a7e8 backport 7862 to 1.5.latest (#7878)
* cherry pick f767943fb2

* Regenerate event proto types

---------

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-06-16 11:23:15 -04:00
github-actions[bot]
e1a9d9f94f Readd exp_path for config deprecation warnings (#7536) (#7737)
(cherry picked from commit 9c7e01dbca)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-06-14 13:15:52 -04:00
Jeremy Cohen
57dacb02b9 Backport #7838 to 1.5.latest (#7875)
* Changelog entry

* Fix: `dbt show --inline` with `private` models (#7838)

* Add functional test

* Check resource_type before DbtReferenceError

* Changelog entry
2023-06-14 13:15:34 -04:00
github-actions[bot]
49627fcac8 add ability to select models by access (#7739) (#7801) 2023-06-14 13:14:17 -04:00
github-actions[bot]
126d68732a Respect column quote config in model contracts (#7537) (#7858)
(cherry picked from commit 83d163add5)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-06-14 09:50:52 -04:00
github-actions[bot]
83f02516b1 generalize BaseModelConstraintsRuntimeEnforcement (#7805) (#7852) 2023-06-14 00:43:41 -04:00
Emily Rockman
693a338642 [Backport] #7779 to 1.5.latest (#7856)
* update adapters url (#7779)

* update adapters url

in response to [docs.getedbt.com pr 3465](https://github.com/dbt-labs/docs.getdbt.com/issues/3465), updating this error message to point to the correct URL, which was recently changed.

old URL: https://docs.getdbt.com/docs/supported-data-platforms#adapter-installation
new URL: https://docs.getdbt.com/docs/connect-adapters#install-using-the-cli

thank you @dbeatty10 for your 🦅 👀 !

* adding changie entry

* Update .changes/unreleased/Breaking Changes-20230612-161159.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
# Conflicts:
#	core/dbt/exceptions.py

* update error msg

---------

Co-authored-by: mirnawong1 <89008547+mirnawong1@users.noreply.github.com>
2023-06-13 16:04:59 -05:00
github-actions[bot]
11d9f9979e Use project directory in path selector instead of cwd (#7829) (#7850)
(cherry picked from commit ca73a2aa15)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-06-13 11:34:09 -04:00
github-actions[bot]
27eb48c1ff Fix constraint rendering for expressions and foreign key constraint types (#7512) (#7784)
(cherry picked from commit 05b0ebb184)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-06-09 16:48:52 -04:00
FishtownBuildBot
7f2bdbf85d [Automated] Merged prep-release/1.5.2rc1_5215591201 into target 1.5.latest during release process 2023-06-08 16:30:51 -05:00
Github Build Bot
66e0644558 Bumping version to 1.5.2rc1 and generate changelog 2023-06-08 20:50:36 +00:00
Emily Rockman
3a81fa9f47 add --target-path to snapshot command (#7419) (#7826)
Co-authored-by: Daniel Reeves <31971762+dwreeves@users.noreply.github.com>
2023-06-08 13:45:20 -05:00
github-actions[bot]
467cae06a2 Version 0 for model works for latest_version (#7712) (#7780)
(cherry picked from commit 79bd98560b)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-06-08 10:05:05 -04:00
github-actions[bot]
d94d6518e1 fix error message for empty/None: --warn-error-options handling (#7735) (#7817) 2023-06-07 13:49:52 -04:00
Emily Rockman
963a38fece [BACKPORT] Improve warning for constraints and mat types (#7806)
* Improve warnings for constraints and materialization types (#7696)

* first pass

* debugging

* regen proto types

* refactor to use warn_supported flag

* PR feedback

* regen proto files after conflicts

* fix problems wqith conflict resolution
2023-06-07 10:20:12 -05:00
github-actions[bot]
e5bd8b0233 pass optional sql_header to empty subquery sql rendering (#7734) (#7768) 2023-06-05 12:35:23 -04:00
Gerda Shank
f8cc136bf7 Target path should be relative to project dir, rather than current working directory (#7706) (#7715) 2023-06-05 10:20:05 -04:00
FishtownBuildBot
de9ee6a580 [Automated] Merged prep-release/1.5.1_5122894882 into target 1.5.latest during release process 2023-05-30 10:19:30 -05:00
Github Build Bot
a4ed7cefcb Bumping version to 1.5.1 and generate changelog 2023-05-30 14:46:14 +00:00
FishtownBuildBot
355f918afd [Automated] Merged prep-release/1.5.1rc2_5081528232 into target 1.5.latest during release process 2023-05-25 10:56:26 -05:00
Github Build Bot
630681b57f Bumping version to 1.5.1rc2 and generate changelog 2023-05-25 15:08:24 +00:00
github-actions[bot]
890bc168f3 CT 2516 ensure that paths in Jinja context flags object are strings (#7678) (#7692)
(cherry picked from commit 0516192d69)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-25 09:40:44 -04:00
github-actions[bot]
a3e6a487ef Missed PR fedback (#7642) (#7691)
(cherry picked from commit df23f68dd4)

Co-authored-by: Ian Knox <81931810+iknox-fa@users.noreply.github.com>
2023-05-24 08:29:13 -04:00
github-actions[bot]
8887c0ca0b bugfix: Deps hangs when using relative paths via --project-dir (#7628) (#7643)
(cherry picked from commit dcb5acdf29)

Co-authored-by: Ian Knox <81931810+iknox-fa@users.noreply.github.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-23 19:30:51 -07:00
Peter Webb
b06a8ebef4 Remove DelayedFileHandler (#7661) (#7683)
* remove DelayedFileHandler

* Changelog

* set_path to no-op

* more no-ops for rpc

* Clearer comments

Co-authored-by: Ian Knox <81931810+iknox-fa@users.noreply.github.com>
2023-05-23 09:54:47 -04:00
Emily Rockman
06c7a8a9cb Allow missing profiles.yml for dbt deps and dbt init (#7546) (#7677)
* Allow missing `profiles.yml` for `dbt deps` and `dbt init`

* Some commands allow the `--profiles-dir` to not exist

* Remove fix to verify that CI tests work

* Allow missing `profiles.yml` for `dbt deps` and `dbt init`

* CI is not finding any installed adapters

* Remove functional test for `dbt init`

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-05-22 15:28:26 -05:00
Peter Webb
19e2f2e8ce Profile Jinja Rendering, 1.5 Backport (#7653)
* Exclude some profile fields from Jinja rendering when they are not valid Jinja. (#7630)

* CT-2583: Exclude some profile fields from Jinja rendering.

* CT-2583: Add functional test.

* CT-2583: Change approach to password jinja detection

* CT-2583: Extract string constant and add additional checks

* CT-2583: Improve unit test coverage

* CT-2583: Update changelog entry to reflect new approach
2023-05-18 09:34:30 -04:00
Emily Rockman
9804e6715d cherry pick 7fbeced315 (#7638) 2023-05-16 11:48:50 -05:00
Gerda Shank
9642789817 CT 2510 Throw error for duplicate versioned and non versioned model n… (#7605)
* CT 2510 Throw error for duplicate versioned and non versioned model names (#7577)

* Check for versioned/unversioned duplicates

* Add new exception DuplicateVersionedUnversionedError

* Changie

* Handle packages when finding versioned and unversioned duplicates

(cherry picked from commit 29f2cfc48d)

* Issue AmbiguousAlias error after DuplicateResourceName
2023-05-12 15:42:36 -04:00
FishtownBuildBot
7844ac4bf4 [Automated] Merged prep-release/1.5.1rc1_4960109758 into target 1.5.latest during release process 2023-05-12 10:29:02 -05:00
Github Build Bot
7fa6ce8aa4 Bumping version to 1.5.1rc1 and generate changelog 2023-05-12 14:49:05 +00:00
Kshitij Aranke
2a3cab9ec7 Backport dbt show enhancements into 1.5.latest (#7578) 2023-05-11 09:44:58 -07:00
github-actions[bot]
c578e5b080 fix #7407: print model version in dbt show if specified (#7543) (#7557)
(cherry picked from commit 40aca4bc17)

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-05-10 09:41:21 -04:00
github-actions[bot]
cb9e4d5dae [Fix] Typo in statically parsed ref unpacking (#7365) (#7572)
test statically parsed two-argument ref

(cherry picked from commit 0891aef8d7)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-05-09 18:51:06 -04:00
github-actions[bot]
70c98a5495 CT 2552 pin protobuf to >=4.0.0 (#7566) (#7567)
* Pin protobuf to >=4.0.0

* Changie

(cherry picked from commit d34c511fa5)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-05-09 13:52:38 -04:00
github-actions[bot]
14796b2bd4 Do not rewrite manifest.json during 'docs serve' command (#7554) (#7555)
(cherry picked from commit 5a7b73be26)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-05-09 09:33:52 -04:00
github-actions[bot]
0cd20fffc3 Fix inverted --print/--no-print flag (#7524) (#7547)
(cherry picked from commit 19d6dab973)

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-05-08 15:39:38 -04:00
Jeremy Cohen
251917198c Back compat for previous return type of collect_freshness (#7535) (#7548)
* Back compat for previous retrurn type of 'collect_freshness'

* Test fixups

* PR feedback
2023-05-08 15:39:02 -04:00
FishtownBuildBot
2eaa4f084c [Automated] Merged prep-release/1.5.0_4819829606 into target 1.5.latest during release process 2023-04-27 08:24:23 -05:00
Github Build Bot
dafb6aeb93 Bumping version to 1.5.0 and generate changelog 2023-04-27 12:41:02 +00:00
github-actions[bot]
c79a65828f UX improvements to model versions (#7435) (#7462)
* Latest version should use un-suffixed alias

* Latest version can be in un-suffixed file

* FYI when unpinned ref to model with prerelease version

* [WIP] Nicer error if versioned ref to unversioned model

* Revert "Latest version should use un-suffixed alias"

This reverts commit 3616c52c1eed7588b9e210e1c957dfda598be550.

* Revert "[WIP] Nicer error if versioned ref to unversioned model"

This reverts commit c9ae4af1cfbd6b7bfc5dcbb445556233eb4bd2c0.

* Define real event for UnpinnedRefNewVersionAvailable

* Update pp test for implicit unsuffixed defined_in

* Add changelog entry

* Fix unit test

* marky feedback

* Add test case for UnpinnedRefNewVersionAvailable event

(cherry picked from commit d53bb37186)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-04-26 11:53:19 +02:00
github-actions[bot]
f37cb927ae Fix groupable node partial parsing, raise DbtReferenceError in RuntimeRefResolver (#7438) (#7461)
(cherry picked from commit 9874f9e004)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-04-25 14:04:44 -04:00
github-actions[bot]
77867d76f8 fix partial parsing of versioned models - schedule child nodes if latest version has been modified (#7439) (#7460)
(cherry picked from commit 2739d5f4c4)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-04-25 12:01:17 -04:00
FishtownBuildBot
1da608f2a0 [Automated] Merged prep-release/1.5.0rc2_4758640507 into target 1.5.latest during release process 2023-04-20 16:12:09 -05:00
Github Build Bot
1b6cf1a67f Bumping version to 1.5.0rc2 and generate changelog 2023-04-20 20:37:48 +00:00
github-actions[bot]
1d24e94e5d fix target dir behavior with sources.json (#7412) (#7423) 2023-04-20 15:54:13 -04:00
github-actions[bot]
437870eb13 fix v0 ref resolution and latest_version configuration(#7415) (#7422) 2023-04-20 15:26:09 -04:00
github-actions[bot]
dee5e70a06 sqlparse <0.4.4 (#7394) (#7398)
(cherry picked from commit 57e9096816)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-04-19 22:18:07 -04:00
github-actions[bot]
f0530e66b3 [Fix] safe version attribute access in _check_resource_uniqueness (#7376) (#7405)
safe version attribute access in _check_resource_uniqueness
2023-04-19 21:50:28 -04:00
github-actions[bot]
faa279d38d Update docs link in ContractBreakingChangeError message (#7367) (#7392)
(cherry picked from commit 6fedfe0ece)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-04-19 08:56:05 +02:00
github-actions[bot]
c1f3cc1f13 Update --help text for cache-related parameters (#7389) (#7409)
(cherry picked from commit ada8860e48)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-04-19 08:54:20 +02:00
Kshitij Aranke
d6526b037c Run 'changie new' on 1.5.latest branch (#7380) 2023-04-18 09:48:23 -07:00
Mike Alfare
ffb5a8ff37 updating make recipe for explicit order (#7385) 2023-04-18 01:08:19 -04:00
FishtownBuildBot
a67c14ee87 [Automated] Merged prep-release/1.5.0rc1_4694810085 into target 1.5.latest during release process 2023-04-13 19:20:25 -05:00
Github Build Bot
9ffe2647ee Bumping version to 1.5.0rc1 and generate changelog 2023-04-13 23:45:51 +00:00
Kshitij Aranke
06437df286 dbt 1.5.0rc1 (#7355) 2023-04-13 16:31:28 -07:00
339 changed files with 6432 additions and 18546 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.7.0a1
current_version = 1.5.11
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

View File

@@ -3,8 +3,6 @@
For information on prior major and minor releases, see their changelogs:
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)

189
.changes/1.5.0.md Normal file
View File

@@ -0,0 +1,189 @@
## dbt-core 1.5.0 - April 27, 2023
### Breaking Changes
- Allow `--select` and `--exclude` multiple times ([#7158](https://github.com/dbt-labs/dbt-core/issues/7158))
- Specifying "log-path" and "target-path" in "dbt_project.yml" is deprecated. This functionality will be removed in a future version of dbt-core. If you need to specify a custom path for logs or artifacts, please set via CLI flag or env var instead. ([#6882](https://github.com/dbt-labs/dbt-core/issues/6882))
- Remove exception functions marked as deprecated in 1.4 release ([#6578](https://github.com/dbt-labs/dbt-core/issues/6578))
### Features
- Data type constraints are now native to SQL table materializations. Enforce columns are specific data types and not null depending on database functionality. ([#6079](https://github.com/dbt-labs/dbt-core/issues/6079))
- Have dbt debug spit out structured json logs with flags enabled. ([#5353](https://github.com/dbt-labs/dbt-core/issues/5353))
- ✨ add unix-style wildcard selector method ([#6598](https://github.com/dbt-labs/dbt-core/issues/6598))
- add adapter_response to dbt test and freshness result ([#2964](https://github.com/dbt-labs/dbt-core/issues/2964))
- add support for DBT_PROJECT_DIR env var ([#6078](https://github.com/dbt-labs/dbt-core/issues/6078))
- Improve error message for packages missing `dbt_project.yml` ([#6663](https://github.com/dbt-labs/dbt-core/issues/6663))
- Make project version optional ([#6603](https://github.com/dbt-labs/dbt-core/issues/6603))
- Adjust makefile to have clearer instructions for CI env var changes. ([#6689](https://github.com/dbt-labs/dbt-core/issues/6689))
- Stand-alone Python module for PostgresColumn ([#6772](https://github.com/dbt-labs/dbt-core/issues/6772))
- Enable diff based partial parsing ([#6592](https://github.com/dbt-labs/dbt-core/issues/6592))
- Exposure owner requires one of name or email keys, and accepts additional arbitrary keys ([#6833](https://github.com/dbt-labs/dbt-core/issues/6833))
- Parse 'group' resource ([#6921](https://github.com/dbt-labs/dbt-core/issues/6921))
- parse 'group' config on groupable nodes ([#6823](https://github.com/dbt-labs/dbt-core/issues/6823))
- Implemented new log cli parameters for finer-grained control. ([#6639](https://github.com/dbt-labs/dbt-core/issues/6639))
- Add access attribute to parsed nodes ([#6824](https://github.com/dbt-labs/dbt-core/issues/6824))
- Enforce contracts on models materialized as tables, views, and incremental ([#6751](https://github.com/dbt-labs/dbt-core/issues/6751), [#7034](https://github.com/dbt-labs/dbt-core/issues/7034), [#6756](https://github.com/dbt-labs/dbt-core/issues/6756), [#7154](https://github.com/dbt-labs/dbt-core/issues/7154))
- Add ability to select by group resource ([#6825](https://github.com/dbt-labs/dbt-core/issues/6825))
- Disallow refing private model across groups ([#6826](https://github.com/dbt-labs/dbt-core/issues/6826))
- make version configs optional ([#7054](https://github.com/dbt-labs/dbt-core/issues/7054))
- [CT-1584] New top level commands: interactive compile ([#6358](https://github.com/dbt-labs/dbt-core/issues/6358))
- Make model contracts agnostic to ordering ([#6975](https://github.com/dbt-labs/dbt-core/issues/6975), [#7064](https://github.com/dbt-labs/dbt-core/issues/7064))
- Unified constraints and check_constraints properties for columns and models ([#7066](https://github.com/dbt-labs/dbt-core/issues/7066))
- Switch from betterproto to google protobuf and enable more flexible meta dictionary in logs ([#6832](https://github.com/dbt-labs/dbt-core/issues/6832))
- Ignore duplicate edges in subgraph to speed up dbt build ([#7191](https://github.com/dbt-labs/dbt-core/issues/7191))
- Support setting of callbacks for programmatic uses of `dbtRunner` ([#6763](https://github.com/dbt-labs/dbt-core/issues/6763))
- Detect breaking changes to contracts in state:modified check ([#6869](https://github.com/dbt-labs/dbt-core/issues/6869))
- New command: dbt show ([#7207](https://github.com/dbt-labs/dbt-core/issues/7207), [#7179](https://github.com/dbt-labs/dbt-core/issues/7179), [#6359](https://github.com/dbt-labs/dbt-core/issues/6359))
- Added prettier printing to ContractError class ([#7209](https://github.com/dbt-labs/dbt-core/issues/7209))
- Add support for model-level constraints ([#6754](https://github.com/dbt-labs/dbt-core/issues/6754))
- model versions ([##7263](https://github.com/dbt-labs/dbt-core/issues/#7263))
- Add relation info (database, schema, alias) to node_info dictionary in structured logging ([#6724](https://github.com/dbt-labs/dbt-core/issues/6724))
- Add --no-populate-cache to optionally skip relation cache population ([#1751](https://github.com/dbt-labs/dbt-core/issues/1751))
- select resources by patch path ([#7315](https://github.com/dbt-labs/dbt-core/issues/7315))
- Add version selector method ([#7199](https://github.com/dbt-labs/dbt-core/issues/7199))
### Fixes
- Remove trailing slashes from source paths (#6102) ([#6102](https://github.com/dbt-labs/dbt-core/issues/6102))
- add merge_exclude_columns adapter tests ([#6699](https://github.com/dbt-labs/dbt-core/issues/6699))
- Include adapter_response in NodeFinished run_result log event ([#6703](https://github.com/dbt-labs/dbt-core/issues/6703))
- Sort cli vars before hashing for partial parsing ([#6710](https://github.com/dbt-labs/dbt-core/issues/6710))
- [Regression] exposure_content referenced incorrectly ([#6738](https://github.com/dbt-labs/dbt-core/issues/6738))
- Snapshot strategies: add a newline for subquery ([#6781](https://github.com/dbt-labs/dbt-core/issues/6781))
- Remove pin on packaging and stop using it for prerelease comparisons ([#6834](https://github.com/dbt-labs/dbt-core/issues/6834))
- Readd depends_on.macros to SeedNode, to support seeds with hooks calling macros ([#6806](https://github.com/dbt-labs/dbt-core/issues/6806))
- Fix regression of --quiet cli parameter behavior ([#6749](https://github.com/dbt-labs/dbt-core/issues/6749))
- Add double type to list of float column types for the column class ([#6876](https://github.com/dbt-labs/dbt-core/issues/6876))
- Ensure results from hooks contain nodes when processing them ([#6796](https://github.com/dbt-labs/dbt-core/issues/6796))
- Always flush stdout after logging ([#6901](https://github.com/dbt-labs/dbt-core/issues/6901))
- Reapply logging fixes which were accidentally reverted ([#6936](https://github.com/dbt-labs/dbt-core/issues/6936))
- Set relation_name in test nodes at compile time ([#6930](https://github.com/dbt-labs/dbt-core/issues/6930))
- Readd initialization events, --log-cache-events in new CLI ([#6933](https://github.com/dbt-labs/dbt-core/issues/6933))
- Fix previous state tests and disabled exposures, metrics ([#6752](https://github.com/dbt-labs/dbt-core/issues/6752), [#6753](https://github.com/dbt-labs/dbt-core/issues/6753))
- Make use of hashlib.md5() FIPS compliant ([#6900](https://github.com/dbt-labs/dbt-core/issues/6900))
- add timeout for dbt --version command ([#6992](https://github.com/dbt-labs/dbt-core/issues/6992))
- Fix compilation logic for ephemeral nodes ([#6885](https://github.com/dbt-labs/dbt-core/issues/6885))
- Fix semver comparison logic by ensuring numeric values ([#7039](https://github.com/dbt-labs/dbt-core/issues/7039))
- add pytz dependency ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
- allow adapters to change model name resolution in py models ([#7114](https://github.com/dbt-labs/dbt-core/issues/7114))
- Add exception handling in postflight decorator to address exit codes ([#7010](https://github.com/dbt-labs/dbt-core/issues/7010))
- Recreates missing tracking events ([#6097](https://github.com/dbt-labs/dbt-core/issues/6097), [#6098](https://github.com/dbt-labs/dbt-core/issues/6098))
- Fix partial parsing error due to not requiring "version" ([#7236](https://github.com/dbt-labs/dbt-core/issues/7236))
- Handle internal exceptions ([#7118](https://github.com/dbt-labs/dbt-core/issues/7118))
- Improved failed event serialization handling and associated tests ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))
- Fix handling of artifacts in read_and_check_versions ([#7252](https://github.com/dbt-labs/dbt-core/issues/7252))
- Stringify datetimes in logging for prettier messages ([#7255](https://github.com/dbt-labs/dbt-core/issues/7255))
- avoid dbtRunner default callbacks being shared across instances ([#7278](https://github.com/dbt-labs/dbt-core/issues/7278))
- Ensure same_contract is called for state:modified ([#7282](https://github.com/dbt-labs/dbt-core/issues/7282))
- Avoid revoking grants for views when `copy_grants=true` ([#7280](https://github.com/dbt-labs/dbt-core/issues/7280))
- Duplicated flags now throw errors instead of being overidden by parent-level flag ([#6913](https://github.com/dbt-labs/dbt-core/issues/6913))
- Ensure that invocation_id changes between programmatic invocations. ([#7197](https://github.com/dbt-labs/dbt-core/issues/7197))
- Adding a new column is not a breaking contract change ([#7332](https://github.com/dbt-labs/dbt-core/issues/7332))
- fix versioned model selection in subdirectories ([#7348](https://github.com/dbt-labs/dbt-core/issues/7348))
- safe version attribute access in _check_resource_uniqueness ([#7375](https://github.com/dbt-labs/dbt-core/issues/7375))
- Fix dbt command missing target-path param ([# 7411](https://github.com/dbt-labs/dbt-core/issues/ 7411))
- Fix v0 ref resolution ([#7408](https://github.com/dbt-labs/dbt-core/issues/7408))
- fix groupable node partial parsing, raise DbtReferenceError at runtime for safety ([#7437](https://github.com/dbt-labs/dbt-core/issues/7437))
- Fix partial parsing of latest_version changes for downstream references ([#7369](https://github.com/dbt-labs/dbt-core/issues/7369))
### Docs
- Improve displayed message under "Arguments" section for argumentless macro ([dbt-docs/#358](https://github.com/dbt-labs/dbt-docs/issues/358))
- update link to installation instructions ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Fix JSON path to overview docs ([dbt-docs/#366](https://github.com/dbt-labs/dbt-docs/issues/366))
- Searchable column descriptions ([dbt-docs/#140](https://github.com/dbt-labs/dbt-docs/issues/140), [dbt-docs/#322](https://github.com/dbt-labs/dbt-docs/issues/322), [dbt-docs/#369](https://github.com/dbt-labs/dbt-docs/issues/369))
- Add access property to model details ([dbt-docs/#381](https://github.com/dbt-labs/dbt-docs/issues/381))
- Display model owner by name and email ([dbt-docs/#377](https://github.com/dbt-labs/dbt-docs/issues/377))
- Add view of public models sorted by group to left navigation ([dbt-docs/#379](https://github.com/dbt-labs/dbt-docs/issues/379))
- Distiguish node "access" in the DAG with node borders & opacity. ([dbt-docs/#378](https://github.com/dbt-labs/dbt-docs/issues/378))
- Fix JSON path to package overview docs ([dbt-docs/#390](https://github.com/dbt-labs/dbt-docs/issues/390))
- Add selection by group to DAG ([dbt-docs/#380](https://github.com/dbt-labs/dbt-docs/issues/380))
- Add support for model versions ([dbt-docs/#406](https://github.com/dbt-labs/dbt-docs/issues/406))
### Under the Hood
- [CT-921] dbt compile works in click ([#5545](https://github.com/dbt-labs/dbt-core/issues/5545))
- Fix use of ConnectionReused logging event ([#6168](https://github.com/dbt-labs/dbt-core/issues/6168))
- Port docs tests to pytest ([#6573](https://github.com/dbt-labs/dbt-core/issues/6573))
- Update deprecated github action command ([#6153](https://github.com/dbt-labs/dbt-core/issues/6153))
- dbt snapshot works in click ([#5554](https://github.com/dbt-labs/dbt-core/issues/5554))
- dbt list working with click ([#5549](https://github.com/dbt-labs/dbt-core/issues/5549))
- Add dbt run-operation to click CLI ([#5552](https://github.com/dbt-labs/dbt-core/issues/5552))
- dbt build working with new click framework ([#5541](https://github.com/dbt-labs/dbt-core/issues/5541))
- dbt docs generate works with new click framework ([#5543](https://github.com/dbt-labs/dbt-core/issues/5543))
- Replaced the EmptyLine event with a more general Formatting event, and added a Note event. ([#6481](https://github.com/dbt-labs/dbt-core/issues/6481))
- Small optimization on manifest parsing benefitting large DAGs ([#6697](https://github.com/dbt-labs/dbt-core/issues/6697))
- Revised and simplified various structured logging events ([#6664](https://github.com/dbt-labs/dbt-core/issues/6664), [#6665](https://github.com/dbt-labs/dbt-core/issues/6665), [#6666](https://github.com/dbt-labs/dbt-core/issues/6666))
- dbt init works with click ([#5548](https://github.com/dbt-labs/dbt-core/issues/5548))
- [CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs commands ([#5544](https://github.com/dbt-labs/dbt-core/issues/5544), [#6722](https://github.com/dbt-labs/dbt-core/issues/6722))
- Migrate debug task to click ([#5546](https://github.com/dbt-labs/dbt-core/issues/5546))
- Optimized GraphQueue to remove graph analysis bottleneck in large dags. ([#6759](https://github.com/dbt-labs/dbt-core/issues/6759))
- Implement --version for click cli ([#6757](https://github.com/dbt-labs/dbt-core/issues/6757))
- [CT-1841] Convert custom target test to Pytest ([#6638](https://github.com/dbt-labs/dbt-core/issues/6638))
- Remove BigQuery-specific btye abbreviations ([#6741](https://github.com/dbt-labs/dbt-core/issues/6741))
- warn_error/warn_error_options mutual exclusivity in click ([#6579](https://github.com/dbt-labs/dbt-core/issues/6579))
- Enables the new Click Cli on the commandline! 🚀 ([#6784](https://github.com/dbt-labs/dbt-core/issues/6784))
- Lazily call --version ([#6812](https://github.com/dbt-labs/dbt-core/issues/6812))
- Moving simple_seed to adapter zone to help adapter test conversions ([#CT-1959](https://github.com/dbt-labs/dbt-core/issues/CT-1959))
- flags.THREADS defaults to None ([#6887](https://github.com/dbt-labs/dbt-core/issues/6887))
- Fixing target type exposure error ([#6928](https://github.com/dbt-labs/dbt-core/issues/6928))
- Test binary serialization of logging events ([#6852](https://github.com/dbt-labs/dbt-core/issues/6852))
- Treat contract config as a python object ([#6748](https://github.com/dbt-labs/dbt-core/issues/6748), [#7184](https://github.com/dbt-labs/dbt-core/issues/7184))
- Add deprecation warning for DBT_NO_PRINT ([#6960](https://github.com/dbt-labs/dbt-core/issues/6960))
- Make output_keys click param multi-option instead of a string ([#6676](https://github.com/dbt-labs/dbt-core/issues/6676))
- Remove cli doc generation workflow ([#7088](https://github.com/dbt-labs/dbt-core/issues/7088))
- Move validation of group earlier ([#7087](https://github.com/dbt-labs/dbt-core/issues/7087))
- Deprecate additional environment variables ([#6903](https://github.com/dbt-labs/dbt-core/issues/6903))
- Add CommandCompleted event, and fire it upon completion of every command ([#6878](https://github.com/dbt-labs/dbt-core/issues/6878))
- Improves build times for common selections by improving subgraph calculation ([#7195](https://github.com/dbt-labs/dbt-core/issues/7195))
- Remove upper pin for hologram/jsonschema ([#6775](https://github.com/dbt-labs/dbt-core/issues/6775))
- Generalize constraint compatibility warnings ([#7067](https://github.com/dbt-labs/dbt-core/issues/7067))
- Add kwargs support to dbtRunner ([#7070](https://github.com/dbt-labs/dbt-core/issues/7070))
- Add unique_id to ShowNode and CompiledNode logging events ([#7305](https://github.com/dbt-labs/dbt-core/issues/7305))
- Prettify message for ListRelations event ([#7310](https://github.com/dbt-labs/dbt-core/issues/7310))
- `Parse` now returns manifest when invoked via dbtRunner ([#6547](https://github.com/dbt-labs/dbt-core/issues/6547))
- Track data about group, access, contract, version usage ([#7170](https://github.com/dbt-labs/dbt-core/issues/7170), [#7171](https://github.com/dbt-labs/dbt-core/issues/7171))
- Update docs link in ContractBreakingChangeError message ([#7366](https://github.com/dbt-labs/dbt-core/issues/7366))
- Update --help text for cache-related parameters ([#7381](https://github.com/dbt-labs/dbt-core/issues/7381))
- Small UX improvements to model versions: Support defining latest_version in unsuffixed file by default. Notify on unpinned ref when a prerelease version is available. ([#7443](https://github.com/dbt-labs/dbt-core/issues/7443))
### Dependencies
- Update pathspec requirement from <0.11,>=0.9 to >=0.9,<0.12 in /core ([#6737](https://github.com/dbt-labs/dbt-core/pull/6737))
- Bump ubuntu from 22.04 to 23.04 ([#6865](https://github.com/dbt-labs/dbt-core/pull/6865))
- Revert hoisting dbt.cli.main into the dbt.name namespace ([#](https://github.com/dbt-labs/dbt-core/pull/))
- Bump python from 3.11.1-slim-bullseye to 3.11.2-slim-bullseye in /docker ([#7196](https://github.com/dbt-labs/dbt-core/pull/7196))
- Bump black from 22.12.0 to 23.3.0 ([#7243](https://github.com/dbt-labs/dbt-core/pull/7243))
- Bump mashumaro[msgpack] from 3.3.1 to 3.6 ([#7294](https://github.com/dbt-labs/dbt-core/pull/7294))
### Dependency
- Bump mypy from 0.971 to 0.981 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904))
- Bump python from 3.10.7-slim-bullseye to 3.11.1-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904))
- Bump black from 22.10.0 to 22.12.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904))
### Contributors
- [@@ttusing](https://github.com/@ttusing) ([#7195](https://github.com/dbt-labs/dbt-core/issues/7195))
- [@Goodkat](https://github.com/Goodkat) ([#6992](https://github.com/dbt-labs/dbt-core/issues/6992))
- [@MartinGuindon](https://github.com/MartinGuindon) ([#358](https://github.com/dbt-labs/dbt-core/issues/358))
- [@MatthieuBlais](https://github.com/MatthieuBlais) ([#7191](https://github.com/dbt-labs/dbt-core/issues/7191))
- [@RobbertDM](https://github.com/RobbertDM) ([#6781](https://github.com/dbt-labs/dbt-core/issues/6781))
- [@aezomz](https://github.com/aezomz) ([#2964](https://github.com/dbt-labs/dbt-core/issues/2964))
- [@benallard](https://github.com/benallard) ([#7294](https://github.com/dbt-labs/dbt-core/pull/7294))
- [@boxysean](https://github.com/boxysean) ([#6697](https://github.com/dbt-labs/dbt-core/issues/6697))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#6928](https://github.com/dbt-labs/dbt-core/issues/6928))
- [@chamini2](https://github.com/chamini2) ([#7278](https://github.com/dbt-labs/dbt-core/issues/7278))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#7054](https://github.com/dbt-labs/dbt-core/issues/7054), [#7315](https://github.com/dbt-labs/dbt-core/issues/7315), [#6699](https://github.com/dbt-labs/dbt-core/issues/6699))
- [@davidbloss](https://github.com/davidbloss) ([#6153](https://github.com/dbt-labs/dbt-core/issues/6153))
- [@halvorlu](https://github.com/halvorlu) ([#366](https://github.com/dbt-labs/dbt-core/issues/366))
- [@jmg-duarte](https://github.com/jmg-duarte) ([#6102](https://github.com/dbt-labs/dbt-core/issues/6102))
- [@kentkr](https://github.com/kentkr) ([#7209](https://github.com/dbt-labs/dbt-core/issues/7209))
- [@leo-schick](https://github.com/leo-schick) ([#6078](https://github.com/dbt-labs/dbt-core/issues/6078))
- [@nielspardon](https://github.com/nielspardon) ([#6900](https://github.com/dbt-labs/dbt-core/issues/6900))
- [@rlh1994](https://github.com/rlh1994) ([#6876](https://github.com/dbt-labs/dbt-core/issues/6876), [#390](https://github.com/dbt-labs/dbt-core/issues/390))
- [@ryancharris](https://github.com/ryancharris) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@sdebruyn](https://github.com/sdebruyn) ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
- [@seub](https://github.com/seub) ([#6603](https://github.com/dbt-labs/dbt-core/issues/6603))
- [@sungchun12](https://github.com/sungchun12) ([#6079](https://github.com/dbt-labs/dbt-core/issues/6079))
- [@z3z1ma](https://github.com/z3z1ma) ([#6598](https://github.com/dbt-labs/dbt-core/issues/6598))

23
.changes/1.5.1.md Normal file
View File

@@ -0,0 +1,23 @@
## dbt-core 1.5.1 - May 30, 2023
### Fixes
- fix typo in unpacking statically parsed ref ([#7364](https://github.com/dbt-labs/dbt-core/issues/7364))
- Fix inverted `--print/--no-print` flag ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
- Back-compat for previous return type of 'collect_freshness' macro ([#7489](https://github.com/dbt-labs/dbt-core/issues/7489))
- print model version in dbt show if specified ([#7407](https://github.com/dbt-labs/dbt-core/issues/7407))
- Allow missing `profiles.yml` for `dbt deps` and `dbt init` ([#7511](https://github.com/dbt-labs/dbt-core/issues/7511))
- Do not rewrite manifest.json during 'docs serve' command ([#7553](https://github.com/dbt-labs/dbt-core/issues/7553))
- Pin protobuf to greater than 4.0.0 ([#7565](https://github.com/dbt-labs/dbt-core/issues/7565))
- Throw error for duplicated versioned and unversioned models ([#7487](https://github.com/dbt-labs/dbt-core/issues/7487))
- Fix: Relative project paths weren't working with deps ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491))
- Fall back if rendering the password field fails. ([#7629](https://github.com/dbt-labs/dbt-core/issues/7629))
- Stringify flag paths for Jinja context ([#7495](https://github.com/dbt-labs/dbt-core/issues/7495))
### Under the Hood
- Remove legacy file logger code ([#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
### Contributors
- [@iknox-fa](https://github.com/iknox-fa) ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491), [#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
- [@thomasgjerdekog](https://github.com/thomasgjerdekog) ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))

9
.changes/1.5.10.md Normal file
View File

@@ -0,0 +1,9 @@
## dbt-core 1.5.10 - February 28, 2024
### Under the Hood
- Restrict protobuf to major version 4. ([#9566](https://github.com/dbt-labs/dbt-core/issues/9566))
### Security
- Update Jinja2 to >= 3.1.3 to address CVE-2024-22195 ([#CVE-2024-22195](https://github.com/dbt-labs/dbt-core/pull/CVE-2024-22195))

5
.changes/1.5.11.md Normal file
View File

@@ -0,0 +1,5 @@
## dbt-core 1.5.11 - March 28, 2024
### Dependencies
- Restrict protobuf to 4.* versions ([#9566](https://github.com/dbt-labs/dbt-core/pull/9566))

26
.changes/1.5.2.md Normal file
View File

@@ -0,0 +1,26 @@
## dbt-core 1.5.2 - June 22, 2023
### Features
- add access selection syntax ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- Add AdapterRegistered event log message ([#7038](https://github.com/dbt-labs/dbt-core/issues/7038))
### Fixes
- Add --target-path to dbt snapshot command. ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- Constraint rendering fixes: wrap check expression in parentheses, foreign key 'references', support expression in all constraint types ([#7417](https://github.com/dbt-labs/dbt-core/issues/7417), [#7480](https://github.com/dbt-labs/dbt-core/issues/7480), [#7416](https://github.com/dbt-labs/dbt-core/issues/7416))
- Fix warning messages for deprecated dbt_project.yml configs ([#7424](https://github.com/dbt-labs/dbt-core/issues/7424))
- Respect column 'quote' config in model contracts ([#7370](https://github.com/dbt-labs/dbt-core/issues/7370))
- Improve warnings for constraints and materialization types ([#7335](https://github.com/dbt-labs/dbt-core/issues/7335))
- Incorrect paths used for "target" and "state" directories ([#7465](https://github.com/dbt-labs/dbt-core/issues/7465))
- Using version 0 works when resolving single model ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
- Fix empty --warn-error-options error message ([#7730](https://github.com/dbt-labs/dbt-core/issues/7730))
- send sql header on contract enforcement ([#7714](https://github.com/dbt-labs/dbt-core/issues/7714))
- Fix path selector when using project-dir ([#7819](https://github.com/dbt-labs/dbt-core/issues/7819))
- Allow dbt show --inline preview of private models ([#7837](https://github.com/dbt-labs/dbt-core/issues/7837))
- Updating this error message to point to the correct URL ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))
### Contributors
- [@dave-connors-3](https://github.com/dave-connors-3) ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- [@dwreeves](https://github.com/dwreeves) ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- [@mirnawong1](https://github.com/mirnawong1) ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))

18
.changes/1.5.3.md Normal file
View File

@@ -0,0 +1,18 @@
## dbt-core 1.5.3 - July 17, 2023
### Fixes
- Add --target-path to more CLI subcommands ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- Remove limitation on use of sqlparse 0.4.4 ([#7515](https://github.com/dbt-labs/dbt-core/issues/7515))
- Move project_root contextvar into events.contextvars ([#7937](https://github.com/dbt-labs/dbt-core/issues/7937))
- Inline query emit proper error message ([#7940](https://github.com/dbt-labs/dbt-core/issues/7940))
- Allow on_schema_change = fail for contracted incremental models ([#7975](https://github.com/dbt-labs/dbt-core/issues/7975))
- Nicer error message if model with enforced contract is missing 'columns' specification ([#7943](https://github.com/dbt-labs/dbt-core/issues/7943))
- Detect breaking contract changes to versioned models ([#8030](https://github.com/dbt-labs/dbt-core/issues/8030))
### Dependencies
- Pin click>=7.0,<8.1.4 ([#8050](https://github.com/dbt-labs/dbt-core/pull/8050))
### Contributors
- [@dwreeves](https://github.com/dwreeves) ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))

10
.changes/1.5.4.md Normal file
View File

@@ -0,0 +1,10 @@
## dbt-core 1.5.4 - July 28, 2023
### Fixes
- Ensure `warn_error_options` get serialized in `invocation_args_dict` ([#7694](https://github.com/dbt-labs/dbt-core/issues/7694))
- Improve handling of CTE injection with ephemeral models ([#8213](https://github.com/dbt-labs/dbt-core/issues/8213))
### Under the Hood
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))

16
.changes/1.5.5.md Normal file
View File

@@ -0,0 +1,16 @@
## dbt-core 1.5.5 - August 17, 2023
### Fixes
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Add status to Parse Inline Error ([#8173](https://github.com/dbt-labs/dbt-core/issues/8173))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
### Under the Hood
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
### Dependencies
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))

5
.changes/1.5.6.md Normal file
View File

@@ -0,0 +1,5 @@
## dbt-core 1.5.6 - August 21, 2023
### Under the Hood
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))

12
.changes/1.5.7.md Normal file
View File

@@ -0,0 +1,12 @@
## dbt-core 1.5.7 - September 28, 2023
### Features
- Accept a `dbt-cloud` config in dbt_project.yml ([#8438](https://github.com/dbt-labs/dbt-core/issues/8438))
### Fixes
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
- update dbt show to include limit in DWH query ([#8496,](https://github.com/dbt-labs/dbt-core/issues/8496,), [#8417](https://github.com/dbt-labs/dbt-core/issues/8417))
- Support quoted parameter list for MultiOption CLI options. ([#8598](https://github.com/dbt-labs/dbt-core/issues/8598))
- avoid double-rendering sql_header in dbt show ([#8739](https://github.com/dbt-labs/dbt-core/issues/8739))

9
.changes/1.5.8.md Normal file
View File

@@ -0,0 +1,9 @@
## dbt-core 1.5.8 - October 11, 2023
### Fixes
- Support global flags passed in after subcommands ([#6497](https://github.com/dbt-labs/dbt-core/issues/6497))
- Foreign key constraint on incremental model results in Database Error ([#8022](https://github.com/dbt-labs/dbt-core/issues/8022))
- Support docs blocks on versioned model column descriptions ([#8540](https://github.com/dbt-labs/dbt-core/issues/8540))
- Enable seeds to be handled from stored manifest data ([#6875](https://github.com/dbt-labs/dbt-core/issues/6875))
- Group updates on unmodified nodes are handled gracefully for state:modified ([#8371](https://github.com/dbt-labs/dbt-core/issues/8371))

5
.changes/1.5.9.md Normal file
View File

@@ -0,0 +1,5 @@
## dbt-core 1.5.9 - October 31, 2023
### Fixes
- Add back contract enforcement for temporary tables on postgres ([#8857](https://github.com/dbt-labs/dbt-core/issues/8857))

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Pin click<9 + sqlparse<0.5
time: 2023-07-19T12:37:43.716495+02:00
custom:
Author: jtcohen6
PR: "8146"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Fix for column tests not rendering on quoted columns
time: 2023-05-31T11:54:19.687363-04:00
custom:
Author: drewbanin
Issue: "201"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Remove static SQL codeblock for metrics
time: 2023-07-18T19:24:22.155323+02:00
custom:
Author: marcodamore
Issue: "436"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Enable converting deprecation warnings to errors
time: 2023-07-18T12:55:18.03914-04:00
custom:
Author: michelleark
Issue: "8130"

View File

@@ -0,0 +1,6 @@
kind: Security
body: Bump sqlparse to >=0.5.0, <0.6.0 to address GHSA-2m57-hf25-phgg
time: 2024-04-17T14:13:16.896353-05:00
custom:
Author: emmoop
Issue: "9951"

53
.github/CODEOWNERS vendored
View File

@@ -11,24 +11,44 @@
# As a default for areas with no assignment,
# the core team as a whole will be assigned
* @dbt-labs/core-team
* @dbt-labs/core
### OSS Tooling Guild
# Changes to GitHub configurations including Actions
/.github/ @leahwicz
/.github/ @dbt-labs/guild-oss-tooling
.bumpversion.cfg @dbt-labs/guild-oss-tooling
### LANGUAGE
.changie.yaml @dbt-labs/guild-oss-tooling
# Language core modules
/core/dbt/config/ @dbt-labs/core-language
/core/dbt/context/ @dbt-labs/core-language
/core/dbt/contracts/ @dbt-labs/core-language
/core/dbt/deps/ @dbt-labs/core-language
/core/dbt/events/ @dbt-labs/core-language # structured logging
/core/dbt/parser/ @dbt-labs/core-language
pre-commit-config.yaml @dbt-labs/guild-oss-tooling
pytest.ini @dbt-labs/guild-oss-tooling
tox.ini @dbt-labs/guild-oss-tooling
# Language misc files
/core/dbt/dataclass_schema.py @dbt-labs/core-language
/core/dbt/hooks.py @dbt-labs/core-language
/core/dbt/node_types.py @dbt-labs/core-language
/core/dbt/semver.py @dbt-labs/core-language
### EXECUTION
# Execution core modules
/core/dbt/graph/ @dbt-labs/core-execution
/core/dbt/task/ @dbt-labs/core-execution
# Execution misc files
/core/dbt/compilation.py @dbt-labs/core-execution
/core/dbt/flags.py @dbt-labs/core-execution
/core/dbt/lib.py @dbt-labs/core-execution
/core/dbt/main.py @dbt-labs/core-execution
/core/dbt/profiler.py @dbt-labs/core-execution
/core/dbt/selected_resources.py @dbt-labs/core-execution
/core/dbt/tracking.py @dbt-labs/core-execution
/core/dbt/version.py @dbt-labs/core-execution
pyproject.toml @dbt-labs/guild-oss-tooling
requirements.txt @dbt-labs/guild-oss-tooling
dev_requirements.txt @dbt-labs/guild-oss-tooling
/core/setup.py @dbt-labs/guild-oss-tooling
/core/MANIFEST.in @dbt-labs/guild-oss-tooling
### ADAPTERS
@@ -40,7 +60,6 @@ dev_requirements.txt @dbt-labs/guild-oss-tooling
# Postgres plugin
/plugins/ @dbt-labs/core-adapters
/plugins/postgres/setup.py @dbt-labs/core-adapters @dbt-labs/guild-oss-tooling
# Functional tests for adapter plugins
/tests/adapter @dbt-labs/core-adapters
@@ -52,9 +71,5 @@ dev_requirements.txt @dbt-labs/guild-oss-tooling
# Perf regression testing framework
# This excludes the test project files itself since those aren't specific
# framework changes (excluded by not setting an owner next to it- no owner)
/performance @nathaniel-may
/performance @nathaniel-may
/performance/projects
### ARTIFACTS
/schemas/dbt @dbt-labs/cloud-artifacts

2
.github/_README.md vendored
View File

@@ -197,7 +197,7 @@ ___
```yaml
- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v2
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

View File

@@ -35,7 +35,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v1
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler

View File

@@ -13,7 +13,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler

View File

@@ -1,35 +1,23 @@
resolves #
[docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose) dbt-labs/docs.getdbt.com/#
resolves #
<!---
Include the number of the issue addressed by this PR above if applicable.
PRs for code changes without an associated issue *will not be merged*.
See CONTRIBUTING.md for more information.
Include the number of the docs issue that was opened for this PR. If
this change has no user-facing implications, "N/A" suffices instead. New
docs tickets can be created by clicking the link above or by going to
https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose.
-->
### Problem
### Description
<!---
Describe the problem this PR is solving. What is the application state
before this PR is merged?
-->
### Solution
<!---
Describe the way this PR solves the above problem. Add as much detail as you
can to help reviewers understand your changes. Include any alternatives and
tradeoffs you considered.
Describe the Pull Request here. Add any references and info to help reviewers
understand your changes. Include any tradeoffs you considered.
-->
### Checklist
- [ ] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me
- [ ] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [ ] I have [opened an issue to add/update docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose), or docs changes are not required/relevant for this PR
- [ ] I have run `changie new` to [create a changelog entry](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-a-changelog-entry)

View File

@@ -50,7 +50,7 @@ jobs:
- name: Create and commit changelog on bot PR
if: ${{ contains(github.event.pull_request.labels.*.name, matrix.label) }}
id: bot_changelog
uses: emmyoop/changie_bot@v1.1.0
uses: emmyoop/changie_bot@v1.0.1
with:
GITHUB_TOKEN: ${{ secrets.FISHTOWN_BOT_PAT }}
commit_author_name: "Github Build Bot"

View File

@@ -18,8 +18,8 @@ permissions:
issues: write
jobs:
call-creation-action:
uses: dbt-labs/actions/.github/workflows/jira-creation-actions.yml@main
call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-creation.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -19,7 +19,7 @@ permissions:
jobs:
call-label-action:
uses: dbt-labs/actions/.github/workflows/jira-label-actions.yml@main
uses: dbt-labs/jira-actions/.github/workflows/jira-label.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -19,8 +19,8 @@ on:
permissions: read-all
jobs:
call-transition-action:
uses: dbt-labs/actions/.github/workflows/jira-transition-actions.yml@main
call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-transition.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -33,6 +33,11 @@ defaults:
run:
shell: bash
# top-level adjustments can be made here
env:
# number of parallel processes to spawn for python integration testing
PYTHON_INTEGRATION_TEST_WORKERS: ${{ vars.PYTHON_INTEGRATION_TEST_WORKERS }}
jobs:
code-quality:
name: code-quality
@@ -42,10 +47,10 @@ jobs:
steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: '3.8'
@@ -69,17 +74,18 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
env:
TOXENV: "unit"
PYTEST_ADDOPTS: "-v --color=yes --csv unit_results.csv"
steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
@@ -100,31 +106,64 @@ jobs:
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- name: Upload Unit Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: unit_results_${{ matrix.python-version }}-${{ steps.date.outputs.date }}.csv
path: unit_results.csv
integration-metadata:
name: integration test metadata generation
runs-on: ubuntu-latest
outputs:
split-groups: ${{ steps.generate-split-groups.outputs.split-groups }}
include: ${{ steps.generate-include.outputs.include }}
steps:
- name: generate split-groups
id: generate-split-groups
run: |
MATRIX_JSON="["
for B in $(seq 1 ${{ env.PYTHON_INTEGRATION_TEST_WORKERS }}); do
MATRIX_JSON+=$(sed 's/^/"/;s/$/"/' <<< "${B}")
done
MATRIX_JSON="${MATRIX_JSON//\"\"/\", \"}"
MATRIX_JSON+="]"
echo "split-groups=${MATRIX_JSON}"
echo "split-groups=${MATRIX_JSON}" >> $GITHUB_OUTPUT
- name: generate include
id: generate-include
run: |
INCLUDE=('"python-version":"3.8","os":"windows-latest"' '"python-version":"3.8","os":"macos-latest"' )
INCLUDE_GROUPS="["
for include in ${INCLUDE[@]}; do
for group in $(seq 1 ${{ env.PYTHON_INTEGRATION_TEST_WORKERS }}); do
INCLUDE_GROUPS+=$(sed 's/$/, /' <<< "{\"split-group\":\"${group}\",${include}}")
done
done
INCLUDE_GROUPS=$(echo $INCLUDE_GROUPS | sed 's/,*$//g')
INCLUDE_GROUPS+="]"
echo "include=${INCLUDE_GROUPS}"
echo "include=${INCLUDE_GROUPS}" >> $GITHUB_OUTPUT
integration:
name: integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
name: (${{ matrix.split-group }}) integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 60
timeout-minutes: 30
needs:
- integration-metadata
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-20.04]
include:
- python-version: 3.8
os: windows-latest
- python-version: 3.8
os: macos-latest
split-group: ${{ fromJson(needs.integration-metadata.outputs.split-groups) }}
include: ${{ fromJson(needs.integration-metadata.outputs.include) }}
env:
TOXENV: integration
PYTEST_ADDOPTS: "-v --color=yes -n4 --csv integration_results.csv"
DBT_INVOCATION_ENV: github-actions
DBT_TEST_USER_1: dbt_test_user_1
DBT_TEST_USER_2: dbt_test_user_2
@@ -137,10 +176,10 @@ jobs:
steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
@@ -165,6 +204,8 @@ jobs:
- name: Run tests
run: tox -- --ddtrace
env:
PYTEST_ADDOPTS: ${{ format('--splits {0} --group {1}', env.PYTHON_INTEGRATION_TEST_WORKERS, matrix.split-group) }}
- name: Get current date
if: always()
@@ -173,17 +214,27 @@ jobs:
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
if: always()
with:
name: logs_${{ matrix.python-version }}_${{ matrix.os }}_${{ steps.date.outputs.date }}
name: logs_${{ matrix.python-version }}_${{ matrix.os }}_${{ matrix.split-group }}_${{ steps.date.outputs.date }}
path: ./logs
- name: Upload Integration Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: integration_results_${{ matrix.python-version }}_${{ matrix.os }}_${{ steps.date.outputs.date }}.csv
path: integration_results.csv
integration-report:
if: ${{ always() }}
name: Integration Test Suite
runs-on: ubuntu-latest
needs: integration
steps:
- name: "[Notification] Integration test suite passes"
run: |
echo "::notice title="Integration test suite passes""
build:
name: build packages
@@ -192,10 +243,10 @@ jobs:
steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: '3.8'
@@ -230,7 +281,7 @@ jobs:
- name: Install source distributions
# ignore dbt-1.0.0, which intentionally raises an error when installed from source
run: |
find ./dist/dbt-[a-z]*.gz -maxdepth 1 -type f | xargs python -m pip install --force-reinstall --find-links=dist/
find ./dist/*.gz -maxdepth 1 -type f | xargs python -m pip install --force-reinstall --find-links=dist/
- name: Check source distributions
run: |

View File

@@ -1,265 +0,0 @@
# **what?**
# This workflow models the performance characteristics of a point in time in dbt.
# It runs specific dbt commands on committed projects multiple times to create and
# commit information about the distribution to the current branch. For more information
# see the readme in the performance module at /performance/README.md.
#
# **why?**
# When developing new features, we can take quick performance samples and compare
# them against the commited baseline measurements produced by this workflow to detect
# some performance regressions at development time before they reach users.
#
# **when?**
# This is only run once directly after each release (for non-prereleases). If for some
# reason the results of a run are not satisfactory, it can also be triggered manually.
name: Model Performance Characteristics
on:
# runs after non-prereleases are published.
release:
types: [released]
# run manually from the actions tab
workflow_dispatch:
inputs:
release_id:
description: 'dbt version to model (must be non-prerelease in Pypi)'
type: string
required: true
env:
RUNNER_CACHE_PATH: performance/runner/target/release/runner
# both jobs need to write
permissions:
contents: write
pull-requests: write
jobs:
set-variables:
name: Setting Variables
runs-on: ubuntu-latest
outputs:
cache_key: ${{ steps.variables.outputs.cache_key }}
release_id: ${{ steps.semver.outputs.base-version }}
release_branch: ${{ steps.variables.outputs.release_branch }}
steps:
# explicitly checkout the performance runner from main regardless of which
# version we are modeling.
- name: Checkout
uses: actions/checkout@v3
with:
ref: main
- name: Parse version into parts
id: semver
uses: dbt-labs/actions/parse-semver@v1
with:
version: ${{ github.event.inputs.release_id || github.event.release.tag_name }}
# collect all the variables that need to be used in subsequent jobs
- name: Set variables
id: variables
run: |
# create a cache key that will be used in the next job. without this the
# next job would have to checkout from main and hash the files itself.
echo "cache_key=${{ runner.os }}-${{ hashFiles('performance/runner/Cargo.toml')}}-${{ hashFiles('performance/runner/src/*') }}" >> $GITHUB_OUTPUT
branch_name="${{steps.semver.outputs.major}}.${{steps.semver.outputs.minor}}.latest"
echo "release_branch=$branch_name" >> $GITHUB_OUTPUT
echo "release branch is inferred to be ${branch_name}"
latest-runner:
name: Build or Fetch Runner
runs-on: ubuntu-latest
needs: [set-variables]
env:
RUSTFLAGS: "-D warnings"
steps:
- name: '[DEBUG] print variables'
run: |
echo "all variables defined in set-variables"
echo "cache_key: ${{ needs.set-variables.outputs.cache_key }}"
echo "release_id: ${{ needs.set-variables.outputs.release_id }}"
echo "release_branch: ${{ needs.set-variables.outputs.release_branch }}"
# explicitly checkout the performance runner from main regardless of which
# version we are modeling.
- name: Checkout
uses: actions/checkout@v3
with:
ref: main
# attempts to access a previously cached runner
- uses: actions/cache@v3
id: cache
with:
path: ${{ env.RUNNER_CACHE_PATH }}
key: ${{ needs.set-variables.outputs.cache_key }}
- name: Fetch Rust Toolchain
if: steps.cache.outputs.cache-hit != 'true'
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: stable
override: true
- name: Add fmt
if: steps.cache.outputs.cache-hit != 'true'
run: rustup component add rustfmt
- name: Cargo fmt
if: steps.cache.outputs.cache-hit != 'true'
uses: actions-rs/cargo@v1
with:
command: fmt
args: --manifest-path performance/runner/Cargo.toml --all -- --check
- name: Test
if: steps.cache.outputs.cache-hit != 'true'
uses: actions-rs/cargo@v1
with:
command: test
args: --manifest-path performance/runner/Cargo.toml
- name: Build (optimized)
if: steps.cache.outputs.cache-hit != 'true'
uses: actions-rs/cargo@v1
with:
command: build
args: --release --manifest-path performance/runner/Cargo.toml
# the cache action automatically caches this binary at the end of the job
model:
# depends on `latest-runner` as a separate job so that failures in this job do not prevent
# a successfully tested and built binary from being cached.
needs: [set-variables, latest-runner]
name: Model a release
runs-on: ubuntu-latest
steps:
- name: '[DEBUG] print variables'
run: |
echo "all variables defined in set-variables"
echo "cache_key: ${{ needs.set-variables.outputs.cache_key }}"
echo "release_id: ${{ needs.set-variables.outputs.release_id }}"
echo "release_branch: ${{ needs.set-variables.outputs.release_branch }}"
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.8"
- name: Install dbt
run: pip install dbt-postgres==${{ needs.set-variables.outputs.release_id }}
- name: Install Hyperfine
run: wget https://github.com/sharkdp/hyperfine/releases/download/v1.11.0/hyperfine_1.11.0_amd64.deb && sudo dpkg -i hyperfine_1.11.0_amd64.deb
# explicitly checkout main to get the latest project definitions
- name: Checkout
uses: actions/checkout@v3
with:
ref: main
# this was built in the previous job so it will be there.
- name: Fetch Runner
uses: actions/cache@v3
id: cache
with:
path: ${{ env.RUNNER_CACHE_PATH }}
key: ${{ needs.set-variables.outputs.cache_key }}
- name: Move Runner
run: mv performance/runner/target/release/runner performance/app
- name: Change Runner Permissions
run: chmod +x ./performance/app
- name: '[DEBUG] ls baseline directory before run'
run: ls -R performance/baselines/
# `${{ github.workspace }}` is used to pass the absolute path
- name: Create directories
run: |
mkdir ${{ github.workspace }}/performance/tmp/
mkdir -p performance/baselines/${{ needs.set-variables.outputs.release_id }}/
# Run modeling with taking 20 samples
- name: Run Measurement
run: |
performance/app model -v ${{ needs.set-variables.outputs.release_id }} -b ${{ github.workspace }}/performance/baselines/ -p ${{ github.workspace }}/performance/projects/ -t ${{ github.workspace }}/performance/tmp/ -n 20
- name: '[DEBUG] ls baseline directory after run'
run: ls -R performance/baselines/
- uses: actions/upload-artifact@v3
with:
name: baseline
path: performance/baselines/${{ needs.set-variables.outputs.release_id }}/
create-pr:
name: Open PR for ${{ matrix.base-branch }}
# depends on `model` as a separate job so that the baseline can be committed to more than one branch
# i.e. release branch and main
needs: [set-variables, latest-runner, model]
runs-on: ubuntu-latest
strategy:
matrix:
include:
- base-branch: refs/heads/main
target-branch: performance-bot/main_${{ needs.set-variables.outputs.release_id }}_${{GITHUB.RUN_ID}}
- base-branch: refs/heads/${{ needs.set-variables.outputs.release_branch }}
target-branch: performance-bot/release_${{ needs.set-variables.outputs.release_id }}_${{GITHUB.RUN_ID}}
steps:
- name: '[DEBUG] print variables'
run: |
echo "all variables defined in set-variables"
echo "cache_key: ${{ needs.set-variables.outputs.cache_key }}"
echo "release_id: ${{ needs.set-variables.outputs.release_id }}"
echo "release_branch: ${{ needs.set-variables.outputs.release_branch }}"
- name: Checkout
uses: actions/checkout@v3
with:
ref: ${{ matrix.base-branch }}
- name: Create PR branch
run: |
git checkout -b ${{ matrix.target-branch }}
git push origin ${{ matrix.target-branch }}
git branch --set-upstream-to=origin/${{ matrix.target-branch }} ${{ matrix.target-branch }}
- uses: actions/download-artifact@v3
with:
name: baseline
path: performance/baselines/${{ needs.set-variables.outputs.release_id }}
- name: '[DEBUG] ls baselines after artifact download'
run: ls -R performance/baselines/
- name: Commit baseline
uses: EndBug/add-and-commit@v9
with:
add: 'performance/baselines/*'
author_name: 'Github Build Bot'
author_email: 'buildbot@fishtownanalytics.com'
message: 'adding performance baseline for ${{ needs.set-variables.outputs.release_id }}'
push: 'origin origin/${{ matrix.target-branch }}'
- name: Create Pull Request
uses: peter-evans/create-pull-request@v5
with:
author: 'Github Build Bot <buildbot@fishtownanalytics.com>'
base: ${{ matrix.base-branch }}
branch: '${{ matrix.target-branch }}'
title: 'Adding performance modeling for ${{needs.set-variables.outputs.release_id}} to ${{ matrix.base-branch }}'
body: 'Committing perf results for tracking for the ${{needs.set-variables.outputs.release_id}}'
labels: |
Skip Changelog
Performance

View File

@@ -39,7 +39,7 @@ jobs:
steps:
- name: "Checkout ${{ github.repository }} Branch ${{ env.RELEASE_BRANCH }}"
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
ref: ${{ env.RELEASE_BRANCH }}

View File

@@ -1,7 +1,11 @@
# **what?**
# The purpose of this workflow is to trigger CI to run for each
# release branch and main branch on a regular cadence. If the CI workflow
# fails for a branch, it will post to #dev-core-alerts to raise awareness.
# fails for a branch, it will post to dev-core-alerts to raise awareness.
# The 'aurelien-baudet/workflow-dispatch' Action triggers the existing
# CI worklow file on the given branch to run so that even if we change the
# CI workflow file in the future, the one that is tailored for the given
# release branch will be used.
# **why?**
# Ensures release branches and main are always shippable and not broken.
@@ -24,8 +28,63 @@ on:
permissions: read-all
jobs:
run_tests:
uses: dbt-labs/actions/.github/workflows/release-branch-tests.yml@main
with:
workflows_to_run: '["main.yml"]'
secrets: inherit
fetch-latest-branches:
runs-on: ubuntu-latest
outputs:
latest-branches: ${{ steps.get-latest-branches.outputs.repo-branches }}
steps:
- name: "Fetch dbt-core Latest Branches"
uses: dbt-labs/actions/fetch-repo-branches@v1.1.1
id: get-latest-branches
with:
repo_name: ${{ github.event.repository.name }}
organization: "dbt-labs"
pat: ${{ secrets.GITHUB_TOKEN }}
fetch_protected_branches_only: true
regex: "^1.[0-9]+.latest$"
perform_match_method: "match"
retries: 3
- name: "[ANNOTATION] ${{ github.event.repository.name }} - branches to test"
run: |
title="${{ github.event.repository.name }} - branches to test"
message="The workflow will run tests for the following branches of the ${{ github.event.repository.name }} repo: ${{ steps.get-latest-branches.outputs.repo-branches }}"
echo "::notice $title::$message"
kick-off-ci:
needs: [fetch-latest-branches]
name: Kick-off CI
runs-on: ubuntu-latest
strategy:
# must run CI 1 branch at a time b/c the workflow-dispatch Action polls for
# latest run for results and it gets confused when we kick off multiple runs
# at once. There is a race condition so we will just run in sequential order.
max-parallel: 1
fail-fast: false
matrix:
branch: ${{ fromJSON(needs.fetch-latest-branches.outputs.latest-branches) }}
include:
- branch: 'main'
steps:
- name: Call CI workflow for ${{ matrix.branch }} branch
id: trigger-step
uses: aurelien-baudet/workflow-dispatch@v2.1.1
with:
workflow: main.yml
ref: ${{ matrix.branch }}
token: ${{ secrets.FISHTOWN_BOT_PAT }}
- name: Post failure to Slack
uses: ravsamhq/notify-slack-action@v1
if: ${{ always() && !contains(steps.trigger-step.outputs.workflow-conclusion,'success') }}
with:
status: ${{ job.status }}
notification_title: 'dbt-core scheduled run of "${{ matrix.branch }}" branch not successful'
message_format: ':x: CI on branch "${{ matrix.branch }}" ${{ steps.trigger-step.outputs.workflow-conclusion }}'
footer: 'Linked failed CI run ${{ steps.trigger-step.outputs.workflow-url }}'
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}

View File

@@ -36,7 +36,7 @@ jobs:
latest: ${{ steps.latest.outputs.latest }}
minor_latest: ${{ steps.latest.outputs.minor_latest }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Split version
id: version
run: |

View File

@@ -37,17 +37,17 @@ jobs:
steps:
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: 3.8
- name: Checkout dbt repo
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
path: ${{ env.DBT_REPO_DIRECTORY }}
- name: Checkout schemas.getdbt.com repo
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
repository: dbt-labs/schemas.getdbt.com
ref: 'main'
@@ -83,7 +83,7 @@ jobs:
fi
- name: Upload schema diff
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
if: ${{ failure() }}
with:
name: 'schema_schanges.txt'

View File

@@ -18,11 +18,41 @@ on:
permissions: read-all
# top-level adjustments can be made here
env:
# number of parallel processes to spawn for python testing
PYTHON_INTEGRATION_TEST_WORKERS: ${{ vars.PYTHON_INTEGRATION_TEST_WORKERS }}
jobs:
integration-metadata:
name: integration test metadata generation
runs-on: ubuntu-latest
outputs:
split-groups: ${{ steps.generate-split-groups.outputs.split-groups }}
steps:
- name: generate split-groups
id: generate-split-groups
run: |
MATRIX_JSON="["
for B in $(seq 1 ${{ env.PYTHON_INTEGRATION_TEST_WORKERS }}); do
MATRIX_JSON+=$(sed 's/^/"/;s/$/"/' <<< "${B}")
done
MATRIX_JSON="${MATRIX_JSON//\"\"/\", \"}"
MATRIX_JSON+="]"
echo "split-groups=${MATRIX_JSON}" >> $GITHUB_OUTPUT
# run the performance measurements on the current or default branch
test-schema:
name: Test Log Schema
runs-on: ubuntu-20.04
timeout-minutes: 30
needs:
- integration-metadata
strategy:
fail-fast: false
matrix:
split-group: ${{ fromJson(needs.integration-metadata.outputs.split-groups) }}
env:
# turns warnings into errors
RUSTFLAGS: "-D warnings"
@@ -39,12 +69,12 @@ jobs:
steps:
- name: checkout dev
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Setup Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.8"
@@ -65,3 +95,14 @@ jobs:
# we actually care if these pass, because the normal test run doesn't usually include many json log outputs
- name: Run integration tests
run: tox -e integration -- -nauto
env:
PYTEST_ADDOPTS: ${{ format('--splits {0} --group {1}', env.PYTHON_INTEGRATION_TEST_WORKERS, matrix.split-group) }}
test-schema-report:
name: Log Schema Test Suite
runs-on: ubuntu-latest
needs: test-schema
steps:
- name: "[Notification] Log test suite passes"
run: |
echo "::notice title="Log test suite passes""

View File

@@ -83,12 +83,12 @@ jobs:
steps:
- name: "Checkout code"
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
ref: ${{ inputs.branch }}
- name: "Setup Python"
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "${{ inputs.python_version }}"

View File

@@ -24,8 +24,10 @@ permissions:
jobs:
triage_label:
if: contains(github.event.issue.labels.*.name, 'awaiting_response')
uses: dbt-labs/actions/.github/workflows/swap-labels.yml@main
with:
add_label: "triage"
remove_label: "awaiting_response"
secrets: inherit
runs-on: ubuntu-latest
steps:
- name: initial labeling
uses: andymckay/labeler@master
with:
add-labels: "triage"
remove-labels: "awaiting_response"

3
.gitignore vendored
View File

@@ -11,7 +11,6 @@ __pycache__/
env*/
dbt_env/
build/
!tests/functional/build
!core/dbt/docs/build
develop-eggs/
dist/
@@ -29,8 +28,6 @@ var/
.mypy_cache/
.dmypy.json
logs/
.user.yml
profiles.yml
# PyInstaller
# Usually these files are written by a python script from a template

View File

@@ -37,7 +37,7 @@ repos:
alias: flake8-check
stages: [manual]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.3.0
rev: v0.981
hooks:
- id: mypy
# N.B.: Mypy is... a bit fragile.

View File

@@ -5,13 +5,352 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.5.11 - March 28, 2024
### Dependencies
- Restrict protobuf to 4.* versions ([#9566](https://github.com/dbt-labs/dbt-core/pull/9566))
## dbt-core 1.5.10 - February 28, 2024
### Under the Hood
- Restrict protobuf to major version 4. ([#9566](https://github.com/dbt-labs/dbt-core/issues/9566))
### Security
- Update Jinja2 to >= 3.1.3 to address CVE-2024-22195 ([#CVE-2024-22195](https://github.com/dbt-labs/dbt-core/pull/CVE-2024-22195))
## dbt-core 1.5.9 - October 31, 2023
### Fixes
- Add back contract enforcement for temporary tables on postgres ([#8857](https://github.com/dbt-labs/dbt-core/issues/8857))
## dbt-core 1.5.8 - October 11, 2023
### Fixes
- Support global flags passed in after subcommands ([#6497](https://github.com/dbt-labs/dbt-core/issues/6497))
- Foreign key constraint on incremental model results in Database Error ([#8022](https://github.com/dbt-labs/dbt-core/issues/8022))
- Support docs blocks on versioned model column descriptions ([#8540](https://github.com/dbt-labs/dbt-core/issues/8540))
- Enable seeds to be handled from stored manifest data ([#6875](https://github.com/dbt-labs/dbt-core/issues/6875))
- Group updates on unmodified nodes are handled gracefully for state:modified ([#8371](https://github.com/dbt-labs/dbt-core/issues/8371))
## dbt-core 1.5.7 - September 28, 2023
### Features
- Accept a `dbt-cloud` config in dbt_project.yml ([#8438](https://github.com/dbt-labs/dbt-core/issues/8438))
### Fixes
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
- update dbt show to include limit in DWH query ([#8496,](https://github.com/dbt-labs/dbt-core/issues/8496,), [#8417](https://github.com/dbt-labs/dbt-core/issues/8417))
- Support quoted parameter list for MultiOption CLI options. ([#8598](https://github.com/dbt-labs/dbt-core/issues/8598))
- avoid double-rendering sql_header in dbt show ([#8739](https://github.com/dbt-labs/dbt-core/issues/8739))
## dbt-core 1.5.6 - August 21, 2023
### Under the Hood
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))
## dbt-core 1.5.5 - August 17, 2023
### Fixes
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Add status to Parse Inline Error ([#8173](https://github.com/dbt-labs/dbt-core/issues/8173))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
### Under the Hood
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
### Dependencies
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))
## dbt-core 1.5.4 - July 28, 2023
### Fixes
- Ensure `warn_error_options` get serialized in `invocation_args_dict` ([#7694](https://github.com/dbt-labs/dbt-core/issues/7694))
- Improve handling of CTE injection with ephemeral models ([#8213](https://github.com/dbt-labs/dbt-core/issues/8213))
### Under the Hood
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))
## dbt-core 1.5.3 - July 17, 2023
### Fixes
- Add --target-path to more CLI subcommands ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- Remove limitation on use of sqlparse 0.4.4 ([#7515](https://github.com/dbt-labs/dbt-core/issues/7515))
- Move project_root contextvar into events.contextvars ([#7937](https://github.com/dbt-labs/dbt-core/issues/7937))
- Inline query emit proper error message ([#7940](https://github.com/dbt-labs/dbt-core/issues/7940))
- Allow on_schema_change = fail for contracted incremental models ([#7975](https://github.com/dbt-labs/dbt-core/issues/7975))
- Nicer error message if model with enforced contract is missing 'columns' specification ([#7943](https://github.com/dbt-labs/dbt-core/issues/7943))
- Detect breaking contract changes to versioned models ([#8030](https://github.com/dbt-labs/dbt-core/issues/8030))
### Dependencies
- Pin click>=7.0,<8.1.4 ([#8050](https://github.com/dbt-labs/dbt-core/pull/8050))
### Contributors
- [@dwreeves](https://github.com/dwreeves) ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
## dbt-core 1.5.2 - June 22, 2023
### Features
- add access selection syntax ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- Add AdapterRegistered event log message ([#7038](https://github.com/dbt-labs/dbt-core/issues/7038))
### Fixes
- Add --target-path to dbt snapshot command. ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- Constraint rendering fixes: wrap check expression in parentheses, foreign key 'references', support expression in all constraint types ([#7417](https://github.com/dbt-labs/dbt-core/issues/7417), [#7480](https://github.com/dbt-labs/dbt-core/issues/7480), [#7416](https://github.com/dbt-labs/dbt-core/issues/7416))
- Fix warning messages for deprecated dbt_project.yml configs ([#7424](https://github.com/dbt-labs/dbt-core/issues/7424))
- Respect column 'quote' config in model contracts ([#7370](https://github.com/dbt-labs/dbt-core/issues/7370))
- Improve warnings for constraints and materialization types ([#7335](https://github.com/dbt-labs/dbt-core/issues/7335))
- Incorrect paths used for "target" and "state" directories ([#7465](https://github.com/dbt-labs/dbt-core/issues/7465))
- Using version 0 works when resolving single model ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
- Fix empty --warn-error-options error message ([#7730](https://github.com/dbt-labs/dbt-core/issues/7730))
- send sql header on contract enforcement ([#7714](https://github.com/dbt-labs/dbt-core/issues/7714))
- Fix path selector when using project-dir ([#7819](https://github.com/dbt-labs/dbt-core/issues/7819))
- Allow dbt show --inline preview of private models ([#7837](https://github.com/dbt-labs/dbt-core/issues/7837))
- Updating this error message to point to the correct URL ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))
### Contributors
- [@dave-connors-3](https://github.com/dave-connors-3) ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- [@dwreeves](https://github.com/dwreeves) ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- [@mirnawong1](https://github.com/mirnawong1) ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))
## dbt-core 1.5.1 - May 30, 2023
### Fixes
- fix typo in unpacking statically parsed ref ([#7364](https://github.com/dbt-labs/dbt-core/issues/7364))
- Fix inverted `--print/--no-print` flag ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
- Back-compat for previous return type of 'collect_freshness' macro ([#7489](https://github.com/dbt-labs/dbt-core/issues/7489))
- print model version in dbt show if specified ([#7407](https://github.com/dbt-labs/dbt-core/issues/7407))
- Allow missing `profiles.yml` for `dbt deps` and `dbt init` ([#7511](https://github.com/dbt-labs/dbt-core/issues/7511))
- Do not rewrite manifest.json during 'docs serve' command ([#7553](https://github.com/dbt-labs/dbt-core/issues/7553))
- Pin protobuf to greater than 4.0.0 ([#7565](https://github.com/dbt-labs/dbt-core/issues/7565))
- Throw error for duplicated versioned and unversioned models ([#7487](https://github.com/dbt-labs/dbt-core/issues/7487))
- Fix: Relative project paths weren't working with deps ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491))
- Fall back if rendering the password field fails. ([#7629](https://github.com/dbt-labs/dbt-core/issues/7629))
- Stringify flag paths for Jinja context ([#7495](https://github.com/dbt-labs/dbt-core/issues/7495))
### Under the Hood
- Remove legacy file logger code ([#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
### Contributors
- [@iknox-fa](https://github.com/iknox-fa) ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491), [#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
- [@thomasgjerdekog](https://github.com/thomasgjerdekog) ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
## dbt-core 1.5.0 - April 27, 2023
### Breaking Changes
- Allow `--select` and `--exclude` multiple times ([#7158](https://github.com/dbt-labs/dbt-core/issues/7158))
- Specifying "log-path" and "target-path" in "dbt_project.yml" is deprecated. This functionality will be removed in a future version of dbt-core. If you need to specify a custom path for logs or artifacts, please set via CLI flag or env var instead. ([#6882](https://github.com/dbt-labs/dbt-core/issues/6882))
- Remove exception functions marked as deprecated in 1.4 release ([#6578](https://github.com/dbt-labs/dbt-core/issues/6578))
### Features
- Data type constraints are now native to SQL table materializations. Enforce columns are specific data types and not null depending on database functionality. ([#6079](https://github.com/dbt-labs/dbt-core/issues/6079))
- Have dbt debug spit out structured json logs with flags enabled. ([#5353](https://github.com/dbt-labs/dbt-core/issues/5353))
- add unix-style wildcard selector method ([#6598](https://github.com/dbt-labs/dbt-core/issues/6598))
- add adapter_response to dbt test and freshness result ([#2964](https://github.com/dbt-labs/dbt-core/issues/2964))
- add support for DBT_PROJECT_DIR env var ([#6078](https://github.com/dbt-labs/dbt-core/issues/6078))
- Improve error message for packages missing `dbt_project.yml` ([#6663](https://github.com/dbt-labs/dbt-core/issues/6663))
- Make project version optional ([#6603](https://github.com/dbt-labs/dbt-core/issues/6603))
- Adjust makefile to have clearer instructions for CI env var changes. ([#6689](https://github.com/dbt-labs/dbt-core/issues/6689))
- Stand-alone Python module for PostgresColumn ([#6772](https://github.com/dbt-labs/dbt-core/issues/6772))
- Enable diff based partial parsing ([#6592](https://github.com/dbt-labs/dbt-core/issues/6592))
- Exposure owner requires one of name or email keys, and accepts additional arbitrary keys ([#6833](https://github.com/dbt-labs/dbt-core/issues/6833))
- Parse 'group' resource ([#6921](https://github.com/dbt-labs/dbt-core/issues/6921))
- parse 'group' config on groupable nodes ([#6823](https://github.com/dbt-labs/dbt-core/issues/6823))
- Implemented new log cli parameters for finer-grained control. ([#6639](https://github.com/dbt-labs/dbt-core/issues/6639))
- Add access attribute to parsed nodes ([#6824](https://github.com/dbt-labs/dbt-core/issues/6824))
- Enforce contracts on models materialized as tables, views, and incremental ([#6751](https://github.com/dbt-labs/dbt-core/issues/6751), [#7034](https://github.com/dbt-labs/dbt-core/issues/7034), [#6756](https://github.com/dbt-labs/dbt-core/issues/6756), [#7154](https://github.com/dbt-labs/dbt-core/issues/7154))
- Add ability to select by group resource ([#6825](https://github.com/dbt-labs/dbt-core/issues/6825))
- Disallow refing private model across groups ([#6826](https://github.com/dbt-labs/dbt-core/issues/6826))
- make version configs optional ([#7054](https://github.com/dbt-labs/dbt-core/issues/7054))
- [CT-1584] New top level commands: interactive compile ([#6358](https://github.com/dbt-labs/dbt-core/issues/6358))
- Make model contracts agnostic to ordering ([#6975](https://github.com/dbt-labs/dbt-core/issues/6975), [#7064](https://github.com/dbt-labs/dbt-core/issues/7064))
- Unified constraints and check_constraints properties for columns and models ([#7066](https://github.com/dbt-labs/dbt-core/issues/7066))
- Switch from betterproto to google protobuf and enable more flexible meta dictionary in logs ([#6832](https://github.com/dbt-labs/dbt-core/issues/6832))
- Ignore duplicate edges in subgraph to speed up dbt build ([#7191](https://github.com/dbt-labs/dbt-core/issues/7191))
- Support setting of callbacks for programmatic uses of `dbtRunner` ([#6763](https://github.com/dbt-labs/dbt-core/issues/6763))
- Detect breaking changes to contracts in state:modified check ([#6869](https://github.com/dbt-labs/dbt-core/issues/6869))
- New command: dbt show ([#7207](https://github.com/dbt-labs/dbt-core/issues/7207), [#7179](https://github.com/dbt-labs/dbt-core/issues/7179), [#6359](https://github.com/dbt-labs/dbt-core/issues/6359))
- Added prettier printing to ContractError class ([#7209](https://github.com/dbt-labs/dbt-core/issues/7209))
- Add support for model-level constraints ([#6754](https://github.com/dbt-labs/dbt-core/issues/6754))
- model versions ([##7263](https://github.com/dbt-labs/dbt-core/issues/#7263))
- Add relation info (database, schema, alias) to node_info dictionary in structured logging ([#6724](https://github.com/dbt-labs/dbt-core/issues/6724))
- Add --no-populate-cache to optionally skip relation cache population ([#1751](https://github.com/dbt-labs/dbt-core/issues/1751))
- select resources by patch path ([#7315](https://github.com/dbt-labs/dbt-core/issues/7315))
- Add version selector method ([#7199](https://github.com/dbt-labs/dbt-core/issues/7199))
### Fixes
- Remove trailing slashes from source paths (#6102) ([#6102](https://github.com/dbt-labs/dbt-core/issues/6102))
- add merge_exclude_columns adapter tests ([#6699](https://github.com/dbt-labs/dbt-core/issues/6699))
- Include adapter_response in NodeFinished run_result log event ([#6703](https://github.com/dbt-labs/dbt-core/issues/6703))
- Sort cli vars before hashing for partial parsing ([#6710](https://github.com/dbt-labs/dbt-core/issues/6710))
- [Regression] exposure_content referenced incorrectly ([#6738](https://github.com/dbt-labs/dbt-core/issues/6738))
- Snapshot strategies: add a newline for subquery ([#6781](https://github.com/dbt-labs/dbt-core/issues/6781))
- Remove pin on packaging and stop using it for prerelease comparisons ([#6834](https://github.com/dbt-labs/dbt-core/issues/6834))
- Readd depends_on.macros to SeedNode, to support seeds with hooks calling macros ([#6806](https://github.com/dbt-labs/dbt-core/issues/6806))
- Fix regression of --quiet cli parameter behavior ([#6749](https://github.com/dbt-labs/dbt-core/issues/6749))
- Add double type to list of float column types for the column class ([#6876](https://github.com/dbt-labs/dbt-core/issues/6876))
- Ensure results from hooks contain nodes when processing them ([#6796](https://github.com/dbt-labs/dbt-core/issues/6796))
- Always flush stdout after logging ([#6901](https://github.com/dbt-labs/dbt-core/issues/6901))
- Reapply logging fixes which were accidentally reverted ([#6936](https://github.com/dbt-labs/dbt-core/issues/6936))
- Set relation_name in test nodes at compile time ([#6930](https://github.com/dbt-labs/dbt-core/issues/6930))
- Readd initialization events, --log-cache-events in new CLI ([#6933](https://github.com/dbt-labs/dbt-core/issues/6933))
- Fix previous state tests and disabled exposures, metrics ([#6752](https://github.com/dbt-labs/dbt-core/issues/6752), [#6753](https://github.com/dbt-labs/dbt-core/issues/6753))
- Make use of hashlib.md5() FIPS compliant ([#6900](https://github.com/dbt-labs/dbt-core/issues/6900))
- add timeout for dbt --version command ([#6992](https://github.com/dbt-labs/dbt-core/issues/6992))
- Fix compilation logic for ephemeral nodes ([#6885](https://github.com/dbt-labs/dbt-core/issues/6885))
- Fix semver comparison logic by ensuring numeric values ([#7039](https://github.com/dbt-labs/dbt-core/issues/7039))
- add pytz dependency ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
- allow adapters to change model name resolution in py models ([#7114](https://github.com/dbt-labs/dbt-core/issues/7114))
- Add exception handling in postflight decorator to address exit codes ([#7010](https://github.com/dbt-labs/dbt-core/issues/7010))
- Recreates missing tracking events ([#6097](https://github.com/dbt-labs/dbt-core/issues/6097), [#6098](https://github.com/dbt-labs/dbt-core/issues/6098))
- Fix partial parsing error due to not requiring "version" ([#7236](https://github.com/dbt-labs/dbt-core/issues/7236))
- Handle internal exceptions ([#7118](https://github.com/dbt-labs/dbt-core/issues/7118))
- Improved failed event serialization handling and associated tests ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))
- Fix handling of artifacts in read_and_check_versions ([#7252](https://github.com/dbt-labs/dbt-core/issues/7252))
- Stringify datetimes in logging for prettier messages ([#7255](https://github.com/dbt-labs/dbt-core/issues/7255))
- avoid dbtRunner default callbacks being shared across instances ([#7278](https://github.com/dbt-labs/dbt-core/issues/7278))
- Ensure same_contract is called for state:modified ([#7282](https://github.com/dbt-labs/dbt-core/issues/7282))
- Avoid revoking grants for views when `copy_grants=true` ([#7280](https://github.com/dbt-labs/dbt-core/issues/7280))
- Duplicated flags now throw errors instead of being overidden by parent-level flag ([#6913](https://github.com/dbt-labs/dbt-core/issues/6913))
- Ensure that invocation_id changes between programmatic invocations. ([#7197](https://github.com/dbt-labs/dbt-core/issues/7197))
- Adding a new column is not a breaking contract change ([#7332](https://github.com/dbt-labs/dbt-core/issues/7332))
- fix versioned model selection in subdirectories ([#7348](https://github.com/dbt-labs/dbt-core/issues/7348))
- safe version attribute access in _check_resource_uniqueness ([#7375](https://github.com/dbt-labs/dbt-core/issues/7375))
- Fix dbt command missing target-path param ([# 7411](https://github.com/dbt-labs/dbt-core/issues/ 7411))
- Fix v0 ref resolution ([#7408](https://github.com/dbt-labs/dbt-core/issues/7408))
- fix groupable node partial parsing, raise DbtReferenceError at runtime for safety ([#7437](https://github.com/dbt-labs/dbt-core/issues/7437))
- Fix partial parsing of latest_version changes for downstream references ([#7369](https://github.com/dbt-labs/dbt-core/issues/7369))
### Docs
- Improve displayed message under "Arguments" section for argumentless macro ([dbt-docs/#358](https://github.com/dbt-labs/dbt-docs/issues/358))
- update link to installation instructions ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Fix JSON path to overview docs ([dbt-docs/#366](https://github.com/dbt-labs/dbt-docs/issues/366))
- Searchable column descriptions ([dbt-docs/#140](https://github.com/dbt-labs/dbt-docs/issues/140), [dbt-docs/#322](https://github.com/dbt-labs/dbt-docs/issues/322), [dbt-docs/#369](https://github.com/dbt-labs/dbt-docs/issues/369))
- Add access property to model details ([dbt-docs/#381](https://github.com/dbt-labs/dbt-docs/issues/381))
- Display model owner by name and email ([dbt-docs/#377](https://github.com/dbt-labs/dbt-docs/issues/377))
- Add view of public models sorted by group to left navigation ([dbt-docs/#379](https://github.com/dbt-labs/dbt-docs/issues/379))
- Distiguish node "access" in the DAG with node borders & opacity. ([dbt-docs/#378](https://github.com/dbt-labs/dbt-docs/issues/378))
- Fix JSON path to package overview docs ([dbt-docs/#390](https://github.com/dbt-labs/dbt-docs/issues/390))
- Add selection by group to DAG ([dbt-docs/#380](https://github.com/dbt-labs/dbt-docs/issues/380))
- Add support for model versions ([dbt-docs/#406](https://github.com/dbt-labs/dbt-docs/issues/406))
### Under the Hood
- [CT-921] dbt compile works in click ([#5545](https://github.com/dbt-labs/dbt-core/issues/5545))
- Fix use of ConnectionReused logging event ([#6168](https://github.com/dbt-labs/dbt-core/issues/6168))
- Port docs tests to pytest ([#6573](https://github.com/dbt-labs/dbt-core/issues/6573))
- Update deprecated github action command ([#6153](https://github.com/dbt-labs/dbt-core/issues/6153))
- dbt snapshot works in click ([#5554](https://github.com/dbt-labs/dbt-core/issues/5554))
- dbt list working with click ([#5549](https://github.com/dbt-labs/dbt-core/issues/5549))
- Add dbt run-operation to click CLI ([#5552](https://github.com/dbt-labs/dbt-core/issues/5552))
- dbt build working with new click framework ([#5541](https://github.com/dbt-labs/dbt-core/issues/5541))
- dbt docs generate works with new click framework ([#5543](https://github.com/dbt-labs/dbt-core/issues/5543))
- Replaced the EmptyLine event with a more general Formatting event, and added a Note event. ([#6481](https://github.com/dbt-labs/dbt-core/issues/6481))
- Small optimization on manifest parsing benefitting large DAGs ([#6697](https://github.com/dbt-labs/dbt-core/issues/6697))
- Revised and simplified various structured logging events ([#6664](https://github.com/dbt-labs/dbt-core/issues/6664), [#6665](https://github.com/dbt-labs/dbt-core/issues/6665), [#6666](https://github.com/dbt-labs/dbt-core/issues/6666))
- dbt init works with click ([#5548](https://github.com/dbt-labs/dbt-core/issues/5548))
- [CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs commands ([#5544](https://github.com/dbt-labs/dbt-core/issues/5544), [#6722](https://github.com/dbt-labs/dbt-core/issues/6722))
- Migrate debug task to click ([#5546](https://github.com/dbt-labs/dbt-core/issues/5546))
- Optimized GraphQueue to remove graph analysis bottleneck in large dags. ([#6759](https://github.com/dbt-labs/dbt-core/issues/6759))
- Implement --version for click cli ([#6757](https://github.com/dbt-labs/dbt-core/issues/6757))
- [CT-1841] Convert custom target test to Pytest ([#6638](https://github.com/dbt-labs/dbt-core/issues/6638))
- Remove BigQuery-specific btye abbreviations ([#6741](https://github.com/dbt-labs/dbt-core/issues/6741))
- warn_error/warn_error_options mutual exclusivity in click ([#6579](https://github.com/dbt-labs/dbt-core/issues/6579))
- Enables the new Click Cli on the commandline! 🚀 ([#6784](https://github.com/dbt-labs/dbt-core/issues/6784))
- Lazily call --version ([#6812](https://github.com/dbt-labs/dbt-core/issues/6812))
- Moving simple_seed to adapter zone to help adapter test conversions ([#CT-1959](https://github.com/dbt-labs/dbt-core/issues/CT-1959))
- flags.THREADS defaults to None ([#6887](https://github.com/dbt-labs/dbt-core/issues/6887))
- Fixing target type exposure error ([#6928](https://github.com/dbt-labs/dbt-core/issues/6928))
- Test binary serialization of logging events ([#6852](https://github.com/dbt-labs/dbt-core/issues/6852))
- Treat contract config as a python object ([#6748](https://github.com/dbt-labs/dbt-core/issues/6748), [#7184](https://github.com/dbt-labs/dbt-core/issues/7184))
- Add deprecation warning for DBT_NO_PRINT ([#6960](https://github.com/dbt-labs/dbt-core/issues/6960))
- Make output_keys click param multi-option instead of a string ([#6676](https://github.com/dbt-labs/dbt-core/issues/6676))
- Remove cli doc generation workflow ([#7088](https://github.com/dbt-labs/dbt-core/issues/7088))
- Move validation of group earlier ([#7087](https://github.com/dbt-labs/dbt-core/issues/7087))
- Deprecate additional environment variables ([#6903](https://github.com/dbt-labs/dbt-core/issues/6903))
- Add CommandCompleted event, and fire it upon completion of every command ([#6878](https://github.com/dbt-labs/dbt-core/issues/6878))
- Improves build times for common selections by improving subgraph calculation ([#7195](https://github.com/dbt-labs/dbt-core/issues/7195))
- Remove upper pin for hologram/jsonschema ([#6775](https://github.com/dbt-labs/dbt-core/issues/6775))
- Generalize constraint compatibility warnings ([#7067](https://github.com/dbt-labs/dbt-core/issues/7067))
- Add kwargs support to dbtRunner ([#7070](https://github.com/dbt-labs/dbt-core/issues/7070))
- Add unique_id to ShowNode and CompiledNode logging events ([#7305](https://github.com/dbt-labs/dbt-core/issues/7305))
- Prettify message for ListRelations event ([#7310](https://github.com/dbt-labs/dbt-core/issues/7310))
- `Parse` now returns manifest when invoked via dbtRunner ([#6547](https://github.com/dbt-labs/dbt-core/issues/6547))
- Track data about group, access, contract, version usage ([#7170](https://github.com/dbt-labs/dbt-core/issues/7170), [#7171](https://github.com/dbt-labs/dbt-core/issues/7171))
- Update docs link in ContractBreakingChangeError message ([#7366](https://github.com/dbt-labs/dbt-core/issues/7366))
- Update --help text for cache-related parameters ([#7381](https://github.com/dbt-labs/dbt-core/issues/7381))
- Small UX improvements to model versions: Support defining latest_version in unsuffixed file by default. Notify on unpinned ref when a prerelease version is available. ([#7443](https://github.com/dbt-labs/dbt-core/issues/7443))
### Dependencies
- Update pathspec requirement from <0.11,>=0.9 to >=0.9,<0.12 in /core ([#6737](https://github.com/dbt-labs/dbt-core/pull/6737))
- Bump ubuntu from 22.04 to 23.04 ([#6865](https://github.com/dbt-labs/dbt-core/pull/6865))
- Revert hoisting dbt.cli.main into the dbt.name namespace ([#](https://github.com/dbt-labs/dbt-core/pull/))
- Bump python from 3.11.1-slim-bullseye to 3.11.2-slim-bullseye in /docker ([#7196](https://github.com/dbt-labs/dbt-core/pull/7196))
- Bump black from 22.12.0 to 23.3.0 ([#7243](https://github.com/dbt-labs/dbt-core/pull/7243))
- Bump mashumaro[msgpack] from 3.3.1 to 3.6 ([#7294](https://github.com/dbt-labs/dbt-core/pull/7294))
### Dependency
- Bump mypy from 0.971 to 0.981 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904))
- Bump python from 3.10.7-slim-bullseye to 3.11.1-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904))
- Bump black from 22.10.0 to 22.12.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904))
### Contributors
- [@@ttusing](https://github.com/@ttusing) ([#7195](https://github.com/dbt-labs/dbt-core/issues/7195))
- [@Goodkat](https://github.com/Goodkat) ([#6992](https://github.com/dbt-labs/dbt-core/issues/6992))
- [@MartinGuindon](https://github.com/MartinGuindon) ([#358](https://github.com/dbt-labs/dbt-core/issues/358))
- [@MatthieuBlais](https://github.com/MatthieuBlais) ([#7191](https://github.com/dbt-labs/dbt-core/issues/7191))
- [@RobbertDM](https://github.com/RobbertDM) ([#6781](https://github.com/dbt-labs/dbt-core/issues/6781))
- [@aezomz](https://github.com/aezomz) ([#2964](https://github.com/dbt-labs/dbt-core/issues/2964))
- [@benallard](https://github.com/benallard) ([#7294](https://github.com/dbt-labs/dbt-core/pull/7294))
- [@boxysean](https://github.com/boxysean) ([#6697](https://github.com/dbt-labs/dbt-core/issues/6697))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#6928](https://github.com/dbt-labs/dbt-core/issues/6928))
- [@chamini2](https://github.com/chamini2) ([#7278](https://github.com/dbt-labs/dbt-core/issues/7278))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#7054](https://github.com/dbt-labs/dbt-core/issues/7054), [#7315](https://github.com/dbt-labs/dbt-core/issues/7315), [#6699](https://github.com/dbt-labs/dbt-core/issues/6699))
- [@davidbloss](https://github.com/davidbloss) ([#6153](https://github.com/dbt-labs/dbt-core/issues/6153))
- [@halvorlu](https://github.com/halvorlu) ([#366](https://github.com/dbt-labs/dbt-core/issues/366))
- [@jmg-duarte](https://github.com/jmg-duarte) ([#6102](https://github.com/dbt-labs/dbt-core/issues/6102))
- [@kentkr](https://github.com/kentkr) ([#7209](https://github.com/dbt-labs/dbt-core/issues/7209))
- [@leo-schick](https://github.com/leo-schick) ([#6078](https://github.com/dbt-labs/dbt-core/issues/6078))
- [@nielspardon](https://github.com/nielspardon) ([#6900](https://github.com/dbt-labs/dbt-core/issues/6900))
- [@rlh1994](https://github.com/rlh1994) ([#6876](https://github.com/dbt-labs/dbt-core/issues/6876), [#390](https://github.com/dbt-labs/dbt-core/issues/390))
- [@ryancharris](https://github.com/ryancharris) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@sdebruyn](https://github.com/sdebruyn) ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
- [@seub](https://github.com/seub) ([#6603](https://github.com/dbt-labs/dbt-core/issues/6603))
- [@sungchun12](https://github.com/sungchun12) ([#6079](https://github.com/dbt-labs/dbt-core/issues/6079))
- [@z3z1ma](https://github.com/z3z1ma) ([#6598](https://github.com/dbt-labs/dbt-core/issues/6598))
## Previous Releases
For information on prior major and minor releases, see their changelogs:
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)

View File

@@ -5,10 +5,10 @@
1. [About this document](#about-this-document)
2. [Getting the code](#getting-the-code)
3. [Setting up an environment](#setting-up-an-environment)
4. [Running dbt-core in development](#running-dbt-core-in-development)
4. [Running `dbt` in development](#running-dbt-core-in-development)
5. [Testing dbt-core](#testing)
6. [Debugging](#debugging)
7. [Adding or modifying a changelog entry](#adding-or-modifying-a-changelog-entry)
7. [Adding a changelog entry](#adding-a-changelog-entry)
8. [Submitting a Pull Request](#submitting-a-pull-request)
## About this document
@@ -56,7 +56,7 @@ There are some tools that will be helpful to you in developing locally. While th
These are the tools used in `dbt-core` development and testing:
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.8, 3.9, 3.10 and 3.11
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.7, 3.8, 3.9, 3.10 and 3.11
- [`pytest`](https://docs.pytest.org/en/latest/) to define, discover, and run tests
- [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting
- [`black`](https://github.com/psf/black) for code formatting
@@ -113,7 +113,7 @@ When installed in this way, any changes you make to your local copy of the sourc
With your virtualenv activated, the `dbt` script should point back to the source code you've cloned on your machine. You can verify this by running `which dbt`. This command should show you a path to an executable in your virtualenv.
Configure your [profile](https://docs.getdbt.com/docs/configure-your-profile) as necessary to connect to your target databases. It may be a good idea to add a new profile pointing to a local Postgres instance, or a specific test sandbox within your data warehouse if appropriate. Make sure to create a profile before running integration tests.
Configure your [profile](https://docs.getdbt.com/docs/configure-your-profile) as necessary to connect to your target databases. It may be a good idea to add a new profile pointing to a local Postgres instance, or a specific test sandbox within your data warehouse if appropriate.
## Testing
@@ -163,7 +163,7 @@ suites.
#### `tox`
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.8, Python 3.9, Python 3.10 and Python 3.11 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py38`. The configuration for these tests in located in `tox.ini`.
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.7, Python 3.8, Python 3.9, Python 3.10 and Python 3.11 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration for these tests in located in `tox.ini`.
#### `pytest`
@@ -171,10 +171,12 @@ Finally, you can also run a specific test or group of tests using [`pytest`](htt
```sh
# run all unit tests in a file
python3 -m pytest tests/unit/test_graph.py
python3 -m pytest test/unit/test_graph.py
# run a specific unit test
python3 -m pytest tests/unit/test_graph.py::GraphTest::test__dependency_list
# run specific Postgres functional tests
python3 -m pytest test/unit/test_graph.py::GraphTest::test__dependency_list
# run specific Postgres integration tests (old way)
python3 -m pytest -m profile_postgres test/integration/074_postgres_unlogged_table_tests
# run specific Postgres integration tests (new way)
python3 -m pytest tests/functional/sources
```
@@ -183,8 +185,9 @@ python3 -m pytest tests/functional/sources
### Unit, Integration, Functional?
Here are some general rules for adding tests:
* unit tests (`tests/unit`) dont need to access a database; "pure Python" tests should be written as unit tests
* functional tests (`tests/functional`) cover anything that interacts with a database, namely adapter
* unit tests (`test/unit` & `tests/unit`) dont need to access a database; "pure Python" tests should be written as unit tests
* functional tests (`test/integration` & `tests/functional`) cover anything that interacts with a database, namely adapter
* *everything in* `test/*` *is being steadily migrated to* `tests/*`
## Debugging

View File

@@ -3,13 +3,13 @@
# See `/docker` for a generic and production-ready docker file
##
FROM ubuntu:22.04
FROM ubuntu:23.04
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
software-properties-common gpg-agent \
software-properties-common \
&& add-apt-repository ppa:git-core/ppa -y \
&& apt-get dist-upgrade -y \
&& apt-get install -y --no-install-recommends \
@@ -30,9 +30,16 @@ RUN apt-get update \
unixodbc-dev \
&& add-apt-repository ppa:deadsnakes/ppa \
&& apt-get install -y \
python-is-python3 \
python-dev-is-python3 \
python \
python-dev \
python3-pip \
python3.6 \
python3.6-dev \
python3-pip \
python3.6-venv \
python3.7 \
python3.7-dev \
python3.7-venv \
python3.8 \
python3.8-dev \
python3.8-venv \

View File

@@ -274,7 +274,7 @@ class BaseAdapter(metaclass=AdapterMeta):
@available.parse(lambda *a, **k: ("", empty_table()))
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
self, sql: str, auto_begin: bool = False, fetch: bool = False
) -> Tuple[AdapterResponse, agate.Table]:
"""Execute the given SQL. This is a thin wrapper around
ConnectionManager.execute.
@@ -283,22 +283,10 @@ class BaseAdapter(metaclass=AdapterMeta):
:param bool auto_begin: If set, and dbt is not currently inside a
transaction, automatically begin one.
:param bool fetch: If set, fetch results.
:param Optional[int] limit: If set, only fetch n number of rows
:return: A tuple of the query status and results (empty if fetch=False).
:rtype: Tuple[AdapterResponse, agate.Table]
"""
return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)
def validate_sql(self, sql: str) -> AdapterResponse:
"""Submit the given SQL to the engine for validation, but not execution.
This should throw an appropriate exception if the input SQL is invalid, although
in practice that will generally be handled by delegating to an existing method
for execution and allowing the error handler to take care of the rest.
:param str sql: The sql to validate
"""
raise NotImplementedError("`validate_sql` is not implemented for this adapter!")
return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch)
@available.parse(lambda *a, **k: [])
def get_column_schema_from_query(self, sql: str) -> List[BaseColumn]:
@@ -395,7 +383,7 @@ class BaseAdapter(metaclass=AdapterMeta):
return {
self.Relation.create_from(self.config, node).without_identifier()
for node in manifest.nodes.values()
if (node.is_relational and not node.is_ephemeral_model and not node.is_external_node)
if (node.is_relational and not node.is_ephemeral_model)
}
def _get_catalog_schemas(self, manifest: Manifest) -> SchemaSearchMap:
@@ -426,7 +414,7 @@ class BaseAdapter(metaclass=AdapterMeta):
return info_schema_name_map
def _relations_cache_for_schemas(
self, manifest: Manifest, cache_schemas: Optional[Set[BaseRelation]] = None
self, manifest: Manifest, cache_schemas: Set[BaseRelation] = None
) -> None:
"""Populate the relations cache for the given schemas. Returns an
iterable of the schemas populated, as strings.
@@ -462,7 +450,7 @@ class BaseAdapter(metaclass=AdapterMeta):
self,
manifest: Manifest,
clear: bool = False,
required_schemas: Optional[Set[BaseRelation]] = None,
required_schemas: Set[BaseRelation] = None,
) -> None:
"""Run a query that gets a populated cache of the relations in the
database and set the cache on this adapter.
@@ -796,6 +784,7 @@ class BaseAdapter(metaclass=AdapterMeta):
schema: str,
identifier: str,
) -> List[BaseRelation]:
matches = []
search = self._make_match_kwargs(database, schema, identifier)
@@ -996,7 +985,7 @@ class BaseAdapter(metaclass=AdapterMeta):
manifest: Optional[Manifest] = None,
project: Optional[str] = None,
context_override: Optional[Dict[str, Any]] = None,
kwargs: Optional[Dict[str, Any]] = None,
kwargs: Dict[str, Any] = None,
text_only_columns: Optional[Iterable[str]] = None,
) -> AttrDict:
"""Look macro_name up in the manifest and execute its results.
@@ -1073,6 +1062,7 @@ class BaseAdapter(metaclass=AdapterMeta):
schemas: Set[str],
manifest: Manifest,
) -> agate.Table:
kwargs = {"information_schema": information_schema, "schemas": schemas}
table = self.execute_macro(
GET_CATALOG_MACRO_NAME,
@@ -1462,6 +1452,7 @@ join diff_count using (id)
def catch_as_completed(
futures, # typing: List[Future[agate.Table]]
) -> Tuple[agate.Table, List[Exception]]:
# catalogs: agate.Table = agate.Table(rows=[])
tables: List[agate.Table] = []
exceptions: List[Exception] = []

View File

@@ -227,7 +227,7 @@ class BaseRelation(FakeAPIObject, Hashable):
def create_from_node(
cls: Type[Self],
config: HasQuoting,
node,
node: ManifestNode,
quote_policy: Optional[Dict[str, bool]] = None,
**kwargs: Any,
) -> Self:
@@ -328,10 +328,6 @@ class BaseRelation(FakeAPIObject, Hashable):
def is_view(self) -> bool:
return self.type == RelationType.View
@property
def is_materialized_view(self) -> bool:
return self.type == RelationType.MaterializedView
@classproperty
def Table(cls) -> str:
return str(RelationType.Table)
@@ -348,10 +344,6 @@ class BaseRelation(FakeAPIObject, Hashable):
def External(cls) -> str:
return str(RelationType.External)
@classproperty
def MaterializedView(cls) -> str:
return str(RelationType.MaterializedView)
@classproperty
def get_relation_type(cls) -> Type[RelationType]:
return RelationType

View File

@@ -165,9 +165,6 @@ class AdapterContainer:
def get_adapter_type_names(self, name: Optional[str]) -> List[str]:
return [p.adapter.type() for p in self.get_adapter_plugins(name)]
def get_adapter_constraint_support(self, name: Optional[str]) -> List[str]:
return self.lookup_adapter(name).CONSTRAINT_SUPPORT # type: ignore
FACTORY: AdapterContainer = AdapterContainer()
@@ -224,10 +221,6 @@ def get_adapter_type_names(name: Optional[str]) -> List[str]:
return FACTORY.get_adapter_type_names(name)
def get_adapter_constraint_support(name: Optional[str]) -> List[str]:
return FACTORY.get_adapter_constraint_support(name)
@contextmanager
def adapter_management():
reset_adapters()

View File

@@ -1,25 +0,0 @@
# RelationConfig
This package serves as an initial abstraction for managing the inspection of existing relations and determining
changes on those relations. It arose from the materialized view work and is currently only supporting
materialized views for Postgres and Redshift as well as dynamic tables for Snowflake. There are three main
classes in this package.
## RelationConfigBase
This is a very small class that only has a `from_dict()` method and a default `NotImplementedError()`. At some
point this could be replaced by a more robust framework, like `mashumaro` or `pydantic`.
## RelationConfigChange
This class inherits from `RelationConfigBase` ; however, this can be thought of as a separate class. The subclassing
merely points to the idea that both classes would likely inherit from the same class in a `mashumaro` or
`pydantic` implementation. This class is much more restricted in attribution. It should really only
ever need an `action` and a `context`. This can be though of as being analogous to a web request. You need to
know what you're doing (`action`: 'create' = GET, 'drop' = DELETE, etc.) and the information (`context`) needed
to make the change. In our scenarios, the context tends to be an instance of `RelationConfigBase` corresponding
to the new state.
## RelationConfigValidationMixin
This mixin provides optional validation mechanics that can be applied to either `RelationConfigBase` or
`RelationConfigChange` subclasses. A validation rule is a combination of a `validation_check`, something
that should evaluate to `True`, and an optional `validation_error`, an instance of `DbtRuntimeError`
that should be raised in the event the `validation_check` fails. While optional, it's recommended that
the `validation_error` be provided for clearer transparency to the end user.

View File

@@ -1,12 +0,0 @@
from dbt.adapters.relation_configs.config_base import ( # noqa: F401
RelationConfigBase,
RelationResults,
)
from dbt.adapters.relation_configs.config_change import ( # noqa: F401
RelationConfigChangeAction,
RelationConfigChange,
)
from dbt.adapters.relation_configs.config_validation import ( # noqa: F401
RelationConfigValidationMixin,
RelationConfigValidationRule,
)

View File

@@ -1,44 +0,0 @@
from dataclasses import dataclass
from typing import Union, Dict
import agate
from dbt.utils import filter_null_values
"""
This is what relation metadata from the database looks like. It's a dictionary because there will be
multiple grains of data for a single object. For example, a materialized view in Postgres has base level information,
like name. But it also can have multiple indexes, which needs to be a separate query. It might look like this:
{
"base": agate.Row({"table_name": "table_abc", "query": "select * from table_def"})
"indexes": agate.Table("rows": [
agate.Row({"name": "index_a", "columns": ["column_a"], "type": "hash", "unique": False}),
agate.Row({"name": "index_b", "columns": ["time_dim_a"], "type": "btree", "unique": False}),
])
}
"""
RelationResults = Dict[str, Union[agate.Row, agate.Table]]
@dataclass(frozen=True)
class RelationConfigBase:
@classmethod
def from_dict(cls, kwargs_dict) -> "RelationConfigBase":
"""
This assumes the subclass of `RelationConfigBase` is flat, in the sense that no attribute is
itself another subclass of `RelationConfigBase`. If that's not the case, this should be overriden
to manually manage that complexity.
Args:
kwargs_dict: the dict representation of this instance
Returns: the `RelationConfigBase` representation associated with the provided dict
"""
return cls(**filter_null_values(kwargs_dict)) # type: ignore
@classmethod
def _not_implemented_error(cls) -> NotImplementedError:
return NotImplementedError(
"This relation type has not been fully configured for this adapter."
)

View File

@@ -1,23 +0,0 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Hashable
from dbt.adapters.relation_configs.config_base import RelationConfigBase
from dbt.dataclass_schema import StrEnum
class RelationConfigChangeAction(StrEnum):
alter = "alter"
create = "create"
drop = "drop"
@dataclass(frozen=True, eq=True, unsafe_hash=True)
class RelationConfigChange(RelationConfigBase, ABC):
action: RelationConfigChangeAction
context: Hashable # this is usually a RelationConfig, e.g. IndexConfig, but shouldn't be limited
@property
@abstractmethod
def requires_full_refresh(self) -> bool:
raise self._not_implemented_error()

View File

@@ -1,57 +0,0 @@
from dataclasses import dataclass
from typing import Set, Optional
from dbt.exceptions import DbtRuntimeError
@dataclass(frozen=True, eq=True, unsafe_hash=True)
class RelationConfigValidationRule:
validation_check: bool
validation_error: Optional[DbtRuntimeError]
@property
def default_error(self):
return DbtRuntimeError(
"There was a validation error in preparing this relation config."
"No additional context was provided by this adapter."
)
@dataclass(frozen=True)
class RelationConfigValidationMixin:
def __post_init__(self):
self.run_validation_rules()
@property
def validation_rules(self) -> Set[RelationConfigValidationRule]:
"""
A set of validation rules to run against the object upon creation.
A validation rule is a combination of a validation check (bool) and an optional error message.
This defaults to no validation rules if not implemented. It's recommended to override this with values,
but that may not always be necessary.
Returns: a set of validation rules
"""
return set()
def run_validation_rules(self):
for validation_rule in self.validation_rules:
try:
assert validation_rule.validation_check
except AssertionError:
if validation_rule.validation_error:
raise validation_rule.validation_error
else:
raise validation_rule.default_error
self.run_child_validation_rules()
def run_child_validation_rules(self):
for attr_value in vars(self).values():
if hasattr(attr_value, "validation_rules"):
attr_value.run_validation_rules()
if isinstance(attr_value, set):
for member in attr_value:
if hasattr(member, "validation_rules"):
member.run_validation_rules()

View File

@@ -52,6 +52,7 @@ class SQLConnectionManager(BaseConnectionManager):
bindings: Optional[Any] = None,
abridge_sql_log: bool = False,
) -> Tuple[Connection, Any]:
connection = self.get_thread_connection()
if auto_begin and connection.transaction_open is False:
self.begin()
@@ -117,16 +118,13 @@ class SQLConnectionManager(BaseConnectionManager):
return [dict(zip(column_names, row)) for row in rows]
@classmethod
def get_result_from_cursor(cls, cursor: Any, limit: Optional[int]) -> agate.Table:
def get_result_from_cursor(cls, cursor: Any) -> agate.Table:
data: List[Any] = []
column_names: List[str] = []
if cursor.description is not None:
column_names = [col[0] for col in cursor.description]
if limit:
rows = cursor.fetchmany(limit)
else:
rows = cursor.fetchall()
rows = cursor.fetchall()
data = cls.process_results(column_names, rows)
return dbt.clients.agate_helper.table_from_data_flat(data, column_names)
@@ -140,13 +138,13 @@ class SQLConnectionManager(BaseConnectionManager):
)
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
self, sql: str, auto_begin: bool = False, fetch: bool = False
) -> Tuple[AdapterResponse, agate.Table]:
sql = self._add_query_comment(sql)
_, cursor = self.add_query(sql, auto_begin)
response = self.get_response(cursor)
if fetch:
table = self.get_result_from_cursor(cursor, limit)
table = self.get_result_from_cursor(cursor)
else:
table = dbt.clients.agate_helper.empty_table()
return response, table

View File

@@ -1,7 +1,7 @@
import agate
from typing import Any, Optional, Tuple, Type, List
from dbt.contracts.connection import Connection, AdapterResponse
from dbt.contracts.connection import Connection
from dbt.exceptions import RelationTypeNullError
from dbt.adapters.base import BaseAdapter, available
from dbt.adapters.cache import _make_ref_key_dict
@@ -22,7 +22,6 @@ RENAME_RELATION_MACRO_NAME = "rename_relation"
TRUNCATE_RELATION_MACRO_NAME = "truncate_relation"
DROP_RELATION_MACRO_NAME = "drop_relation"
ALTER_COLUMN_TYPE_MACRO_NAME = "alter_column_type"
VALIDATE_SQL_MACRO_NAME = "validate_sql"
class SQLAdapter(BaseAdapter):
@@ -219,34 +218,6 @@ class SQLAdapter(BaseAdapter):
results = self.execute_macro(CHECK_SCHEMA_EXISTS_MACRO_NAME, kwargs=kwargs)
return results[0][0] > 0
def validate_sql(self, sql: str) -> AdapterResponse:
"""Submit the given SQL to the engine for validation, but not execution.
By default we simply prefix the query with the explain keyword and allow the
exceptions thrown by the underlying engine on invalid SQL inputs to bubble up
to the exception handler. For adjustments to the explain statement - such as
for adapters that have different mechanisms for hinting at query validation
or dry-run - callers may be able to override the validate_sql_query macro with
the addition of an <adapter>__validate_sql implementation.
:param sql str: The sql to validate
"""
kwargs = {
"sql": sql,
}
result = self.execute_macro(VALIDATE_SQL_MACRO_NAME, kwargs=kwargs)
# The statement macro always returns an AdapterResponse in the output AttrDict's
# `response` property, and we preserve the full payload in case we want to
# return fetched output for engines where explain plans are emitted as columnar
# results. Any macro override that deviates from this behavior may encounter an
# assertion error in the runtime.
adapter_response = result.response # type: ignore[attr-defined]
assert isinstance(adapter_response, AdapterResponse), (
f"Expected AdapterResponse from validate_sql macro execution, "
f"got {type(adapter_response)}."
)
return adapter_response
# This is for use in the test suite
def run_sql_for_tests(self, sql, fetch, conn):
cursor = conn.handle.cursor()

View File

@@ -1,25 +1,3 @@
# Adding a new command
## `main.py`
Add the new command with all necessary decorators. Every command will need at minimum:
- a decorator for the click group it belongs to which also names the command
- the postflight decorator (must come before other decorators from the `requires` module for error handling)
- the preflight decorator
```py
@cli.command("my-new-command")
@requires.postflight
@requires.preflight
def my_new_command(ctx, **kwargs):
...
```
## `types.py`
Add an entry to the `Command` enum with your new command. Commands that are sub-commands should have entries
that represent their full command path (e.g. `source freshness -> SOURCE_FRESHNESS`, `docs serve -> DOCS_SERVE`).
## `flags.py`
Add the new command to the dictionary within the `command_args` function.
# Exception Handling
## `requires.py`

View File

@@ -4,16 +4,14 @@ from dataclasses import dataclass
from importlib import import_module
from multiprocessing import get_context
from pprint import pformat as pf
from typing import Any, Callable, Dict, List, Optional, Set, Union
from typing import Callable, Dict, List, Set, Union
from click import Context, get_current_context, Parameter
from click.core import Command as ClickCommand, Group, ParameterSource
from click import Context, get_current_context
from click.core import Command, Group, ParameterSource
from dbt.cli.exceptions import DbtUsageException
from dbt.cli.resolvers import default_log_path, default_project_dir
from dbt.cli.types import Command as CliCommand
from dbt.config.profile import read_user_config
from dbt.contracts.project import UserConfig
from dbt.exceptions import DbtInternalError
from dbt.deprecations import renamed_env_var
from dbt.helper_types import WarnErrorOptions
@@ -39,9 +37,6 @@ DEPRECATED_PARAMS = {
}
WHICH_KEY = "which"
def convert_config(config_name, config_value):
"""Convert the values from config and original set_from_args to the correct type."""
ret = config_value
@@ -57,16 +52,16 @@ def args_to_context(args: List[str]) -> Context:
from dbt.cli.main import cli
cli_ctx = cli.make_context(cli.name, args)
# Split args if they're a comma seperated string.
# Split args if they're a comma separated string.
if len(args) == 1 and "," in args[0]:
args = args[0].split(",")
sub_command_name, sub_command, args = cli.resolve_command(cli_ctx, args)
# Handle source and docs group.
if isinstance(sub_command, Group):
if type(sub_command) == Group:
sub_command_name, sub_command, args = sub_command.resolve_command(cli_ctx, args)
assert isinstance(sub_command, ClickCommand)
assert type(sub_command) == Command
sub_command_ctx = sub_command.make_context(sub_command_name, args)
sub_command_ctx.parent = cli_ctx
return sub_command_ctx
@@ -76,9 +71,7 @@ def args_to_context(args: List[str]) -> Context:
class Flags:
"""Primary configuration artifact for running dbt"""
def __init__(
self, ctx: Optional[Context] = None, user_config: Optional[UserConfig] = None
) -> None:
def __init__(self, ctx: Context = None, user_config: UserConfig = None) -> None:
# Set the default flags.
for key, value in FLAGS_DEFAULTS.items():
@@ -206,9 +199,6 @@ class Flags:
profiles_dir = getattr(self, "PROFILES_DIR", None)
user_config = read_user_config(profiles_dir) if profiles_dir else None
# Add entire invocation command to flags
object.__setattr__(self, "INVOCATION_COMMAND", "dbt " + " ".join(sys.argv[1:]))
# Overwrite default assignments with user config if available.
if user_config:
param_assigned_from_default_copy = params_assigned_from_default.copy()
@@ -287,118 +277,3 @@ class Flags:
# It is necessary to remove this attr from the class so it does
# not get pickled when written to disk as json.
object.__delattr__(self, "deprecated_env_var_warnings")
@classmethod
def from_dict(cls, command: CliCommand, args_dict: Dict[str, Any]) -> "Flags":
command_arg_list = command_params(command, args_dict)
ctx = args_to_context(command_arg_list)
flags = cls(ctx=ctx)
flags.fire_deprecations()
return flags
CommandParams = List[str]
def command_params(command: CliCommand, args_dict: Dict[str, Any]) -> CommandParams:
"""Given a command and a dict, returns a list of strings representing
the CLI params for that command. The order of this list is consistent with
which flags are expected at the parent level vs the command level.
e.g. fn("run", {"defer": True, "print": False}) -> ["--no-print", "run", "--defer"]
The result of this function can be passed in to the args_to_context function
to produce a click context to instantiate Flags with.
"""
cmd_args = set(command_args(command))
prnt_args = set(parent_args())
default_args = set([x.lower() for x in FLAGS_DEFAULTS.keys()])
res = command.to_list()
for k, v in args_dict.items():
k = k.lower()
# if a "which" value exists in the args dict, it should match the command provided
if k == WHICH_KEY:
if v != command.value:
raise DbtInternalError(
f"Command '{command.value}' does not match value of which: '{v}'"
)
continue
# param was assigned from defaults and should not be included
if k not in (cmd_args | prnt_args) - default_args:
continue
# if the param is in parent args, it should come before the arg name
# e.g. ["--print", "run"] vs ["run", "--print"]
add_fn = res.append
if k in prnt_args:
def add_fn(x):
res.insert(0, x)
spinal_cased = k.replace("_", "-")
if k == "macro" and command == CliCommand.RUN_OPERATION:
add_fn(v)
elif v in (None, False):
add_fn(f"--no-{spinal_cased}")
elif v is True:
add_fn(f"--{spinal_cased}")
else:
add_fn(f"--{spinal_cased}={v}")
return res
ArgsList = List[str]
def parent_args() -> ArgsList:
"""Return a list representing the params the base click command takes."""
from dbt.cli.main import cli
return format_params(cli.params)
def command_args(command: CliCommand) -> ArgsList:
"""Given a command, return a list of strings representing the params
that command takes. This function only returns params assigned to a
specific command, not those of its parent command.
e.g. fn("run") -> ["defer", "favor_state", "exclude", ...]
"""
import dbt.cli.main as cli
CMD_DICT: Dict[CliCommand, ClickCommand] = {
CliCommand.BUILD: cli.build,
CliCommand.CLEAN: cli.clean,
CliCommand.CLONE: cli.clone,
CliCommand.COMPILE: cli.compile,
CliCommand.DOCS_GENERATE: cli.docs_generate,
CliCommand.DOCS_SERVE: cli.docs_serve,
CliCommand.DEBUG: cli.debug,
CliCommand.DEPS: cli.deps,
CliCommand.INIT: cli.init,
CliCommand.LIST: cli.list,
CliCommand.PARSE: cli.parse,
CliCommand.RUN: cli.run,
CliCommand.RUN_OPERATION: cli.run_operation,
CliCommand.SEED: cli.seed,
CliCommand.SHOW: cli.show,
CliCommand.SNAPSHOT: cli.snapshot,
CliCommand.SOURCE_FRESHNESS: cli.freshness,
CliCommand.TEST: cli.test,
CliCommand.RETRY: cli.retry,
}
click_cmd: Optional[ClickCommand] = CMD_DICT.get(command, None)
if click_cmd is None:
raise DbtInternalError(f"No command found for name '{command.name}'")
return format_params(click_cmd.params)
def format_params(params: List[Parameter]) -> ArgsList:
return [str(x.name) for x in params if not str(x.name).lower().startswith("deprecated_")]

View File

@@ -1,3 +1,4 @@
import functools
from copy import copy
from dataclasses import dataclass
from typing import Callable, List, Optional, Union
@@ -19,11 +20,11 @@ from dbt.contracts.graph.manifest import Manifest
from dbt.contracts.results import (
CatalogArtifact,
RunExecutionResult,
RunOperationResultsArtifact,
)
from dbt.events.base_types import EventMsg
from dbt.task.build import BuildTask
from dbt.task.clean import CleanTask
from dbt.task.clone import CloneTask
from dbt.task.compile import CompileTask
from dbt.task.debug import DebugTask
from dbt.task.deps import DepsTask
@@ -31,7 +32,6 @@ from dbt.task.freshness import FreshnessTask
from dbt.task.generate import GenerateTask
from dbt.task.init import InitTask
from dbt.task.list import ListTask
from dbt.task.retry import RetryTask
from dbt.task.run import RunTask
from dbt.task.run_operation import RunOperationTask
from dbt.task.seed import SeedTask
@@ -54,7 +54,8 @@ class dbtRunnerResult:
List[str], # list/ls
Manifest, # parse
None, # clean, deps, init, source
RunExecutionResult, # build, compile, run, seed, snapshot, test, run-operation
RunExecutionResult, # build, compile, run, seed, snapshot, test
RunOperationResultsArtifact, # run-operation
] = None
@@ -62,8 +63,8 @@ class dbtRunnerResult:
class dbtRunner:
def __init__(
self,
manifest: Optional[Manifest] = None,
callbacks: Optional[List[Callable[[EventMsg], None]]] = None,
manifest: Manifest = None,
callbacks: List[Callable[[EventMsg], None]] = None,
):
self.manifest = manifest
@@ -118,6 +119,42 @@ class dbtRunner:
)
# approach from https://github.com/pallets/click/issues/108#issuecomment-280489786
def global_flags(func):
@p.cache_selected_only
@p.debug
@p.deprecated_print
@p.enable_legacy_logger
@p.fail_fast
@p.log_cache_events
@p.log_file_max_bytes
@p.log_format_file
@p.log_level
@p.log_level_file
@p.log_path
@p.macro_debugging
@p.partial_parse
@p.populate_cache
@p.print
@p.printer_width
@p.quiet
@p.record_timing_info
@p.send_anonymous_usage_stats
@p.single_threaded
@p.static_parser
@p.use_colors
@p.use_colors_file
@p.use_experimental_parser
@p.version
@p.version_check
@p.write_json
@functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
# dbt
@click.group(
context_settings={"help_option_names": ["-h", "--help"]},
@@ -126,36 +163,10 @@ class dbtRunner:
epilog="Specify one of these sub-commands and you can find more help from there.",
)
@click.pass_context
@p.cache_selected_only
@p.debug
@p.deprecated_print
@p.enable_legacy_logger
@p.fail_fast
@p.log_cache_events
@p.log_format
@p.log_format_file
@p.log_level
@p.log_level_file
@p.log_path
@p.macro_debugging
@p.partial_parse
@p.partial_parse_file_path
@p.populate_cache
@p.print
@p.printer_width
@p.quiet
@p.record_timing_info
@p.send_anonymous_usage_stats
@p.single_threaded
@p.static_parser
@p.use_colors
@p.use_colors_file
@p.use_experimental_parser
@p.version
@p.version_check
@global_flags
@p.warn_error
@p.warn_error_options
@p.write_json
@p.log_format
def cli(ctx, **kwargs):
"""An ELT tool for managing your SQL transformations and data models.
For more documentation on these commands, visit: docs.getdbt.com
@@ -165,10 +176,10 @@ def cli(ctx, **kwargs):
# dbt build
@cli.command("build")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@p.fail_fast
@p.favor_state
@p.deprecated_favor_state
@p.full_refresh
@@ -181,14 +192,12 @@ def cli(ctx, **kwargs):
@p.selector
@p.show
@p.state
@p.defer_state
@p.deprecated_state
@p.store_failures
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -211,6 +220,7 @@ def build(ctx, **kwargs):
# dbt clean
@cli.command("clean")
@click.pass_context
@global_flags
@p.profile
@p.profiles_dir
@p.project_dir
@@ -233,6 +243,7 @@ def clean(ctx, **kwargs):
# dbt docs
@cli.group()
@click.pass_context
@global_flags
def docs(ctx, **kwargs):
"""Generate or serve the documentation website for your project"""
@@ -240,6 +251,7 @@ def docs(ctx, **kwargs):
# dbt docs generate
@docs.command("generate")
@click.pass_context
@global_flags
@p.compile_docs
@p.defer
@p.deprecated_defer
@@ -251,15 +263,12 @@ def docs(ctx, **kwargs):
@p.project_dir
@p.select
@p.selector
@p.empty_catalog
@p.state
@p.defer_state
@p.deprecated_state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -282,6 +291,7 @@ def docs_generate(ctx, **kwargs):
# dbt docs serve
@docs.command("serve")
@click.pass_context
@global_flags
@p.browser
@p.port
@p.profile
@@ -310,6 +320,7 @@ def docs_serve(ctx, **kwargs):
# dbt compile
@cli.command("compile")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@@ -326,13 +337,11 @@ def docs_serve(ctx, **kwargs):
@p.selector
@p.inline
@p.state
@p.defer_state
@p.deprecated_state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -356,6 +365,7 @@ def compile(ctx, **kwargs):
# dbt show
@cli.command("show")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@@ -373,13 +383,11 @@ def compile(ctx, **kwargs):
@p.selector
@p.inline
@p.state
@p.defer_state
@p.deprecated_state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -403,18 +411,17 @@ def show(ctx, **kwargs):
# dbt debug
@cli.command("debug")
@click.pass_context
@p.debug_connection
@global_flags
@p.config_dir
@p.profile
@p.profiles_dir_exists_false
@p.project_dir
@p.target
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
def debug(ctx, **kwargs):
"""Show information on the current dbt environment and check dependencies, then test the database connection. Not to be confused with the --debug option which increases verbosity."""
"""Test the database connection and show information for debugging purposes. Not to be confused with the --debug option which increases verbosity."""
task = DebugTask(
ctx.obj["flags"],
@@ -429,6 +436,7 @@ def debug(ctx, **kwargs):
# dbt deps
@cli.command("deps")
@click.pass_context
@global_flags
@p.profile
@p.profiles_dir_exists_false
@p.project_dir
@@ -449,6 +457,7 @@ def deps(ctx, **kwargs):
# dbt init
@cli.command("init")
@click.pass_context
@global_flags
# for backwards compatibility, accept 'project_name' as an optional positional argument
@click.argument("project_name", required=False)
@p.profile
@@ -471,6 +480,7 @@ def init(ctx, **kwargs):
# dbt list
@cli.command("list")
@click.pass_context
@global_flags
@p.exclude
@p.indirect_selection
@p.models
@@ -483,7 +493,6 @@ def init(ctx, **kwargs):
@p.raw_select
@p.selector
@p.state
@p.defer_state
@p.deprecated_state
@p.target
@p.target_path
@@ -516,6 +525,7 @@ cli.add_command(ls, "ls")
# dbt parse
@cli.command("parse")
@click.pass_context
@global_flags
@p.profile
@p.profiles_dir
@p.project_dir
@@ -523,7 +533,6 @@ cli.add_command(ls, "ls")
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -540,12 +549,12 @@ def parse(ctx, **kwargs):
# dbt run
@cli.command("run")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.favor_state
@p.deprecated_favor_state
@p.exclude
@p.fail_fast
@p.full_refresh
@p.profile
@p.profiles_dir
@@ -553,13 +562,11 @@ def parse(ctx, **kwargs):
@p.select
@p.selector
@p.state
@p.defer_state
@p.deprecated_state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -579,76 +586,10 @@ def run(ctx, **kwargs):
return results, success
# dbt retry
@cli.command("retry")
@click.pass_context
@p.project_dir
@p.profiles_dir
@p.vars
@p.profile
@p.target
@p.state
@p.threads
@p.fail_fast
@requires.postflight
@requires.preflight
@requires.profile
@requires.project
@requires.runtime_config
@requires.manifest
def retry(ctx, **kwargs):
"""Retry the nodes that failed in the previous run."""
task = RetryTask(
ctx.obj["flags"],
ctx.obj["runtime_config"],
ctx.obj["manifest"],
)
results = task.run()
success = task.interpret_results(results)
return results, success
# dbt clone
@cli.command("clone")
@click.pass_context
@p.defer_state
@p.exclude
@p.full_refresh
@p.profile
@p.profiles_dir
@p.project_dir
@p.resource_type
@p.select
@p.selector
@p.state # required
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.preflight
@requires.profile
@requires.project
@requires.runtime_config
@requires.manifest
@requires.postflight
def clone(ctx, **kwargs):
"""Create clones of selected nodes based on their location in the manifest provided to --state."""
task = CloneTask(
ctx.obj["flags"],
ctx.obj["runtime_config"],
ctx.obj["manifest"],
)
results = task.run()
success = task.interpret_results(results)
return results, success
# dbt run operation
@cli.command("run-operation")
@click.pass_context
@global_flags
@click.argument("macro")
@p.args
@p.profile
@@ -656,7 +597,6 @@ def clone(ctx, **kwargs):
@p.project_dir
@p.target
@p.target_path
@p.threads
@p.vars
@requires.postflight
@requires.preflight
@@ -680,6 +620,7 @@ def run_operation(ctx, **kwargs):
# dbt seed
@cli.command("seed")
@click.pass_context
@global_flags
@p.exclude
@p.full_refresh
@p.profile
@@ -689,13 +630,11 @@ def run_operation(ctx, **kwargs):
@p.selector
@p.show
@p.state
@p.defer_state
@p.deprecated_state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -717,6 +656,7 @@ def seed(ctx, **kwargs):
# dbt snapshot
@cli.command("snapshot")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@@ -728,7 +668,6 @@ def seed(ctx, **kwargs):
@p.select
@p.selector
@p.state
@p.defer_state
@p.deprecated_state
@p.target
@p.target_path
@@ -756,6 +695,7 @@ def snapshot(ctx, **kwargs):
# dbt source
@cli.group()
@click.pass_context
@global_flags
def source(ctx, **kwargs):
"""Manage your project's sources"""
@@ -763,6 +703,7 @@ def source(ctx, **kwargs):
# dbt source freshness
@source.command("freshness")
@click.pass_context
@global_flags
@p.exclude
@p.output_path # TODO: Is this ok to re-use? We have three different output params, how much can we consolidate?
@p.profile
@@ -771,7 +712,6 @@ def source(ctx, **kwargs):
@p.select
@p.selector
@p.state
@p.defer_state
@p.deprecated_state
@p.target
@p.target_path
@@ -805,10 +745,10 @@ cli.commands["source"].add_command(snapshot_freshness, "snapshot-freshness") #
# dbt test
@cli.command("test")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@p.fail_fast
@p.favor_state
@p.deprecated_favor_state
@p.indirect_selection
@@ -818,14 +758,12 @@ cli.commands["source"].add_command(snapshot_freshness, "snapshot-freshness") #
@p.select
@p.selector
@p.state
@p.defer_state
@p.deprecated_state
@p.store_failures
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile

View File

@@ -2,6 +2,7 @@ import click
import inspect
import typing as t
from click import Context
from click.parser import OptionParser, ParsingState
from dbt.cli.option_types import ChoiceTuple
@@ -13,8 +14,9 @@ class MultiOption(click.Option):
nargs = kwargs.pop("nargs", -1)
assert nargs == -1, "nargs, if set, must be -1 not {}".format(nargs)
super(MultiOption, self).__init__(*args, **kwargs)
self._previous_parser_process = None
self._eat_all_parser = None
# this makes mypy happy, setting these to None causes mypy failures
self._previous_parser_process = lambda *args, **kwargs: None
self._eat_all_parser = lambda *args, **kwargs: None
# validate that multiple=True
multiple = kwargs.pop("multiple", None)
@@ -29,34 +31,35 @@ class MultiOption(click.Option):
else:
assert isinstance(option_type, ChoiceTuple), msg
def add_to_parser(self, parser, ctx):
def parser_process(value, state):
def add_to_parser(self, parser: OptionParser, ctx: Context):
def parser_process(value: str, state: ParsingState):
# method to hook to the parser.process
done = False
value = [value]
value_list = str.split(value, " ")
if self.save_other_options:
# grab everything up to the next option
while state.rargs and not done:
for prefix in self._eat_all_parser.prefixes:
for prefix in self._eat_all_parser.prefixes: # type: ignore[attr-defined]
if state.rargs[0].startswith(prefix):
done = True
if not done:
value.append(state.rargs.pop(0))
value_list.append(state.rargs.pop(0))
else:
# grab everything remaining
value += state.rargs
value_list += state.rargs
state.rargs[:] = []
value = tuple(value)
value_tuple = tuple(value_list)
# call the actual process
self._previous_parser_process(value, state)
self._previous_parser_process(value_tuple, state)
retval = super(MultiOption, self).add_to_parser(parser, ctx)
for name in self.opts:
our_parser = parser._long_opt.get(name) or parser._short_opt.get(name)
if our_parser:
self._eat_all_parser = our_parser
self._eat_all_parser = our_parser # type: ignore[assignment]
self._previous_parser_process = our_parser.process
our_parser.process = parser_process
# mypy doesnt like assingment to a method see https://github.com/python/mypy/issues/708
our_parser.process = parser_process # type: ignore[assignment]
break
return retval

View File

@@ -43,7 +43,7 @@ compile_docs = click.option(
config_dir = click.option(
"--config-dir",
envvar=None,
help="Print a system-specific command to access the directory that the current dbt project is searching for a profiles.yml. Then, exit. This flag renders other debug step flags no-ops.",
help="Show the configured location for the profiles.yml file and exit",
is_flag=True,
)
@@ -171,6 +171,15 @@ use_colors_file = click.option(
default=True,
)
log_file_max_bytes = click.option(
"--log-file-max-bytes",
envvar="DBT_LOG_FILE_MAX_BYTES",
help="Configure the max file size in bytes for a single dbt.log file, before rolling over. 0 means no limit.",
default=10 * 1024 * 1024, # 10mb
type=click.INT,
hidden=True,
)
log_path = click.option(
"--log-path",
envvar="DBT_LOG_PATH",
@@ -239,15 +248,6 @@ partial_parse = click.option(
default=True,
)
partial_parse_file_path = click.option(
"--partial-parse-file-path",
envvar="DBT_PARTIAL_PARSE_FILE_PATH",
help="Internal flag for path to partial_parse.manifest file.",
default=None,
hidden=True,
type=click.Path(exists=True, dir_okay=False, resolve_path=True),
)
populate_cache = click.option(
"--populate-cache/--no-populate-cache",
envvar="DBT_POPULATE_CACHE",
@@ -380,9 +380,9 @@ inline = click.option(
# Most CLI arguments should use the combined `select` option that aliases `--models` to `--select`.
# However, if you need to split out these separators (like `dbt ls`), use the `models` and `raw_select` options instead.
# See https://github.com/dbt-labs/dbt-core/pull/6774#issuecomment-1408476095 for more info.
models = click.option(*model_decls, **select_attrs)
raw_select = click.option(*select_decls, **select_attrs)
select = click.option(*select_decls, *model_decls, **select_attrs)
models = click.option(*model_decls, **select_attrs) # type: ignore[arg-type]
raw_select = click.option(*select_decls, **select_attrs) # type: ignore[arg-type]
select = click.option(*select_decls, *model_decls, **select_attrs) # type: ignore[arg-type]
selector = click.option(
"--selector",
@@ -425,30 +425,10 @@ skip_profile_setup = click.option(
is_flag=True,
)
empty_catalog = click.option(
"--empty-catalog",
help="If specified, generate empty catalog.json file during the `dbt docs generate` command.",
default=False,
is_flag=True,
)
state = click.option(
"--state",
envvar="DBT_STATE",
help="Unless overridden, use this state directory for both state comparison and deferral.",
type=click.Path(
dir_okay=True,
file_okay=False,
readable=True,
resolve_path=False,
path_type=Path,
),
)
defer_state = click.option(
"--defer-state",
envvar="DBT_DEFER_STATE",
help="Override the state directory for deferral only.",
help="If set, use the given directory as the source for JSON files to compare with this project.",
type=click.Path(
dir_okay=True,
file_okay=False,
@@ -500,13 +480,6 @@ target_path = click.option(
type=click.Path(),
)
debug_connection = click.option(
"--connection",
envvar=None,
help="Test the connection to the target database independent of dependency checks.",
is_flag=True,
)
threads = click.option(
"--threads",
envvar=None,

View File

@@ -23,7 +23,6 @@ from dbt.parser.manifest import ManifestLoader, write_manifest
from dbt.profiler import profiler
from dbt.tracking import active_user, initialize_from_flags, track_run
from dbt.utils import cast_dict_to_dict_of_strings
from dbt.plugins import set_up_plugin_manager, get_plugin_manager
from click import Context
from functools import update_wrapper
@@ -161,9 +160,6 @@ def project(func):
)
ctx.obj["project"] = project
# Plugins
set_up_plugin_manager(project_name=project.project_name)
if dbt.tracking.active_user is not None:
project_id = None if project is None else project.hashed_name()
@@ -244,17 +240,12 @@ def manifest(*args0, write=True, write_perf_info=False):
# a manifest has already been set on the context, so don't overwrite it
if ctx.obj.get("manifest") is None:
manifest = ManifestLoader.get_full_manifest(
runtime_config,
write_perf_info=write_perf_info,
runtime_config, write_perf_info=write_perf_info
)
ctx.obj["manifest"] = manifest
if write and ctx.obj["flags"].write_json:
write_manifest(manifest, runtime_config.project_target_path)
pm = get_plugin_manager(runtime_config.project_name)
plugin_artifacts = pm.get_manifest_artifacts(manifest)
for path, plugin_artifact in plugin_artifacts.items():
plugin_artifact.write(path)
write_manifest(manifest, ctx.obj["runtime_config"].project_target_path)
return func(*args, **kwargs)

View File

@@ -1,40 +0,0 @@
from enum import Enum
from typing import List
from dbt.exceptions import DbtInternalError
class Command(Enum):
BUILD = "build"
CLEAN = "clean"
COMPILE = "compile"
CLONE = "clone"
DOCS_GENERATE = "generate"
DOCS_SERVE = "serve"
DEBUG = "debug"
DEPS = "deps"
INIT = "init"
LIST = "list"
PARSE = "parse"
RUN = "run"
RUN_OPERATION = "run-operation"
SEED = "seed"
SHOW = "show"
SNAPSHOT = "snapshot"
SOURCE_FRESHNESS = "freshness"
TEST = "test"
RETRY = "retry"
@classmethod
def from_str(cls, s: str) -> "Command":
try:
return cls(s)
except ValueError:
raise DbtInternalError(f"No value '{s}' exists in Command enum")
def to_list(self) -> List[str]:
return {
Command.DOCS_GENERATE: ["docs", "generate"],
Command.DOCS_SERVE: ["docs", "serve"],
Command.SOURCE_FRESHNESS: ["source", "freshness"],
}.get(self, [self.value])

View File

@@ -565,8 +565,6 @@ def _requote_result(raw_value: str, rendered: str) -> str:
# is small enough that I've just chosen the more readable option.
_HAS_RENDER_CHARS_PAT = re.compile(r"({[{%#]|[#}%]})")
_render_cache: Dict[str, Any] = dict()
def get_rendered(
string: str,
@@ -574,21 +572,15 @@ def get_rendered(
node=None,
capture_macros: bool = False,
native: bool = False,
) -> Any:
) -> str:
# performance optimization: if there are no jinja control characters in the
# string, we can just return the input. Fall back to jinja if the type is
# not a string or if native rendering is enabled (so '1' -> 1, etc...)
# If this is desirable in the native env as well, we could handle the
# native=True case by passing the input string to ast.literal_eval, like
# the native renderer does.
has_render_chars = not isinstance(string, str) or _HAS_RENDER_CHARS_PAT.search(string)
if not has_render_chars:
if not native:
return string
elif string in _render_cache:
return _render_cache[string]
if not native and isinstance(string, str) and _HAS_RENDER_CHARS_PAT.search(string) is None:
return string
template = get_template(
string,
ctx,
@@ -596,13 +588,7 @@ def get_rendered(
capture_macros=capture_macros,
native=native,
)
rendered = render_template(template, ctx, node)
if not has_render_chars and native:
_render_cache[string] = rendered
return rendered
return render_template(template, ctx, node)
def undefined_error(msg) -> NoReturn:

View File

@@ -141,7 +141,7 @@ def statically_parse_adapter_dispatch(func_call, ctx, db_wrapper):
macro = db_wrapper.dispatch(func_name, macro_namespace=macro_namespace).macro
func_name = f"{macro.package_name}.{macro.name}"
possible_macro_calls.append(func_name)
else: # this is only for tests/unit/test_macro_calls.py
else: # this is only for test/unit/test_macro_calls.py
if macro_namespace:
packages = [macro_namespace]
else:

View File

@@ -1,6 +1,4 @@
import argparse
import json
import networkx as nx # type: ignore
import os
import pickle
@@ -29,8 +27,8 @@ from dbt.exceptions import (
DbtRuntimeError,
)
from dbt.graph import Graph
from dbt.events.functions import fire_event, get_invocation_id
from dbt.events.types import FoundStats, Note, WritingInjectedSQLForNode
from dbt.events.functions import fire_event
from dbt.events.types import FoundStats, WritingInjectedSQLForNode
from dbt.events.contextvars import get_node_info
from dbt.node_types import NodeType, ModelLanguage
from dbt.events.format import pluralize
@@ -48,10 +46,9 @@ def print_compile_stats(stats):
NodeType.Analysis: "analysis",
NodeType.Macro: "macro",
NodeType.Operation: "operation",
NodeType.Seed: "seed",
NodeType.Seed: "seed file",
NodeType.Source: "source",
NodeType.Exposure: "exposure",
NodeType.SemanticModel: "semantic model",
NodeType.Metric: "metric",
NodeType.Group: "group",
}
@@ -64,8 +61,7 @@ def print_compile_stats(stats):
resource_counts = {k.pluralize(): v for k, v in results.items()}
dbt.tracking.track_resource_counts(resource_counts)
# do not include resource types that are not actually defined in the project
stat_line = ", ".join([pluralize(ct, names.get(t)) for t, ct in stats.items() if t in names])
stat_line = ", ".join([pluralize(ct, names.get(t)) for t, ct in results.items() if t in names])
fire_event(FoundStats(stat_line=stat_line))
@@ -84,16 +80,16 @@ def _generate_stats(manifest: Manifest):
if _node_enabled(node):
stats[node.resource_type] += 1
# Disabled nodes don't appear in the following collections, so we don't check.
stats[NodeType.Source] += len(manifest.sources)
stats[NodeType.Exposure] += len(manifest.exposures)
stats[NodeType.Metric] += len(manifest.metrics)
stats[NodeType.Macro] += len(manifest.macros)
stats[NodeType.Group] += len(manifest.groups)
stats[NodeType.SemanticModel] += len(manifest.semantic_models)
# TODO: should we be counting dimensions + entities?
for source in manifest.sources.values():
stats[source.resource_type] += 1
for exposure in manifest.exposures.values():
stats[exposure.resource_type] += 1
for metric in manifest.metrics.values():
stats[metric.resource_type] += 1
for macro in manifest.macros.values():
stats[macro.resource_type] += 1
for group in manifest.groups.values():
stats[group.resource_type] += 1
return stats
@@ -165,113 +161,6 @@ class Linker:
with open(outfile, "wb") as outfh:
pickle.dump(out_graph, outfh, protocol=pickle.HIGHEST_PROTOCOL)
def link_node(self, node: GraphMemberNode, manifest: Manifest):
self.add_node(node.unique_id)
for dependency in node.depends_on_nodes:
if dependency in manifest.nodes:
self.dependency(node.unique_id, (manifest.nodes[dependency].unique_id))
elif dependency in manifest.sources:
self.dependency(node.unique_id, (manifest.sources[dependency].unique_id))
elif dependency in manifest.metrics:
self.dependency(node.unique_id, (manifest.metrics[dependency].unique_id))
elif dependency in manifest.semantic_models:
self.dependency(node.unique_id, (manifest.semantic_models[dependency].unique_id))
else:
raise GraphDependencyNotFoundError(node, dependency)
def link_graph(self, manifest: Manifest):
for source in manifest.sources.values():
self.add_node(source.unique_id)
for semantic_model in manifest.semantic_models.values():
self.add_node(semantic_model.unique_id)
for node in manifest.nodes.values():
self.link_node(node, manifest)
for exposure in manifest.exposures.values():
self.link_node(exposure, manifest)
for metric in manifest.metrics.values():
self.link_node(metric, manifest)
cycle = self.find_cycles()
if cycle:
raise RuntimeError("Found a cycle: {}".format(cycle))
def add_test_edges(self, manifest: Manifest) -> None:
"""This method adds additional edges to the DAG. For a given non-test
executable node, add an edge from an upstream test to the given node if
the set of nodes the test depends on is a subset of the upstream nodes
for the given node."""
# Given a graph:
# model1 --> model2 --> model3
# | |
# | \/
# \/ test 2
# test1
#
# Produce the following graph:
# model1 --> model2 --> model3
# | /\ | /\ /\
# | | \/ | |
# \/ | test2 ----| |
# test1 ----|---------------|
for node_id in self.graph:
# If node is executable (in manifest.nodes) and does _not_
# represent a test, continue.
if (
node_id in manifest.nodes
and manifest.nodes[node_id].resource_type != NodeType.Test
):
# Get *everything* upstream of the node
all_upstream_nodes = nx.traversal.bfs_tree(self.graph, node_id, reverse=True)
# Get the set of upstream nodes not including the current node.
upstream_nodes = set([n for n in all_upstream_nodes if n != node_id])
# Get all tests that depend on any upstream nodes.
upstream_tests = []
for upstream_node in upstream_nodes:
upstream_tests += _get_tests_for_node(manifest, upstream_node)
for upstream_test in upstream_tests:
# Get the set of all nodes that the test depends on
# including the upstream_node itself. This is necessary
# because tests can depend on multiple nodes (ex:
# relationship tests). Test nodes do not distinguish
# between what node the test is "testing" and what
# node(s) it depends on.
test_depends_on = set(manifest.nodes[upstream_test].depends_on_nodes)
# If the set of nodes that an upstream test depends on
# is a subset of all upstream nodes of the current node,
# add an edge from the upstream test to the current node.
if test_depends_on.issubset(upstream_nodes):
self.graph.add_edge(upstream_test, node_id, edge_type="parent_test")
def get_graph(self, manifest: Manifest) -> Graph:
self.link_graph(manifest)
return Graph(self.graph)
def get_graph_summary(self, manifest: Manifest) -> Dict[int, Dict[str, Any]]:
"""Create a smaller summary of the graph, suitable for basic diagnostics
and performance tuning. The summary includes only the edge structure,
node types, and node names. Each of the n nodes is assigned an integer
index 0, 1, 2,..., n-1 for compactness"""
graph_nodes = dict()
index_dict = dict()
for node_index, node_name in enumerate(self.graph):
index_dict[node_name] = node_index
data = manifest.expect(node_name).to_dict(omit_none=True)
graph_nodes[node_index] = {"name": node_name, "type": data["resource_type"]}
for node_index, node in graph_nodes.items():
successors = [index_dict[n] for n in self.graph.successors(node["name"])]
if successors:
node["succ"] = [index_dict[n] for n in self.graph.successors(node["name"])]
return graph_nodes
class Compiler:
def __init__(self, config):
@@ -304,6 +193,62 @@ class Compiler:
relation_cls = adapter.Relation
return relation_cls.add_ephemeral_prefix(name)
def _inject_ctes_into_sql(self, sql: str, ctes: List[InjectedCTE]) -> str:
"""
`ctes` is a list of InjectedCTEs like:
[
InjectedCTE(
id="cte_id_1",
sql="__dbt__cte__ephemeral as (select * from table)",
),
InjectedCTE(
id="cte_id_2",
sql="__dbt__cte__events as (select id, type from events)",
),
]
Given `sql` like:
"with internal_cte as (select * from sessions)
select * from internal_cte"
This will spit out:
"with __dbt__cte__ephemeral as (select * from table),
__dbt__cte__events as (select id, type from events),
with internal_cte as (select * from sessions)
select * from internal_cte"
(Whitespace enhanced for readability.)
"""
if len(ctes) == 0:
return sql
parsed_stmts = sqlparse.parse(sql)
parsed = parsed_stmts[0]
with_stmt = None
for token in parsed.tokens:
if token.is_keyword and token.normalized == "WITH":
with_stmt = token
break
if with_stmt is None:
# no with stmt, add one, and inject CTEs right at the beginning
first_token = parsed.token_first()
with_stmt = sqlparse.sql.Token(sqlparse.tokens.Keyword, "with")
parsed.insert_before(first_token, with_stmt)
else:
# stmt exists, add a comma (which will come after injected CTEs)
trailing_comma = sqlparse.sql.Token(sqlparse.tokens.Punctuation, ",")
parsed.insert_after(with_stmt, trailing_comma)
token = sqlparse.sql.Token(sqlparse.tokens.Keyword, ", ".join(c.sql for c in ctes))
parsed.insert_after(with_stmt, token)
return str(parsed)
def _recursively_prepend_ctes(
self,
model: ManifestSQLNode,
@@ -378,16 +323,16 @@ class Compiler:
_add_prepended_cte(prepended_ctes, InjectedCTE(id=cte.id, sql=sql))
injected_sql = inject_ctes_into_sql(
model.compiled_code,
prepended_ctes,
)
# Check again before updating for multi-threading
if not model.extra_ctes_injected:
injected_sql = self._inject_ctes_into_sql(
model.compiled_code,
prepended_ctes,
)
model.extra_ctes_injected = True
model._pre_injected_sql = model.compiled_code
model.compiled_code = injected_sql
model.extra_ctes = prepended_ctes
model.extra_ctes_injected = True
# if model.extra_ctes is not set to prepended ctes, something went wrong
return model, model.extra_ctes
@@ -440,39 +385,102 @@ class Compiler:
return node
# This method doesn't actually "compile" any of the nodes. That is done by the
# "compile_node" method. This creates a Linker and builds the networkx graph,
# writes out the graph.gpickle file, and prints the stats, returning a Graph object.
def compile(self, manifest: Manifest, write=True, add_test_edges=False) -> Graph:
self.initialize()
linker = Linker()
linker.link_graph(manifest)
def write_graph_file(self, linker: Linker, manifest: Manifest):
filename = graph_file_name
graph_path = os.path.join(self.config.project_target_path, filename)
flags = get_flags()
if flags.WRITE_JSON:
linker.write_graph(graph_path, manifest)
# Create a file containing basic information about graph structure,
# supporting diagnostics and performance analysis.
summaries: Dict = dict()
summaries["_invocation_id"] = get_invocation_id()
summaries["linked"] = linker.get_graph_summary(manifest)
def link_node(self, linker: Linker, node: GraphMemberNode, manifest: Manifest):
linker.add_node(node.unique_id)
for dependency in node.depends_on_nodes:
if dependency in manifest.nodes:
linker.dependency(node.unique_id, (manifest.nodes[dependency].unique_id))
elif dependency in manifest.sources:
linker.dependency(node.unique_id, (manifest.sources[dependency].unique_id))
elif dependency in manifest.metrics:
linker.dependency(node.unique_id, (manifest.metrics[dependency].unique_id))
else:
raise GraphDependencyNotFoundError(node, dependency)
def link_graph(self, linker: Linker, manifest: Manifest, add_test_edges: bool = False):
for source in manifest.sources.values():
linker.add_node(source.unique_id)
for node in manifest.nodes.values():
self.link_node(linker, node, manifest)
for exposure in manifest.exposures.values():
self.link_node(linker, exposure, manifest)
for metric in manifest.metrics.values():
self.link_node(linker, metric, manifest)
cycle = linker.find_cycles()
if cycle:
raise RuntimeError("Found a cycle: {}".format(cycle))
if add_test_edges:
manifest.build_parent_and_child_maps()
linker.add_test_edges(manifest)
self.add_test_edges(linker, manifest)
# Create another diagnostic summary, just as above, but this time
# including the test edges.
summaries["with_test_edges"] = linker.get_graph_summary(manifest)
def add_test_edges(self, linker: Linker, manifest: Manifest) -> None:
"""This method adds additional edges to the DAG. For a given non-test
executable node, add an edge from an upstream test to the given node if
the set of nodes the test depends on is a subset of the upstream nodes
for the given node."""
with open(
os.path.join(self.config.project_target_path, "graph_summary.json"), "w"
) as out_stream:
try:
out_stream.write(json.dumps(summaries))
except Exception as e: # This is non-essential information, so merely note failures.
fire_event(
Note(
msg=f"An error was encountered writing the graph summary information: {e}"
)
)
# Given a graph:
# model1 --> model2 --> model3
# | |
# | \/
# \/ test 2
# test1
#
# Produce the following graph:
# model1 --> model2 --> model3
# | /\ | /\ /\
# | | \/ | |
# \/ | test2 ----| |
# test1 ----|---------------|
for node_id in linker.graph:
# If node is executable (in manifest.nodes) and does _not_
# represent a test, continue.
if (
node_id in manifest.nodes
and manifest.nodes[node_id].resource_type != NodeType.Test
):
# Get *everything* upstream of the node
all_upstream_nodes = nx.traversal.bfs_tree(linker.graph, node_id, reverse=True)
# Get the set of upstream nodes not including the current node.
upstream_nodes = set([n for n in all_upstream_nodes if n != node_id])
# Get all tests that depend on any upstream nodes.
upstream_tests = []
for upstream_node in upstream_nodes:
upstream_tests += _get_tests_for_node(manifest, upstream_node)
for upstream_test in upstream_tests:
# Get the set of all nodes that the test depends on
# including the upstream_node itself. This is necessary
# because tests can depend on multiple nodes (ex:
# relationship tests). Test nodes do not distinguish
# between what node the test is "testing" and what
# node(s) it depends on.
test_depends_on = set(manifest.nodes[upstream_test].depends_on_nodes)
# If the set of nodes that an upstream test depends on
# is a subset of all upstream nodes of the current node,
# add an edge from the upstream test to the current node.
if test_depends_on.issubset(upstream_nodes):
linker.graph.add_edge(upstream_test, node_id)
def compile(self, manifest: Manifest, write=True, add_test_edges=False) -> Graph:
self.initialize()
linker = Linker()
self.link_graph(linker, manifest, add_test_edges)
stats = _generate_stats(manifest)
@@ -484,18 +492,10 @@ class Compiler:
self.config.args.__class__ == argparse.Namespace
and self.config.args.cls == list_task.ListTask
):
stats = _generate_stats(manifest)
print_compile_stats(stats)
return Graph(linker.graph)
def write_graph_file(self, linker: Linker, manifest: Manifest):
filename = graph_file_name
graph_path = os.path.join(self.config.project_target_path, filename)
flags = get_flags()
if flags.WRITE_JSON:
linker.write_graph(graph_path, manifest)
# writes the "compiled_code" into the target/compiled directory
def _write_node(self, node: ManifestSQLNode) -> ManifestSQLNode:
if not node.extra_ctes_injected or node.resource_type in (
@@ -523,80 +523,15 @@ class Compiler:
the node's raw_code into compiled_code, and then calls the
recursive method to "prepend" the ctes.
"""
# Make sure Lexer for sqlparse 0.4.4 is initialized
from sqlparse.lexer import Lexer # type: ignore
if hasattr(Lexer, "get_default_instance"):
Lexer.get_default_instance()
node = self._compile_code(node, manifest, extra_context)
node, _ = self._recursively_prepend_ctes(node, manifest, extra_context)
if write:
self._write_node(node)
return node
def inject_ctes_into_sql(sql: str, ctes: List[InjectedCTE]) -> str:
"""
`ctes` is a list of InjectedCTEs like:
[
InjectedCTE(
id="cte_id_1",
sql="__dbt__cte__ephemeral as (select * from table)",
),
InjectedCTE(
id="cte_id_2",
sql="__dbt__cte__events as (select id, type from events)",
),
]
Given `sql` like:
"with internal_cte as (select * from sessions)
select * from internal_cte"
This will spit out:
"with __dbt__cte__ephemeral as (select * from table),
__dbt__cte__events as (select id, type from events),
internal_cte as (select * from sessions)
select * from internal_cte"
(Whitespace enhanced for readability.)
"""
if len(ctes) == 0:
return sql
parsed_stmts = sqlparse.parse(sql)
parsed = parsed_stmts[0]
with_stmt = None
for token in parsed.tokens:
if token.is_keyword and token.normalized == "WITH":
with_stmt = token
elif token.is_keyword and token.normalized == "RECURSIVE" and with_stmt is not None:
with_stmt = token
break
elif not token.is_whitespace and with_stmt is not None:
break
if with_stmt is None:
# no with stmt, add one, and inject CTEs right at the beginning
# [original_sql]
first_token = parsed.token_first()
with_token = sqlparse.sql.Token(sqlparse.tokens.Keyword, "with")
parsed.insert_before(first_token, with_token)
# [with][original_sql]
injected_ctes = ", ".join(c.sql for c in ctes) + " "
injected_ctes_token = sqlparse.sql.Token(sqlparse.tokens.Keyword, injected_ctes)
parsed.insert_after(with_token, injected_ctes_token)
# [with][joined_ctes][original_sql]
else:
# with stmt exists so we don't need to add one, but we do need to add a comma
# between the injected ctes and the original sql
# [with][original_sql]
injected_ctes = ", ".join(c.sql for c in ctes)
injected_ctes_token = sqlparse.sql.Token(sqlparse.tokens.Keyword, injected_ctes)
parsed.insert_after(with_stmt, injected_ctes_token)
# [with][joined_ctes][original_sql]
comma_token = sqlparse.sql.Token(sqlparse.tokens.Punctuation, ", ")
parsed.insert_after(injected_ctes_token, comma_token)
# [with][joined_ctes][, ][original_sql]
return str(parsed)

View File

@@ -16,7 +16,6 @@ import os
from dbt.flags import get_flags
from dbt import deprecations
from dbt.constants import DEPENDENCIES_FILE_NAME, PACKAGES_FILE_NAME
from dbt.clients.system import path_exists, resolve_path_from_base, load_file_contents
from dbt.clients.yaml_helper import load_yaml_text
from dbt.contracts.connection import QueryComment
@@ -94,36 +93,17 @@ def _load_yaml(path):
return load_yaml_text(contents)
def package_and_project_data_from_root(project_root):
package_filepath = resolve_path_from_base(PACKAGES_FILE_NAME, project_root)
dependencies_filepath = resolve_path_from_base(DEPENDENCIES_FILE_NAME, project_root)
def package_data_from_root(project_root):
package_filepath = resolve_path_from_base("packages.yml", project_root)
packages_yml_dict = {}
dependencies_yml_dict = {}
if path_exists(package_filepath):
packages_yml_dict = _load_yaml(package_filepath) or {}
if path_exists(dependencies_filepath):
dependencies_yml_dict = _load_yaml(dependencies_filepath) or {}
if "packages" in packages_yml_dict and "packages" in dependencies_yml_dict:
msg = "The 'packages' key cannot be specified in both packages.yml and dependencies.yml"
raise DbtProjectError(msg)
if "projects" in packages_yml_dict:
msg = "The 'projects' key cannot be specified in packages.yml"
raise DbtProjectError(msg)
packages_specified_path = PACKAGES_FILE_NAME
packages_dict = {}
if "packages" in dependencies_yml_dict:
packages_dict["packages"] = dependencies_yml_dict["packages"]
packages_specified_path = DEPENDENCIES_FILE_NAME
else: # don't check for "packages" here so we capture invalid keys in packages.yml
packages_dict = packages_yml_dict
return packages_dict, packages_specified_path
packages_dict = _load_yaml(package_filepath)
else:
packages_dict = None
return packages_dict
def package_config_from_data(packages_data: Dict[str, Any]) -> PackageConfig:
def package_config_from_data(packages_data: Dict[str, Any]):
if not packages_data:
packages_data = {"packages": []}
@@ -264,7 +244,6 @@ class RenderComponents:
@dataclass
class PartialProject(RenderComponents):
# This class includes the project_dict, packages_dict, selectors_dict, etc from RenderComponents
profile_name: Optional[str] = field(
metadata=dict(description="The unrendered profile name in the project, if set")
)
@@ -281,9 +260,6 @@ class PartialProject(RenderComponents):
verify_version: bool = field(
metadata=dict(description=("If True, verify the dbt version matches the required version"))
)
packages_specified_path: str = field(
metadata=dict(description="The filename where packages were specified")
)
def render_profile_name(self, renderer) -> Optional[str]:
if self.profile_name is None:
@@ -296,9 +272,7 @@ class PartialProject(RenderComponents):
) -> RenderComponents:
rendered_project = renderer.render_project(self.project_dict, self.project_root)
rendered_packages = renderer.render_packages(
self.packages_dict, self.packages_specified_path
)
rendered_packages = renderer.render_packages(self.packages_dict)
rendered_selectors = renderer.render_selectors(self.selectors_dict)
return RenderComponents(
@@ -307,7 +281,7 @@ class PartialProject(RenderComponents):
selectors_dict=rendered_selectors,
)
# Called by Project.from_project_root (not PartialProject.from_project_root!)
# Called by 'collect_parts' in RuntimeConfig
def render(self, renderer: DbtProjectYamlRenderer) -> "Project":
try:
rendered = self.get_rendered(renderer)
@@ -428,6 +402,7 @@ class PartialProject(RenderComponents):
metrics: Dict[str, Any]
exposures: Dict[str, Any]
vars_value: VarProvider
dbt_cloud: Dict[str, Any]
dispatch = cfg.dispatch
models = cfg.models
@@ -450,7 +425,7 @@ class PartialProject(RenderComponents):
query_comment = _query_comment_from_cfg(cfg.query_comment)
packages: PackageConfig = package_config_from_data(rendered.packages_dict)
packages = package_config_from_data(rendered.packages_dict)
selectors = selector_config_from_data(rendered.selectors_dict)
manifest_selectors: Dict[str, Any] = {}
if rendered.selectors_dict and rendered.selectors_dict["selectors"]:
@@ -459,6 +434,8 @@ class PartialProject(RenderComponents):
manifest_selectors = SelectorDict.parse_from_selectors_list(
rendered.selectors_dict["selectors"]
)
dbt_cloud = cfg.dbt_cloud
project = Project(
project_name=name,
version=version,
@@ -476,7 +453,6 @@ class PartialProject(RenderComponents):
clean_targets=clean_targets,
log_path=log_path,
packages_install_path=packages_install_path,
packages_specified_path=self.packages_specified_path,
quoting=quoting,
models=models,
on_run_start=on_run_start,
@@ -497,7 +473,7 @@ class PartialProject(RenderComponents):
config_version=cfg.config_version,
unrendered=unrendered,
project_env_vars=project_env_vars,
restrict_access=cfg.restrict_access,
dbt_cloud=dbt_cloud,
)
# sanity check - this means an internal issue
project.validate()
@@ -512,13 +488,11 @@ class PartialProject(RenderComponents):
selectors_dict: Dict[str, Any],
*,
verify_version: bool = False,
packages_specified_path: str = PACKAGES_FILE_NAME,
):
"""Construct a partial project from its constituent dicts."""
project_name = project_dict.get("name")
profile_name = project_dict.get("profile")
# Create a PartialProject
return cls(
profile_name=profile_name,
project_name=project_name,
@@ -527,7 +501,6 @@ class PartialProject(RenderComponents):
packages_dict=packages_dict,
selectors_dict=selectors_dict,
verify_version=verify_version,
packages_specified_path=packages_specified_path,
)
@classmethod
@@ -536,10 +509,7 @@ class PartialProject(RenderComponents):
) -> "PartialProject":
project_root = os.path.normpath(project_root)
project_dict = load_raw_project(project_root)
(
packages_dict,
packages_specified_path,
) = package_and_project_data_from_root(project_root)
packages_dict = package_data_from_root(project_root)
selectors_dict = selector_data_from_root(project_root)
return cls.from_dicts(
project_root=project_root,
@@ -547,7 +517,6 @@ class PartialProject(RenderComponents):
selectors_dict=selectors_dict,
packages_dict=packages_dict,
verify_version=verify_version,
packages_specified_path=packages_specified_path,
)
@@ -587,7 +556,6 @@ class Project:
clean_targets: List[str]
log_path: str
packages_install_path: str
packages_specified_path: str
quoting: Dict[str, Any]
models: Dict[str, Any]
on_run_start: List[str]
@@ -601,14 +569,14 @@ class Project:
exposures: Dict[str, Any]
vars: VarProvider
dbt_version: List[VersionSpecifier]
packages: PackageConfig
packages: Dict[str, Any]
manifest_selectors: Dict[str, Any]
selectors: SelectorConfig
query_comment: QueryComment
config_version: int
unrendered: RenderComponents
project_env_vars: Dict[str, Any]
restrict_access: bool
dbt_cloud: Dict[str, Any]
@property
def all_source_paths(self) -> List[str]:
@@ -677,7 +645,7 @@ class Project:
"vars": self.vars.to_dict(),
"require-dbt-version": [v.to_version_string() for v in self.dbt_version],
"config-version": self.config_version,
"restrict-access": self.restrict_access,
"dbt-cloud": self.dbt_cloud,
}
)
if self.query_comment:
@@ -694,9 +662,13 @@ class Project:
except ValidationError as e:
raise ProjectContractBrokenError(e) from e
# Called by:
# RtConfig.load_dependencies => RtConfig.load_projects => RtConfig.new_project => Project.from_project_root
# RtConfig.from_args => RtConfig.collect_parts => load_project => Project.from_project_root
@classmethod
def partial_load(cls, project_root: str, *, verify_version: bool = False) -> PartialProject:
return PartialProject.from_project_root(
project_root,
verify_version=verify_version,
)
@classmethod
def from_project_root(
cls,

View File

@@ -1,10 +1,9 @@
from typing import Dict, Any, Tuple, Optional, Union, Callable
import re
import os
from datetime import date
from dbt.clients.jinja import get_rendered, catch_jinja
from dbt.constants import SECRET_ENV_PREFIX, DEPENDENCIES_FILE_NAME
from dbt.constants import SECRET_ENV_PREFIX
from dbt.context.target import TargetContext
from dbt.context.secret import SecretContext, SECRET_PLACEHOLDER
from dbt.context.base import BaseContext
@@ -34,10 +33,10 @@ class BaseRenderer:
return self.render_value(value, keypath)
def render_value(self, value: Any, keypath: Optional[Keypath] = None) -> Any:
# keypath is ignored (and someone who knows should explain why here)
# keypath is ignored.
# if it wasn't read as a string, ignore it
if not isinstance(value, str):
return value if not isinstance(value, date) else value.isoformat()
return value
try:
with catch_jinja():
return get_rendered(value, self.context, native=True)
@@ -132,15 +131,10 @@ class DbtProjectYamlRenderer(BaseRenderer):
rendered_project["project-root"] = project_root
return rendered_project
def render_packages(self, packages: Dict[str, Any], packages_specified_path: str):
def render_packages(self, packages: Dict[str, Any]):
"""Render the given packages dict"""
packages = packages or {} # Sometimes this is none in tests
package_renderer = self.get_package_renderer()
if packages_specified_path == DEPENDENCIES_FILE_NAME:
# We don't want to render the "packages" dictionary that came from dependencies.yml
return packages
else:
return package_renderer.render_data(packages)
return package_renderer.render_data(packages)
def render_selectors(self, selectors: Dict[str, Any]):
return self.render_data(selectors)

View File

@@ -38,7 +38,6 @@ from .project import Project
from .renderer import DbtProjectYamlRenderer, ProfileRenderer
# Called by RuntimeConfig.collect_parts class method
def load_project(
project_root: str,
version_check: bool,
@@ -151,7 +150,6 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
clean_targets=project.clean_targets,
log_path=project.log_path,
packages_install_path=project.packages_install_path,
packages_specified_path=project.packages_specified_path,
quoting=quoting,
models=project.models,
on_run_start=project.on_run_start,
@@ -172,7 +170,6 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
config_version=project.config_version,
unrendered=project.unrendered,
project_env_vars=project.project_env_vars,
restrict_access=project.restrict_access,
profile_env_vars=profile.profile_env_vars,
profile_name=profile.profile_name,
target_name=profile.target_name,
@@ -182,6 +179,7 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
args=args,
cli_vars=cli_vars,
dependencies=dependencies,
dbt_cloud=project.dbt_cloud,
)
# Called by 'load_projects' in this class
@@ -239,7 +237,6 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
except ValidationError as e:
raise ConfigContractBrokenError(e) from e
# Called by RuntimeConfig.from_args
@classmethod
def collect_parts(cls: Type["RuntimeConfig"], args: Any) -> Tuple[Project, Profile]:
# profile_name from the project
@@ -254,7 +251,7 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
project = load_project(project_root, bool(flags.VERSION_CHECK), profile, cli_vars)
return project, profile
# Called in task/base.py, in BaseTask.from_args
# Called in main.py, lib.py, task/base.py
@classmethod
def from_args(cls, args: Any) -> "RuntimeConfig":
"""Given arguments, read in dbt_project.yml from the current directory,
@@ -275,11 +272,7 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
)
def get_metadata(self) -> ManifestMetadata:
return ManifestMetadata(
project_name=self.project_name,
project_id=self.hashed_name(),
adapter_type=self.credentials.type,
)
return ManifestMetadata(project_id=self.hashed_name(), adapter_type=self.credentials.type)
def _get_v2_config_paths(
self,
@@ -366,7 +359,6 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
raise UninstalledPackagesFoundError(
count_packages_specified,
count_packages_installed,
self.packages_specified_path,
self.packages_install_path,
)
project_paths = itertools.chain(internal_packages, self._get_project_directories())

View File

@@ -21,7 +21,7 @@ The selectors.yml file in this project is malformed. Please double check
the contents of this file and fix any errors before retrying.
You can find more information on the syntax for this file here:
https://docs.getdbt.com/reference/node-selection/yaml-selectors
https://docs.getdbt.com/docs/package-management
Validator Error:
{error}

View File

@@ -8,9 +8,3 @@ MAXIMUM_SEED_SIZE_NAME = "1MB"
PIN_PACKAGE_URL = (
"https://docs.getdbt.com/docs/package-management#section-specifying-package-versions"
)
PACKAGES_FILE_NAME = "packages.yml"
DEPENDENCIES_FILE_NAME = "dependencies.yml"
MANIFEST_FILE_NAME = "manifest.json"
SEMANTIC_MANIFEST_FILE_NAME = "semantic_manifest.json"
PARTIAL_PARSE_FILE_NAME = "partial_parse.msgpack"

View File

@@ -1,7 +1,6 @@
import json
import os
from typing import Any, Dict, NoReturn, Optional, Mapping, Iterable, Set, List
import threading
from dbt.flags import get_flags
import dbt.flags as flags_module
@@ -597,11 +596,6 @@ class BaseContext(metaclass=ContextMeta):
"""
return get_invocation_id()
@contextproperty
def thread_id(self) -> str:
"""thread_id outputs an ID for the current thread (useful for auditing)"""
return threading.current_thread().name
@contextproperty
def modules(self) -> Dict[str, Any]:
"""The `modules` variable in the Jinja context contains useful Python

View File

@@ -52,11 +52,10 @@ class ConfiguredVar(Var):
adapter_type = self._config.credentials.type
lookup = FQNLookup(self._project_name)
active_vars = self._config.vars.vars_for(lookup, adapter_type)
all_vars = MultiDict([active_vars])
all_vars = MultiDict()
if self._config.project_name != my_config.project_name:
all_vars.add(my_config.vars.vars_for(lookup, adapter_type))
all_vars.add(active_vars)
if var_name in all_vars:
return all_vars[var_name]
@@ -119,9 +118,7 @@ class MacroResolvingContext(ConfiguredContext):
def generate_schema_yml_context(
config: AdapterRequiredConfig,
project_name: str,
schema_yaml_vars: Optional[SchemaYamlVars] = None,
config: AdapterRequiredConfig, project_name: str, schema_yaml_vars: SchemaYamlVars = None
) -> Dict[str, Any]:
ctx = SchemaYamlContext(config, project_name, schema_yaml_vars)
return ctx.to_dict()

View File

@@ -1,7 +1,7 @@
from abc import abstractmethod
from copy import deepcopy
from dataclasses import dataclass
from typing import List, Iterator, Dict, Any, TypeVar, Generic, Optional
from typing import List, Iterator, Dict, Any, TypeVar, Generic
from dbt.config import RuntimeConfig, Project, IsFQNResource
from dbt.contracts.graph.model_config import BaseConfig, get_config_for, _listify
@@ -130,7 +130,7 @@ class BaseContextConfigGenerator(Generic[T]):
resource_type: NodeType,
project_name: str,
base: bool,
patch_config_dict: Optional[Dict[str, Any]] = None,
patch_config_dict: Dict[str, Any] = None,
) -> BaseConfig:
own_config = self.get_node_project(project_name)
@@ -166,7 +166,7 @@ class BaseContextConfigGenerator(Generic[T]):
resource_type: NodeType,
project_name: str,
base: bool,
patch_config_dict: Optional[Dict[str, Any]] = None,
patch_config_dict: Dict[str, Any],
) -> Dict[str, Any]:
...
@@ -200,7 +200,7 @@ class ContextConfigGenerator(BaseContextConfigGenerator[C]):
resource_type: NodeType,
project_name: str,
base: bool,
patch_config_dict: Optional[dict] = None,
patch_config_dict: dict = None,
) -> Dict[str, Any]:
config = self.calculate_node_config(
config_call_dict=config_call_dict,
@@ -225,7 +225,7 @@ class UnrenderedConfigGenerator(BaseContextConfigGenerator[Dict[str, Any]]):
resource_type: NodeType,
project_name: str,
base: bool,
patch_config_dict: Optional[dict] = None,
patch_config_dict: dict = None,
) -> Dict[str, Any]:
# TODO CT-211
return self.calculate_node_config(
@@ -318,11 +318,7 @@ class ContextConfig:
config_call_dict[k] = v
def build_config_dict(
self,
base: bool = False,
*,
rendered: bool = True,
patch_config_dict: Optional[dict] = None,
self, base: bool = False, *, rendered: bool = True, patch_config_dict: dict = None
) -> Dict[str, Any]:
if rendered:
# TODO CT-211

View File

@@ -25,7 +25,6 @@ from dbt.exceptions import (
RelationWrongTypeError,
ContractError,
ColumnTypeMissingError,
FailFastError,
)
@@ -108,10 +107,6 @@ def column_type_missing(column_names) -> NoReturn:
raise ColumnTypeMissingError(column_names)
def raise_fail_fast_error(msg, node=None) -> NoReturn:
raise FailFastError(msg, node=node)
# Update this when a new function should be added to the
# dbt context's `exceptions` key!
CONTEXT_EXPORTS = {
@@ -136,7 +131,6 @@ CONTEXT_EXPORTS = {
relation_wrong_type,
raise_contract_error,
column_type_missing,
raise_fail_fast_error,
]
}

View File

@@ -32,13 +32,13 @@ from dbt.contracts.graph.manifest import Manifest, Disabled
from dbt.contracts.graph.nodes import (
Macro,
Exposure,
Metric,
SeedNode,
SourceDefinition,
Resource,
ManifestNode,
RefArgs,
AccessType,
SemanticModel,
)
from dbt.contracts.graph.metrics import MetricReference, ResolvedMetricReference
from dbt.contracts.graph.unparsed import NodeVersion
@@ -55,7 +55,6 @@ from dbt.exceptions import (
LoadAgateTableNotSeedError,
LoadAgateTableValueError,
MacroDispatchArgError,
MacroResultAlreadyLoadedError,
MacrosSourcesUnWriteableError,
MetricArgsError,
MissingConfigError,
@@ -133,25 +132,6 @@ class BaseDatabaseWrapper:
search_prefixes = get_adapter_type_names(self._adapter.type()) + ["default"]
return search_prefixes
def _get_search_packages(self, namespace: Optional[str] = None) -> List[Optional[str]]:
search_packages: List[Optional[str]] = [None]
if namespace is None:
search_packages = [None]
elif isinstance(namespace, str):
macro_search_order = self._adapter.config.get_macro_search_order(namespace)
if macro_search_order:
search_packages = macro_search_order
elif not macro_search_order and namespace in self._adapter.config.dependencies:
search_packages = [self.config.project_name, namespace]
else:
raise CompilationError(
f"In adapter.dispatch, got a {type(namespace)} macro_namespace argument "
f'("{namespace}"), but macro_namespace should be None or a string.'
)
return search_packages
def dispatch(
self,
macro_name: str,
@@ -173,7 +153,20 @@ class BaseDatabaseWrapper:
if packages is not None:
raise MacroDispatchArgError(macro_name)
search_packages = self._get_search_packages(macro_namespace)
namespace = macro_namespace
if namespace is None:
search_packages = [None]
elif isinstance(namespace, str):
search_packages = self._adapter.config.get_macro_search_order(namespace)
if not search_packages and namespace in self._adapter.config.dependencies:
search_packages = [self.config.project_name, namespace]
else:
# Not a string and not None so must be a list
raise CompilationError(
f"In adapter.dispatch, got a list macro_namespace argument "
f'("{macro_namespace}"), but macro_namespace should be None or a string.'
)
attempts = []
@@ -197,7 +190,7 @@ class BaseDatabaseWrapper:
return macro
searched = ", ".join(repr(a) for a in attempts)
msg = f"In dispatch: No macro named '{macro_name}' found within namespace: '{macro_namespace}'\n Searched for: {searched}"
msg = f"In dispatch: No macro named '{macro_name}' found\n Searched for: {searched}"
raise CompilationError(msg)
@@ -291,7 +284,6 @@ class BaseSourceResolver(BaseResolver):
class BaseMetricResolver(BaseResolver):
@abc.abstractmethod
def resolve(self, name: str, package: Optional[str] = None) -> MetricReference:
...
@@ -474,7 +466,6 @@ class ParseRefResolver(BaseRefResolver):
) -> RelationProxy:
self.model.refs.append(self._repack_args(name, package, version))
# This is not the ref for the "name" passed in, but for the current model.
return self.Relation.create_from(self.config, self.model)
@@ -489,7 +480,6 @@ class RuntimeRefResolver(BaseRefResolver):
target_version: Optional[NodeVersion] = None,
) -> RelationProxy:
target_model = self.manifest.resolve_ref(
self.model,
target_name,
target_package,
target_version,
@@ -506,24 +496,19 @@ class RuntimeRefResolver(BaseRefResolver):
target_version=target_version,
disabled=isinstance(target_model, Disabled),
)
elif self.manifest.is_invalid_private_ref(
self.model, target_model, self.config.dependencies
elif (
target_model.resource_type == NodeType.Model
and target_model.access == AccessType.Private
# don't raise this reference error for ad hoc 'preview' queries
and self.model.resource_type != NodeType.SqlOperation
and self.model.resource_type != NodeType.RPCCall # TODO: rm
):
raise DbtReferenceError(
unique_id=self.model.unique_id,
ref_unique_id=target_model.unique_id,
access=AccessType.Private,
scope=cast_to_str(target_model.group),
)
elif self.manifest.is_invalid_protected_ref(
self.model, target_model, self.config.dependencies
):
raise DbtReferenceError(
unique_id=self.model.unique_id,
ref_unique_id=target_model.unique_id,
access=AccessType.Protected,
scope=target_model.package_name,
)
if not self.model.group or self.model.group != target_model.group:
raise DbtReferenceError(
unique_id=self.model.unique_id,
ref_unique_id=target_model.unique_id,
group=cast_to_str(target_model.group),
)
self.validate(target_model, target_name, target_package, target_version)
return self.create_relation(target_model)
@@ -735,7 +720,7 @@ class ProviderContext(ManifestContext):
self.config: RuntimeConfig
self.model: Union[Macro, ManifestNode] = model
super().__init__(config, manifest, model.package_name)
self.sql_results: Dict[str, Optional[AttrDict]] = {}
self.sql_results: Dict[str, AttrDict] = {}
self.context_config: Optional[ContextConfig] = context_config
self.provider: Provider = provider
self.adapter = get_adapter(self.config)
@@ -763,29 +748,12 @@ class ProviderContext(ManifestContext):
return args_to_dict(self.config.args)
@contextproperty
def _sql_results(self) -> Dict[str, Optional[AttrDict]]:
def _sql_results(self) -> Dict[str, AttrDict]:
return self.sql_results
@contextmember
def load_result(self, name: str) -> Optional[AttrDict]:
if name in self.sql_results:
# handle the special case of "main" macro
# See: https://github.com/dbt-labs/dbt-core/blob/ada8860e48b32ac712d92e8b0977b2c3c9749981/core/dbt/task/run.py#L228
if name == "main":
return self.sql_results["main"]
# handle a None, which indicates this name was populated but has since been loaded
elif self.sql_results[name] is None:
raise MacroResultAlreadyLoadedError(name)
# Handle the regular use case
else:
ret_val = self.sql_results[name]
self.sql_results[name] = None
return ret_val
else:
# Handle trying to load a result that was never stored
return None
return self.sql_results.get(name)
@contextmember
def store_result(
@@ -862,8 +830,18 @@ class ProviderContext(ManifestContext):
def load_agate_table(self) -> agate.Table:
if not isinstance(self.model, SeedNode):
raise LoadAgateTableNotSeedError(self.model.resource_type, node=self.model)
assert self.model.root_path
path = os.path.join(self.model.root_path, self.model.original_file_path)
# include package_path for seeds defined in packages
package_path = (
os.path.join(self.config.packages_install_path, self.model.package_name)
if self.model.package_name != self.config.project_name
else "."
)
path = os.path.join(self.config.project_root, package_path, self.model.original_file_path)
if not os.path.exists(path):
assert self.model.root_path
path = os.path.join(self.model.root_path, self.model.original_file_path)
column_types = self.model.config.column_types
try:
table = agate_helper.from_csv(path, text_columns=column_types)
@@ -1378,30 +1356,20 @@ class ModelContext(ProviderContext):
@contextproperty
def sql(self) -> Optional[str]:
# only doing this in sql model for backward compatible
if self.model.language == ModelLanguage.sql: # type: ignore[union-attr]
# If the model is deferred and the adapter doesn't support zero-copy cloning, then select * from the prod
# relation
if getattr(self.model, "defer_relation", None):
# TODO https://github.com/dbt-labs/dbt-core/issues/7976
return f"select * from {self.model.defer_relation.relation_name or str(self.defer_relation)}" # type: ignore[union-attr]
elif getattr(self.model, "extra_ctes_injected", None):
# TODO CT-211
return self.model.compiled_code # type: ignore[union-attr]
else:
return None
else:
return None
if (
getattr(self.model, "extra_ctes_injected", None)
and self.model.language == ModelLanguage.sql # type: ignore[union-attr]
):
# TODO CT-211
return self.model.compiled_code # type: ignore[union-attr]
return None
@contextproperty
def compiled_code(self) -> Optional[str]:
if getattr(self.model, "defer_relation", None):
# TODO https://github.com/dbt-labs/dbt-core/issues/7976
return f"select * from {self.model.defer_relation.relation_name or str(self.defer_relation)}" # type: ignore[union-attr]
elif getattr(self.model, "extra_ctes_injected", None):
if getattr(self.model, "extra_ctes_injected", None):
# TODO CT-211
return self.model.compiled_code # type: ignore[union-attr]
else:
return None
return None
@contextproperty
def database(self) -> str:
@@ -1446,20 +1414,6 @@ class ModelContext(ProviderContext):
return None
return self.db_wrapper.Relation.create_from(self.config, self.model)
@contextproperty
def defer_relation(self) -> Optional[RelationProxy]:
"""
For commands which add information about this node's corresponding
production version (via a --state artifact), access the Relation
object for that stateful other
"""
if getattr(self.model, "defer_relation", None):
return self.db_wrapper.Relation.create_from_node(
self.config, self.model.defer_relation # type: ignore
)
else:
return None
# This is called by '_context_for', used in 'render_with_context'
def generate_parser_model_context(
@@ -1566,8 +1520,7 @@ def generate_parse_exposure(
}
# applies to SemanticModels
class SemanticModelRefResolver(BaseResolver):
class MetricRefResolver(BaseResolver):
def __call__(self, *args, **kwargs) -> str:
package = None
if len(args) == 1:
@@ -1580,30 +1533,34 @@ class SemanticModelRefResolver(BaseResolver):
version = kwargs.get("version") or kwargs.get("v")
self.validate_args(name, package, version)
# "model" here is any node
self.model.refs.append(RefArgs(package=package, name=name, version=version))
return ""
def validate_args(self, name, package, version):
if not isinstance(name, str):
raise ParsingError(
f"In a semantic model or metrics section in {self.model.original_file_path} "
f"In a metrics section in {self.model.original_file_path} "
"the name argument to ref() must be a string"
)
# used for semantic models
def generate_parse_semantic_models(
semantic_model: SemanticModel,
def generate_parse_metrics(
metric: Metric,
config: RuntimeConfig,
manifest: Manifest,
package_name: str,
) -> Dict[str, Any]:
project = config.load_dependencies()[package_name]
return {
"ref": SemanticModelRefResolver(
"ref": MetricRefResolver(
None,
semantic_model,
metric,
project,
manifest,
),
"metric": ParseMetricResolver(
None,
metric,
project,
manifest,
),

View File

@@ -228,7 +228,6 @@ class SchemaSourceFile(BaseSourceFile):
groups: List[str] = field(default_factory=list)
# node patches contain models, seeds, snapshots, analyses
ndp: List[str] = field(default_factory=list)
semantic_models: List[str] = field(default_factory=list)
# any macro patches in this file by macro unique_id.
mcp: Dict[str, str] = field(default_factory=dict)
# any source patches in this file. The entries are package, name pairs

View File

@@ -1,11 +1,9 @@
import enum
from collections import defaultdict
from dataclasses import dataclass, field
from itertools import chain, islice
from mashumaro.mixins.msgpack import DataClassMessagePackMixin
from multiprocessing.synchronize import Lock
from typing import (
DefaultDict,
Dict,
List,
Optional,
@@ -25,21 +23,18 @@ from typing_extensions import Protocol
from uuid import UUID
from dbt.contracts.graph.nodes import (
BaseNode,
Documentation,
Exposure,
GenericTestNode,
GraphMemberNode,
Group,
Macro,
ManifestNode,
Metric,
ModelNode,
DeferRelation,
ResultNode,
SemanticModel,
Documentation,
SourceDefinition,
GenericTestNode,
Exposure,
Metric,
Group,
UnpatchedSourceDefinition,
ManifestNode,
GraphMemberNode,
ResultNode,
BaseNode,
)
from dbt.contracts.graph.unparsed import SourcePatch, NodeVersion, UnparsedVersion
from dbt.contracts.graph.manifest_upgrade import upgrade_manifest_json
@@ -51,18 +46,16 @@ from dbt.exceptions import (
DuplicateResourceNameError,
DuplicateMacroInPackageError,
DuplicateMaterializationNameError,
AmbiguousResourceNameRefError,
)
from dbt.helper_types import PathSet
from dbt.events.functions import fire_event
from dbt.events.types import MergedFromState, UnpinnedRefNewVersionAvailable
from dbt.events.contextvars import get_node_info
from dbt.node_types import NodeType, AccessType
from dbt.node_types import NodeType
from dbt.flags import get_flags, MP_CONTEXT
from dbt import tracking
import dbt.utils
NodeEdgeMap = Dict[str, List[str]]
PackageName = str
DocName = str
@@ -156,55 +149,38 @@ class RefableLookup(dbtClassMixin):
_lookup_types: ClassVar[set] = set(NodeType.refable())
_versioned_types: ClassVar[set] = set(NodeType.versioned())
# refables are actually unique, so the Dict[PackageName, UniqueID] will
# only ever have exactly one value, but doing 3 dict lookups instead of 1
# is not a big deal at all and retains consistency
def __init__(self, manifest: "Manifest"):
self.storage: Dict[str, Dict[PackageName, UniqueID]] = {}
self.populate(manifest)
def get_unique_id(
self,
key: str,
package: Optional[PackageName],
version: Optional[NodeVersion],
node: Optional[GraphMemberNode] = None,
):
def get_unique_id(self, key, package: Optional[PackageName], version: Optional[NodeVersion]):
if version:
key = f"{key}.v{version}"
unique_ids = self._find_unique_ids_for_package(key, package)
if len(unique_ids) > 1:
raise AmbiguousResourceNameRefError(key, unique_ids, node)
else:
return unique_ids[0] if unique_ids else None
return find_unique_id_for_package(self.storage, key, package)
def find(
self,
key: str,
key,
package: Optional[PackageName],
version: Optional[NodeVersion],
manifest: "Manifest",
source_node: Optional[GraphMemberNode] = None,
):
unique_id = self.get_unique_id(key, package, version, source_node)
unique_id = self.get_unique_id(key, package, version)
if unique_id is not None:
node = self.perform_lookup(unique_id, manifest)
# If this is an unpinned ref (no 'version' arg was passed),
# AND this is a versioned node,
# AND this ref is being resolved at runtime -- get_node_info != {}
# Only ModelNodes can be versioned.
if (
isinstance(node, ModelNode)
and version is None
and node.is_versioned
and get_node_info()
):
if version is None and node.is_versioned and get_node_info():
# Check to see if newer versions are available, and log an "FYI" if so
max_version: UnparsedVersion = max(
[
UnparsedVersion(v.version)
for v in manifest.nodes.values()
if isinstance(v, ModelNode)
and v.name == node.name
and v.version is not None
if v.name == node.name and v.version is not None
]
)
assert node.latest_version is not None # for mypy, whenever i may find it
@@ -241,29 +217,11 @@ class RefableLookup(dbtClassMixin):
self.add_node(node)
def perform_lookup(self, unique_id: UniqueID, manifest) -> ManifestNode:
if unique_id in manifest.nodes:
node = manifest.nodes[unique_id]
else:
if unique_id not in manifest.nodes:
raise dbt.exceptions.DbtInternalError(
f"Node {unique_id} found in cache but not found in manifest"
)
return node
def _find_unique_ids_for_package(self, key, package: Optional[PackageName]) -> List[str]:
if key not in self.storage:
return []
pkg_dct: Mapping[PackageName, UniqueID] = self.storage[key]
if package is None:
if not pkg_dct:
return []
else:
return list(pkg_dct.values())
elif package in pkg_dct:
return [pkg_dct[package]]
else:
return []
return manifest.nodes[unique_id]
class MetricLookup(dbtClassMixin):
@@ -299,49 +257,6 @@ class MetricLookup(dbtClassMixin):
return manifest.metrics[unique_id]
class SemanticModelByMeasureLookup(dbtClassMixin):
"""Lookup utility for finding SemanticModel by measure
This is possible because measure names are supposed to be unique across
the semantic models in a manifest.
"""
def __init__(self, manifest: "Manifest"):
self.storage: DefaultDict[str, Dict[PackageName, UniqueID]] = defaultdict(dict)
self.populate(manifest)
def get_unique_id(self, search_name: str, package: Optional[PackageName]):
return find_unique_id_for_package(self.storage, search_name, package)
def find(
self, search_name: str, package: Optional[PackageName], manifest: "Manifest"
) -> Optional[SemanticModel]:
"""Tries to find a SemanticModel based on a measure name"""
unique_id = self.get_unique_id(search_name, package)
if unique_id is not None:
return self.perform_lookup(unique_id, manifest)
return None
def add(self, semantic_model: SemanticModel):
"""Sets all measures for a SemanticModel as paths to the SemanticModel's `unique_id`"""
for measure in semantic_model.measures:
self.storage[measure.name][semantic_model.package_name] = semantic_model.unique_id
def populate(self, manifest: "Manifest"):
"""Populate storage with all the measure + package paths to the Manifest's SemanticModels"""
for semantic_model in manifest.semantic_models.values():
self.add(semantic_model=semantic_model)
def perform_lookup(self, unique_id: UniqueID, manifest: "Manifest") -> SemanticModel:
"""Tries to get a SemanticModel from the Manifest"""
semantic_model = manifest.semantic_models.get(unique_id)
if semantic_model is None:
raise dbt.exceptions.DbtInternalError(
f"Semantic model `{unique_id}` found in cache but not found in manifest"
)
return semantic_model
# This handles both models/seeds/snapshots and sources/metrics/exposures
class DisabledLookup(dbtClassMixin):
def __init__(self, manifest: "Manifest"):
@@ -388,7 +303,7 @@ class AnalysisLookup(RefableLookup):
_versioned_types: ClassVar[set] = set()
def _packages_to_search(
def _search_packages(
current_project: str,
node_package: str,
target_package: Optional[str] = None,
@@ -408,16 +323,10 @@ class ManifestMetadata(BaseArtifactMetadata):
dbt_schema_version: str = field(
default_factory=lambda: str(WritableManifest.dbt_schema_version)
)
project_name: Optional[str] = field(
default=None,
metadata={
"description": "Name of the root project",
},
)
project_id: Optional[str] = field(
default=None,
metadata={
"description": "A unique identifier for the project, hashed from the project name",
"description": "A unique identifier for the project",
},
)
user_id: Optional[UUID] = field(
@@ -471,7 +380,7 @@ def build_node_edges(nodes: List[ManifestNode]):
forward_edges: Dict[str, List[str]] = {n.unique_id: [] for n in nodes}
for node in nodes:
backward_edges[node.unique_id] = node.depends_on_nodes[:]
for unique_id in backward_edges[node.unique_id]:
for unique_id in node.depends_on_nodes:
if unique_id in forward_edges.keys():
forward_edges[unique_id].append(node.unique_id)
return _sort_values(forward_edges), _sort_values(backward_edges)
@@ -615,6 +524,25 @@ MaybeNonSource = Optional[Union[ManifestNode, Disabled[ManifestNode]]]
T = TypeVar("T", bound=GraphMemberNode)
def _update_into(dest: MutableMapping[str, T], new_item: T):
"""Update dest to overwrite whatever is at dest[new_item.unique_id] with
new_itme. There must be an existing value to overwrite, and the two nodes
must have the same original file path.
"""
unique_id = new_item.unique_id
if unique_id not in dest:
raise dbt.exceptions.DbtRuntimeError(
f"got an update_{new_item.resource_type} call with an "
f"unrecognized {new_item.resource_type}: {new_item.unique_id}"
)
existing = dest[unique_id]
if new_item.original_file_path != existing.original_file_path:
raise dbt.exceptions.DbtRuntimeError(
f"cannot update a {new_item.resource_type} to have a new file path!"
)
dest[unique_id] = new_item
# This contains macro methods that are in both the Manifest
# and the MacroManifest
class MacroMethods:
@@ -647,36 +575,26 @@ class MacroMethods:
return candidates.last()
def find_generate_macro_by_name(
self, component: str, root_project_name: str, imported_package: Optional[str] = None
self, component: str, root_project_name: str
) -> Optional[Macro]:
"""
The default `generate_X_name` macros are similar to regular ones, but only
includes imported packages when searching for a package.
- if package is not provided:
The `generate_X_name` macros are similar to regular ones, but ignore
imported packages.
- if there is a `generate_{component}_name` macro in the root
project, return it
- return the `generate_{component}_name` macro from the 'dbt'
internal project
- if package is provided
- return the `generate_{component}_name` macro from the imported
package, if one exists
"""
def filter(candidate: MacroCandidate) -> bool:
if imported_package:
return (
candidate.locality == Locality.Imported
and imported_package == candidate.macro.package_name
)
else:
return candidate.locality != Locality.Imported
return candidate.locality != Locality.Imported
candidates: CandidateList = self._find_macros_by_name(
name=f"generate_{component}_name",
root_project_name=root_project_name,
# filter out imported packages
filter=filter,
)
return candidates.last()
def _find_macros_by_name(
@@ -741,7 +659,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
source_patches: MutableMapping[SourceKey, SourcePatch] = field(default_factory=dict)
disabled: MutableMapping[str, List[GraphMemberNode]] = field(default_factory=dict)
env_vars: MutableMapping[str, str] = field(default_factory=dict)
semantic_models: MutableMapping[str, SemanticModel] = field(default_factory=dict)
_doc_lookup: Optional[DocLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
@@ -755,9 +672,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
_metric_lookup: Optional[MetricLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
)
_semantic_model_by_measure_lookup: Optional[SemanticModelByMeasureLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
)
_disabled_lookup: Optional[DisabledLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
)
@@ -784,6 +698,18 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
obj._lock = MP_CONTEXT.Lock()
return obj
def update_exposure(self, new_exposure: Exposure):
_update_into(self.exposures, new_exposure)
def update_metric(self, new_metric: Metric):
_update_into(self.metrics, new_metric)
def update_node(self, new_node: ManifestNode):
_update_into(self.nodes, new_node)
def update_source(self, new_source: SourceDefinition):
_update_into(self.sources, new_source)
def build_flat_graph(self):
"""This attribute is used in context.common by each node, so we want to
only build it once and avoid any concurrency issues around it.
@@ -796,9 +722,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
"metrics": {k: v.to_dict(omit_none=False) for k, v in self.metrics.items()},
"nodes": {k: v.to_dict(omit_none=False) for k, v in self.nodes.items()},
"sources": {k: v.to_dict(omit_none=False) for k, v in self.sources.items()},
"semantic_models": {
k: v.to_dict(omit_none=False) for k, v in self.semantic_models.items()
},
}
def build_disabled_by_file_id(self):
@@ -859,7 +782,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.nodes.values(),
self.sources.values(),
self.metrics.values(),
self.semantic_models.values(),
)
for resource in all_resources:
resource_type_plural = resource.resource_type.pluralize()
@@ -894,7 +816,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
disabled={k: _deepcopy(v) for k, v in self.disabled.items()},
files={k: _deepcopy(v) for k, v in self.files.items()},
state_check=_deepcopy(self.state_check),
semantic_models={k: _deepcopy(v) for k, v in self.semantic_models.items()},
)
copy.build_flat_graph()
return copy
@@ -906,7 +827,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.sources.values(),
self.exposures.values(),
self.metrics.values(),
self.semantic_models.values(),
)
)
forward_edges, backward_edges = build_node_edges(edge_members)
@@ -933,10 +853,17 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
group_map = {group.name: [] for group in self.groups.values()}
for node in groupable_nodes:
if node.group is not None:
group_map[node.group].append(node.unique_id)
# group updates are not included with state:modified and
# by ignoring the groups that aren't in the group map we
# can avoid hitting errors for groups that are not getting
# updated. This is a hack but any groups that are not
# valid will be caught in
# parser.manifest.ManifestLoader.check_valid_group_config_node
if node.group in group_map:
group_map[node.group].append(node.unique_id)
self.group_map = group_map
def writable_manifest(self) -> "WritableManifest":
def writable_manifest(self):
self.build_parent_and_child_maps()
self.build_group_map()
return WritableManifest(
@@ -953,7 +880,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
child_map=self.child_map,
parent_map=self.parent_map,
group_map=self.group_map,
semantic_models=self.semantic_models,
)
def write(self, path):
@@ -970,8 +896,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
return self.exposures[unique_id]
elif unique_id in self.metrics:
return self.metrics[unique_id]
elif unique_id in self.semantic_models:
return self.semantic_models[unique_id]
else:
# something terrible has happened
raise dbt.exceptions.DbtInternalError(
@@ -1008,13 +932,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self._metric_lookup = MetricLookup(self)
return self._metric_lookup
@property
def semantic_model_by_measure_lookup(self) -> SemanticModelByMeasureLookup:
"""Gets (and creates if necessary) the lookup utility for getting SemanticModels by measures"""
if self._semantic_model_by_measure_lookup is None:
self._semantic_model_by_measure_lookup = SemanticModelByMeasureLookup(self)
return self._semantic_model_by_measure_lookup
def rebuild_ref_lookup(self):
self._ref_lookup = RefableLookup(self)
@@ -1033,34 +950,10 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self._analysis_lookup = AnalysisLookup(self)
return self._analysis_lookup
@property
def external_node_unique_ids(self):
return [node.unique_id for node in self.nodes.values() if node.is_external_node]
def resolve_refs(
self,
source_node: ModelNode,
current_project: str, # TODO: ModelNode is overly restrictive typing
) -> List[MaybeNonSource]:
resolved_refs: List[MaybeNonSource] = []
for ref in source_node.refs:
resolved = self.resolve_ref(
source_node,
ref.name,
ref.package,
ref.version,
current_project,
source_node.package_name,
)
resolved_refs.append(resolved)
return resolved_refs
# Called by dbt.parser.manifest._process_refs_for_exposure, _process_refs_for_metric,
# Called by dbt.parser.manifest._resolve_refs_for_exposure
# and dbt.parser.manifest._process_refs_for_node
def resolve_ref(
self,
source_node: GraphMemberNode,
target_model_name: str,
target_model_package: Optional[str],
target_model_version: Optional[NodeVersion],
@@ -1071,13 +964,11 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
node: Optional[ManifestNode] = None
disabled: Optional[List[ManifestNode]] = None
candidates = _packages_to_search(current_project, node_package, target_model_package)
candidates = _search_packages(current_project, node_package, target_model_package)
for pkg in candidates:
node = self.ref_lookup.find(
target_model_name, pkg, target_model_version, self, source_node
)
node = self.ref_lookup.find(target_model_name, pkg, target_model_version, self)
if node is not None and hasattr(node, "config") and node.config.enabled:
if node is not None and node.config.enabled:
return node
# it's possible that the node is disabled
@@ -1098,7 +989,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
node_package: str,
) -> MaybeParsedSource:
search_name = f"{target_source_name}.{target_table_name}"
candidates = _packages_to_search(current_project, node_package)
candidates = _search_packages(current_project, node_package)
source: Optional[SourceDefinition] = None
disabled: Optional[List[SourceDefinition]] = None
@@ -1128,7 +1019,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
metric: Optional[Metric] = None
disabled: Optional[List[Metric]] = None
candidates = _packages_to_search(current_project, node_package, target_metric_package)
candidates = _search_packages(current_project, node_package, target_metric_package)
for pkg in candidates:
metric = self.metric_lookup.find(target_metric_name, pkg, self)
@@ -1142,25 +1033,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
return Disabled(disabled[0])
return None
def resolve_semantic_model_for_measure(
self,
target_measure_name: str,
current_project: str,
node_package: str,
target_package: Optional[str] = None,
) -> Optional[SemanticModel]:
"""Tries to find the SemanticModel that a measure belongs to"""
candidates = _packages_to_search(current_project, node_package, target_package)
for pkg in candidates:
semantic_model = self.semantic_model_by_measure_lookup.find(
target_measure_name, pkg, self
)
if semantic_model is not None:
return semantic_model
return None
# Called by DocsRuntimeContext.doc
def resolve_doc(
self,
@@ -1173,7 +1045,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
resolve_ref except the is_enabled checks are unnecessary as docs are
always enabled.
"""
candidates = _packages_to_search(current_project, node_package, package)
candidates = _search_packages(current_project, node_package, package)
for pkg in candidates:
result = self.doc_lookup.find(name, pkg, self)
@@ -1181,50 +1053,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
return result
return None
def is_invalid_private_ref(
self, node: GraphMemberNode, target_model: MaybeNonSource, dependencies: Optional[Mapping]
) -> bool:
dependencies = dependencies or {}
if not isinstance(target_model, ModelNode):
return False
is_private_ref = (
target_model.access == AccessType.Private
# don't raise this reference error for ad hoc 'preview' queries
and node.resource_type != NodeType.SqlOperation
and node.resource_type != NodeType.RPCCall # TODO: rm
)
target_dependency = dependencies.get(target_model.package_name)
restrict_package_access = target_dependency.restrict_access if target_dependency else False
# TODO: SemanticModel and SourceDefinition do not have group, and so should not be able to make _any_ private ref.
return is_private_ref and (
not hasattr(node, "group")
or not node.group
or node.group != target_model.group
or restrict_package_access
)
def is_invalid_protected_ref(
self, node: GraphMemberNode, target_model: MaybeNonSource, dependencies: Optional[Mapping]
) -> bool:
dependencies = dependencies or {}
if not isinstance(target_model, ModelNode):
return False
is_protected_ref = (
target_model.access == AccessType.Protected
# don't raise this reference error for ad hoc 'preview' queries
and node.resource_type != NodeType.SqlOperation
and node.resource_type != NodeType.RPCCall # TODO: rm
)
target_dependency = dependencies.get(target_model.package_name)
restrict_package_access = target_dependency.restrict_access if target_dependency else False
return is_protected_ref and (
node.package_name != target_model.package_name and restrict_package_access
)
# Called by RunTask.defer_to_manifest
def merge_from_artifact(
self,
@@ -1262,25 +1090,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
sample = list(islice(merged, 5))
fire_event(MergedFromState(num_merged=len(merged), sample=sample))
# Called by CloneTask.defer_to_manifest
def add_from_artifact(
self,
other: "WritableManifest",
) -> None:
"""Update this manifest by *adding* information about each node's location
in the other manifest.
Only non-ephemeral refable nodes are examined.
"""
refables = set(NodeType.refable())
for unique_id, node in other.nodes.items():
current = self.nodes.get(unique_id)
if current and (node.resource_type in refables and not node.is_ephemeral):
defer_relation = DeferRelation(
node.database, node.schema, node.alias, node.relation_name
)
self.nodes[unique_id] = current.replace(defer_relation=defer_relation)
# Methods that were formerly in ParseResult
def add_macro(self, source_file: SourceFile, macro: Macro):
@@ -1366,11 +1175,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.docs[doc.unique_id] = doc
source_file.docs.append(doc.unique_id)
def add_semantic_model(self, source_file: SchemaSourceFile, semantic_model: SemanticModel):
_check_duplicates(semantic_model, self.semantic_models)
self.semantic_models[semantic_model.unique_id] = semantic_model
source_file.semantic_models.append(semantic_model.unique_id)
# end of methods formerly in ParseResult
# Provide support for copy.deepcopy() - we just need to avoid the lock!
@@ -1397,12 +1201,10 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.source_patches,
self.disabled,
self.env_vars,
self.semantic_models,
self._doc_lookup,
self._source_lookup,
self._ref_lookup,
self._metric_lookup,
self._semantic_model_by_measure_lookup,
self._disabled_lookup,
self._analysis_lookup,
)
@@ -1422,7 +1224,7 @@ AnyManifest = Union[Manifest, MacroManifest]
@dataclass
@schema_version("manifest", 10)
@schema_version("manifest", 9)
class WritableManifest(ArtifactMixin):
nodes: Mapping[UniqueID, ManifestNode] = field(
metadata=dict(description=("The nodes defined in the dbt project and its dependencies"))
@@ -1468,9 +1270,6 @@ class WritableManifest(ArtifactMixin):
description="A mapping from group names to their nodes",
)
)
semantic_models: Mapping[UniqueID, SemanticModel] = field(
metadata=dict(description=("The semantic models defined in the dbt project"))
)
metadata: ManifestMetadata = field(
metadata=dict(
description="Metadata about the manifest",
@@ -1485,24 +1284,20 @@ class WritableManifest(ArtifactMixin):
("manifest", 6),
("manifest", 7),
("manifest", 8),
("manifest", 9),
]
@classmethod
def upgrade_schema_version(cls, data):
"""This overrides the "upgrade_schema_version" call in VersionedSchema (via
ArtifactMixin) to modify the dictionary passed in from earlier versions of the manifest."""
manifest_schema_version = get_manifest_schema_version(data)
if manifest_schema_version <= 9:
data = upgrade_manifest_json(data, manifest_schema_version)
if get_manifest_schema_version(data) <= 8:
data = upgrade_manifest_json(data)
return cls.from_dict(data)
def __post_serialize__(self, dct):
for unique_id, node in dct["nodes"].items():
if "config_call_dict" in node:
del node["config_call_dict"]
if "defer_relation" in node:
del node["defer_relation"]
return dct

View File

@@ -1,3 +1,42 @@
from dbt import deprecations
from dbt.dataclass_schema import ValidationError
# we renamed these properties in v1.3
# this method allows us to be nice to the early adopters
def rename_metric_attr(data: dict, raise_deprecation_warning: bool = False) -> dict:
metric_name = data["name"]
if raise_deprecation_warning and (
"sql" in data.keys()
or "type" in data.keys()
or data.get("calculation_method") == "expression"
):
deprecations.warn("metric-attr-renamed", metric_name=metric_name)
duplicated_attribute_msg = """\n
The metric '{}' contains both the deprecated metric property '{}'
and the up-to-date metric property '{}'. Please remove the deprecated property.
"""
if "sql" in data.keys():
if "expression" in data.keys():
raise ValidationError(
duplicated_attribute_msg.format(metric_name, "sql", "expression")
)
else:
data["expression"] = data.pop("sql")
if "type" in data.keys():
if "calculation_method" in data.keys():
raise ValidationError(
duplicated_attribute_msg.format(metric_name, "type", "calculation_method")
)
else:
calculation_method = data.pop("type")
data["calculation_method"] = calculation_method
# we also changed "type: expression" -> "calculation_method: derived"
if data.get("calculation_method") == "expression":
data["calculation_method"] = "derived"
return data
def rename_sql_attr(node_content: dict) -> dict:
if "raw_sql" in node_content:
node_content["raw_code"] = node_content.pop("raw_sql")
@@ -49,24 +88,7 @@ def upgrade_seed_content(node_content):
node_content.get("depends_on", {}).pop("nodes", None)
def drop_v9_and_prior_metrics(manifest: dict) -> None:
manifest["metrics"] = {}
filtered_disabled_entries = {}
for entry_name, resource_list in manifest.get("disabled", {}).items():
filtered_resource_list = []
for resource in resource_list:
if resource.get("resource_type") != "metric":
filtered_resource_list.append(resource)
filtered_disabled_entries[entry_name] = filtered_resource_list
manifest["disabled"] = filtered_disabled_entries
def upgrade_manifest_json(manifest: dict, manifest_schema_version: int) -> dict:
# this should remain 9 while the check in `upgrade_schema_version` may change
if manifest_schema_version <= 9:
drop_v9_and_prior_metrics(manifest=manifest)
def upgrade_manifest_json(manifest: dict) -> dict:
for node_content in manifest.get("nodes", {}).values():
upgrade_node_content(node_content)
if node_content["resource_type"] == "seed":
@@ -85,6 +107,7 @@ def upgrade_manifest_json(manifest: dict, manifest_schema_version: int) -> dict:
manifest["group_map"] = {}
for metric_content in manifest.get("metrics", {}).values():
# handle attr renames + value translation ("expression" -> "derived")
metric_content = rename_metric_attr(metric_content)
metric_content = upgrade_ref_content(metric_content)
if "root_path" in metric_content:
del metric_content["root_path"]
@@ -102,6 +125,4 @@ def upgrade_manifest_json(manifest: dict, manifest_schema_version: int) -> dict:
if "root_path" in doc_content:
del doc_content["root_path"]
doc_content["resource_type"] = "doc"
if "semantic_models" not in manifest:
manifest["semantic_models"] = {}
return manifest

View File

@@ -2,17 +2,15 @@ from dataclasses import field, Field, dataclass
from enum import Enum
from itertools import chain
from typing import Any, List, Optional, Dict, Union, Type, TypeVar, Callable
from dbt.dataclass_schema import (
dbtClassMixin,
ValidationError,
register_pattern,
StrEnum,
)
from dbt.contracts.graph.unparsed import AdditionalPropertiesAllowed, Docs
from dbt.contracts.graph.utils import validate_color
from dbt.contracts.util import Replaceable, list_str
from dbt.exceptions import DbtInternalError, CompilationError
from dbt.contracts.util import Replaceable, list_str
from dbt import hooks
from dbt.node_types import NodeType
@@ -191,16 +189,6 @@ class Severity(str):
register_pattern(Severity, insensitive_patterns("warn", "error"))
class OnConfigurationChangeOption(StrEnum):
Apply = "apply"
Continue = "continue"
Fail = "fail"
@classmethod
def default(cls) -> "OnConfigurationChangeOption":
return cls.Apply
@dataclass
class ContractConfig(dbtClassMixin, Replaceable):
enforced: bool = False
@@ -299,17 +287,11 @@ class BaseConfig(AdditionalPropertiesAllowed, Replaceable):
return False
return True
# This is used in 'add_config_call' to create the combined config_call_dict.
# This is used in 'add_config_call' to created the combined config_call_dict.
# 'meta' moved here from node
mergebehavior = {
"append": ["pre-hook", "pre_hook", "post-hook", "post_hook", "tags"],
"update": [
"quoting",
"column_types",
"meta",
"docs",
"contract",
],
"update": ["quoting", "column_types", "meta", "docs", "contract"],
"dict_key_append": ["grants"],
}
@@ -386,11 +368,6 @@ class BaseConfig(AdditionalPropertiesAllowed, Replaceable):
return self.from_dict(dct)
@dataclass
class SemanticModelConfig(BaseConfig):
enabled: bool = True
@dataclass
class MetricConfig(BaseConfig):
enabled: bool = True
@@ -468,9 +445,6 @@ class NodeConfig(NodeAndTestConfig):
# sometimes getting the Union order wrong, causing serialization failures.
unique_key: Union[str, List[str], None] = None
on_schema_change: Optional[str] = "ignore"
on_configuration_change: OnConfigurationChangeOption = field(
default_factory=OnConfigurationChangeOption.default
)
grants: Dict[str, Any] = field(
default_factory=dict, metadata=MergeBehavior.DictKeyAppend.meta()
)
@@ -555,8 +529,6 @@ class SeedConfig(NodeConfig):
@dataclass
class TestConfig(NodeAndTestConfig):
__test__ = False
# this is repeated because of a different default
schema: Optional[str] = field(
default="dbt_test__audit",
@@ -619,6 +591,8 @@ class SnapshotConfig(EmptySnapshotConfig):
@classmethod
def validate(cls, data):
super().validate(data)
# Note: currently you can't just set these keys in schema.yml because this validation
# will fail when parsing the snapshot node.
if not data.get("strategy") or not data.get("unique_key") or not data.get("target_schema"):
raise ValidationError(
"Snapshots must be configured with a 'strategy', 'unique_key', "
@@ -649,6 +623,7 @@ class SnapshotConfig(EmptySnapshotConfig):
if data.get("materialized") and data.get("materialized") != "snapshot":
raise ValidationError("A snapshot must have a materialized value of 'snapshot'")
# Called by "calculate_node_config_dict" in ContextConfigGenerator
def finalize_and_validate(self):
data = self.to_dict(omit_none=True)
self.validate(data)

View File

@@ -1,31 +0,0 @@
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional, List
from dbt.contracts.graph.unparsed import NodeVersion
from dbt.node_types import NodeType, AccessType
@dataclass
class ModelNodeArgs:
name: str
package_name: str
identifier: str
schema: str
database: Optional[str] = None
relation_name: Optional[str] = None
version: Optional[NodeVersion] = None
latest_version: Optional[NodeVersion] = None
deprecation_date: Optional[datetime] = None
access: Optional[str] = AccessType.Protected.value
generated_at: datetime = field(default_factory=datetime.utcnow)
depends_on_nodes: List[str] = field(default_factory=list)
enabled: bool = True
@property
def unique_id(self) -> str:
unique_id = f"{NodeType.Model}.{self.package_name}.{self.name}"
if self.version:
unique_id = f"{unique_id}.v{self.version}"
return unique_id

View File

@@ -1,24 +1,25 @@
import os
from datetime import datetime
import time
from dataclasses import dataclass, field
from enum import Enum
import hashlib
from mashumaro.types import SerializableType
from typing import Optional, Union, List, Dict, Any, Sequence, Tuple, Iterator
from typing import (
Optional,
Union,
List,
Dict,
Any,
Sequence,
Tuple,
Iterator,
)
from dbt.dataclass_schema import dbtClassMixin, ExtensibleDbtClassMixin
from dbt.clients.system import write_file
from dbt.contracts.files import FileHash
from dbt.contracts.graph.semantic_models import (
Defaults,
Dimension,
Entity,
Measure,
SourceFileMetadata,
)
from dbt.contracts.graph.unparsed import (
Docs,
ExposureType,
@@ -27,6 +28,8 @@ from dbt.contracts.graph.unparsed import (
HasYamlMetadata,
MacroArgument,
MaturityType,
MetricFilter,
MetricTime,
Owner,
Quoting,
TestDef,
@@ -35,29 +38,19 @@ from dbt.contracts.graph.unparsed import (
UnparsedSourceTableDefinition,
UnparsedColumn,
)
from dbt.contracts.graph.node_args import ModelNodeArgs
from dbt.contracts.util import Replaceable, AdditionalPropertiesMixin
from dbt.events.functions import warn_or_error
from dbt.exceptions import ParsingError, ContractBreakingChangeError
from dbt.exceptions import ParsingError, InvalidAccessTypeError, ContractBreakingChangeError
from dbt.events.types import (
SeedIncreased,
SeedExceedsLimitSamePath,
SeedExceedsLimitAndPathChanged,
SeedExceedsLimitChecksumChanged,
ValidationWarning,
)
from dbt.events.contextvars import set_log_contextvars
from dbt.flags import get_flags
from dbt.node_types import ModelLanguage, NodeType, AccessType
from dbt_semantic_interfaces.call_parameter_sets import FilterCallParameterSets
from dbt_semantic_interfaces.references import (
MeasureReference,
LinkableElementReference,
SemanticModelReference,
TimeDimensionReference,
)
from dbt_semantic_interfaces.references import MetricReference as DSIMetricReference
from dbt_semantic_interfaces.type_enums import MetricType, TimeGranularity
from dbt_semantic_interfaces.parsing.where_filter_parser import WhereFilterParser
from .model_config import (
NodeConfig,
@@ -68,7 +61,6 @@ from .model_config import (
ExposureConfig,
EmptySnapshotConfig,
SnapshotConfig,
SemanticModelConfig,
)
@@ -261,16 +253,6 @@ class MacroDependsOn(dbtClassMixin, Replaceable):
self.macros.append(value)
@dataclass
class DeferRelation(HasRelationMetadata):
alias: str
relation_name: Optional[str]
@property
def identifier(self):
return self.alias
@dataclass
class DependsOn(MacroDependsOn):
nodes: List[str] = field(default_factory=list)
@@ -455,17 +437,74 @@ class ParsedNode(NodeInfoMixin, ParsedNodeMandatory, SerializableType):
def build_contract_checksum(self):
pass
def same_contract(self, old, adapter_type=None) -> bool:
def same_contract(self, old) -> bool:
# This would only apply to seeds
return True
def same_contents(self, old, adapter_type) -> bool:
def patch(self, patch: "ParsedNodePatch"):
"""Given a ParsedNodePatch, add the new information to the node."""
# NOTE: Constraint patching is awkwardly done in the parse_patch function
# which calls this one. We need to combine the logic.
# explicitly pick out the parts to update so we don't inadvertently
# step on the model name or anything
# Note: config should already be updated
self.patch_path: Optional[str] = patch.file_id
# update created_at so process_docs will run in partial parsing
self.created_at = time.time()
self.description = patch.description
self.columns = patch.columns
self.name = patch.name
# TODO: version, latest_version, and access are specific to ModelNodes, consider splitting out to ModelNode
if self.resource_type != NodeType.Model:
if patch.version:
warn_or_error(
ValidationWarning(
field_name="version",
resource_type=self.resource_type.value,
node_name=patch.name,
)
)
if patch.latest_version:
warn_or_error(
ValidationWarning(
field_name="latest_version",
resource_type=self.resource_type.value,
node_name=patch.name,
)
)
self.version = patch.version
self.latest_version = patch.latest_version
# This might not be the ideal place to validate the "access" field,
# but at this point we have the information we need to properly
# validate and we don't before this.
if patch.access:
if self.resource_type == NodeType.Model:
if AccessType.is_valid(patch.access):
self.access = AccessType(patch.access)
else:
raise InvalidAccessTypeError(
unique_id=self.unique_id,
field_value=patch.access,
)
else:
warn_or_error(
ValidationWarning(
field_name="access",
resource_type=self.resource_type.value,
node_name=patch.name,
)
)
def same_contents(self, old) -> bool:
if old is None:
return False
# Need to ensure that same_contract is called because it
# could throw an error
same_contract = self.same_contract(old, adapter_type)
same_contract = self.same_contract(old)
return (
self.same_body(old)
and self.same_config(old)
@@ -476,10 +515,6 @@ class ParsedNode(NodeInfoMixin, ParsedNodeMandatory, SerializableType):
and True
)
@property
def is_external_node(self):
return False
@dataclass
class InjectedCTE(dbtClassMixin, Replaceable):
@@ -546,6 +581,81 @@ class CompiledNode(ParsedNode):
def depends_on_macros(self):
return self.depends_on.macros
def build_contract_checksum(self):
# We don't need to construct the checksum if the model does not
# have contract enforced, because it won't be used.
# This needs to be executed after contract config is set
# Avoid rebuilding the checksum if it has already been set.
if self.contract.checksum is not None:
return
if self.contract.enforced is True:
contract_state = ""
# We need to sort the columns so that order doesn't matter
# columns is a str: ColumnInfo dictionary
sorted_columns = sorted(self.columns.values(), key=lambda col: col.name)
for column in sorted_columns:
contract_state += f"|{column.name}"
contract_state += str(column.data_type)
data = contract_state.encode("utf-8")
self.contract.checksum = hashlib.new("sha256", data).hexdigest()
def same_contract(self, old) -> bool:
# If the contract wasn't previously enforced:
if old.contract.enforced is False and self.contract.enforced is False:
# No change -- same_contract: True
return True
if old.contract.enforced is False and self.contract.enforced is True:
# Now it's enforced. This is a change, but not a breaking change -- same_contract: False
return False
# Otherwise: The contract was previously enforced, and we need to check for changes.
# Happy path: The contract is still being enforced, and the checksums are identical.
if self.contract.enforced is True and self.contract.checksum == old.contract.checksum:
# No change -- same_contract: True
return True
# Otherwise: There has been a change.
# We need to determine if it is a **breaking** change.
# These are the categories of breaking changes:
contract_enforced_disabled: bool = False
columns_removed: List[str] = []
column_type_changes: List[Tuple[str, str, str]] = []
if old.contract.enforced is True and self.contract.enforced is False:
# Breaking change: the contract was previously enforced, and it no longer is
contract_enforced_disabled = True
# Next, compare each column from the previous contract (old.columns)
for key, value in sorted(old.columns.items()):
# Has this column been removed?
if key not in self.columns.keys():
columns_removed.append(value.name)
# Has this column's data type changed?
elif value.data_type != self.columns[key].data_type:
column_type_changes.append(
(str(value.name), str(value.data_type), str(self.columns[key].data_type))
)
# If a column has been added, it will be missing in the old.columns, and present in self.columns
# That's a change (caught by the different checksums), but not a breaking change
# Did we find any changes that we consider breaking? If so, that's an error
if contract_enforced_disabled or columns_removed or column_type_changes:
raise (
ContractBreakingChangeError(
contract_enforced_disabled=contract_enforced_disabled,
columns_removed=columns_removed,
column_type_changes=column_type_changes,
node=self,
)
)
# Otherwise, though we didn't find any *breaking* changes, the contract has still changed -- same_contract: False
else:
return False
# ====================================
# CompiledNode subclasses
@@ -570,45 +680,6 @@ class ModelNode(CompiledNode):
constraints: List[ModelLevelConstraint] = field(default_factory=list)
version: Optional[NodeVersion] = None
latest_version: Optional[NodeVersion] = None
deprecation_date: Optional[datetime] = None
defer_relation: Optional[DeferRelation] = None
@classmethod
def from_args(cls, args: ModelNodeArgs) -> "ModelNode":
unique_id = args.unique_id
# build unrendered config -- for usage in ParsedNode.same_contents
unrendered_config = {}
unrendered_config["alias"] = args.identifier
unrendered_config["schema"] = args.schema
if args.database:
unrendered_config["database"] = args.database
return cls(
resource_type=NodeType.Model,
name=args.name,
package_name=args.package_name,
unique_id=unique_id,
fqn=[args.package_name, args.name],
version=args.version,
latest_version=args.latest_version,
relation_name=args.relation_name,
database=args.database,
schema=args.schema,
alias=args.identifier,
deprecation_date=args.deprecation_date,
checksum=FileHash.from_contents(f"{unique_id},{args.generated_at}"),
access=AccessType(args.access),
original_file_path="",
path="",
unrendered_config=unrendered_config,
depends_on=DependsOn(nodes=args.depends_on_nodes),
config=NodeConfig(enabled=args.enabled),
)
@property
def is_external_node(self) -> bool:
return not self.original_file_path and not self.path
@property
def is_latest_version(self) -> bool:
@@ -621,155 +692,6 @@ class ModelNode(CompiledNode):
else:
return f"{self.name}.v{self.version}"
@property
def materialization_enforces_constraints(self) -> bool:
return self.config.materialized in ["table", "incremental"]
def build_contract_checksum(self):
# We don't need to construct the checksum if the model does not
# have contract enforced, because it won't be used.
# This needs to be executed after contract config is set
# Avoid rebuilding the checksum if it has already been set.
if self.contract.checksum is not None:
return
if self.contract.enforced is True:
contract_state = ""
# We need to sort the columns so that order doesn't matter
# columns is a str: ColumnInfo dictionary
sorted_columns = sorted(self.columns.values(), key=lambda col: col.name)
for column in sorted_columns:
contract_state += f"|{column.name}"
contract_state += str(column.data_type)
contract_state += str(column.constraints)
if self.materialization_enforces_constraints:
contract_state += self.config.materialized
contract_state += str(self.constraints)
data = contract_state.encode("utf-8")
self.contract.checksum = hashlib.new("sha256", data).hexdigest()
def same_contract(self, old, adapter_type=None) -> bool:
# If the contract wasn't previously enforced:
if old.contract.enforced is False and self.contract.enforced is False:
# No change -- same_contract: True
return True
if old.contract.enforced is False and self.contract.enforced is True:
# Now it's enforced. This is a change, but not a breaking change -- same_contract: False
return False
# Otherwise: The contract was previously enforced, and we need to check for changes.
# Happy path: The contract is still being enforced, and the checksums are identical.
if self.contract.enforced is True and self.contract.checksum == old.contract.checksum:
# No change -- same_contract: True
return True
# Otherwise: There has been a change.
# We need to determine if it is a **breaking** change.
# These are the categories of breaking changes:
contract_enforced_disabled: bool = False
columns_removed: List[str] = []
column_type_changes: List[Tuple[str, str, str]] = []
enforced_column_constraint_removed: List[Tuple[str, str]] = [] # column, constraint_type
enforced_model_constraint_removed: List[
Tuple[str, List[str]]
] = [] # constraint_type, columns
materialization_changed: List[str] = []
if old.contract.enforced is True and self.contract.enforced is False:
# Breaking change: the contract was previously enforced, and it no longer is
contract_enforced_disabled = True
# TODO: this avoid the circular imports but isn't ideal
from dbt.adapters.factory import get_adapter_constraint_support
from dbt.adapters.base import ConstraintSupport
constraint_support = get_adapter_constraint_support(adapter_type)
column_constraints_exist = False
# Next, compare each column from the previous contract (old.columns)
for old_key, old_value in sorted(old.columns.items()):
# Has this column been removed?
if old_key not in self.columns.keys():
columns_removed.append(old_value.name)
# Has this column's data type changed?
elif old_value.data_type != self.columns[old_key].data_type:
column_type_changes.append(
(
str(old_value.name),
str(old_value.data_type),
str(self.columns[old_key].data_type),
)
)
# track if there are any column level constraints for the materialization check late
if old_value.constraints:
column_constraints_exist = True
# Have enforced columns level constraints changed?
# Constraints are only enforced for table and incremental materializations.
# We only really care if the old node was one of those materializations for breaking changes
if (
old_key in self.columns.keys()
and old_value.constraints != self.columns[old_key].constraints
and old.materialization_enforces_constraints
):
for old_constraint in old_value.constraints:
if (
old_constraint not in self.columns[old_key].constraints
and constraint_support[old_constraint.type] == ConstraintSupport.ENFORCED
):
enforced_column_constraint_removed.append(
(old_key, str(old_constraint.type))
)
# Now compare the model level constraints
if old.constraints != self.constraints and old.materialization_enforces_constraints:
for old_constraint in old.constraints:
if (
old_constraint not in self.constraints
and constraint_support[old_constraint.type] == ConstraintSupport.ENFORCED
):
enforced_model_constraint_removed.append(
(str(old_constraint.type), old_constraint.columns)
)
# Check for relevant materialization changes.
if (
old.materialization_enforces_constraints
and not self.materialization_enforces_constraints
and (old.constraints or column_constraints_exist)
):
materialization_changed = [old.config.materialized, self.config.materialized]
# If a column has been added, it will be missing in the old.columns, and present in self.columns
# That's a change (caught by the different checksums), but not a breaking change
# Did we find any changes that we consider breaking? If so, that's an error
if (
contract_enforced_disabled
or columns_removed
or column_type_changes
or enforced_model_constraint_removed
or enforced_column_constraint_removed
or materialization_changed
):
raise (
ContractBreakingChangeError(
contract_enforced_disabled=contract_enforced_disabled,
columns_removed=columns_removed,
column_type_changes=column_type_changes,
enforced_column_constraint_removed=enforced_column_constraint_removed,
enforced_model_constraint_removed=enforced_model_constraint_removed,
materialization_changed=materialization_changed,
node=self,
)
)
# Otherwise, though we didn't find any *breaking* changes, the contract has still changed -- same_contract: False
else:
return False
# TODO: rm?
@dataclass
@@ -795,7 +717,6 @@ class SeedNode(ParsedNode): # No SQLDefaults!
# and we need the root_path to load the seed later
root_path: Optional[str] = None
depends_on: MacroDependsOn = field(default_factory=MacroDependsOn)
defer_relation: Optional[DeferRelation] = None
def same_seeds(self, other: "SeedNode") -> bool:
# for seeds, we check the hashes. If the hashes are different types,
@@ -932,8 +853,6 @@ class SingularTestNode(TestShouldStoreFailures, CompiledNode):
@dataclass
class TestMetadata(dbtClassMixin, Replaceable):
__test__ = False
name: str
# kwargs are the args that are left in the test builder after
# removing configs. They are set from the test builder when
@@ -959,7 +878,7 @@ class GenericTestNode(TestShouldStoreFailures, CompiledNode, HasTestMetadata):
config: TestConfig = field(default_factory=TestConfig) # type: ignore
attached_node: Optional[str] = None
def same_contents(self, other, adapter_type: Optional[str]) -> bool:
def same_contents(self, other) -> bool:
if other is None:
return False
@@ -992,7 +911,6 @@ class IntermediateSnapshotNode(CompiledNode):
class SnapshotNode(CompiledNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Snapshot]})
config: SnapshotConfig
defer_relation: Optional[DeferRelation] = None
# ====================================
@@ -1013,6 +931,14 @@ class Macro(BaseNode):
created_at: float = field(default_factory=lambda: time.time())
supported_languages: Optional[List[ModelLanguage]] = None
def patch(self, patch: "ParsedMacroPatch"):
self.patch_path: Optional[str] = patch.file_id
self.description = patch.description
self.created_at = time.time()
self.meta = patch.meta
self.docs = patch.docs
self.arguments = patch.arguments
def same_contents(self, other: Optional["Macro"]) -> bool:
if other is None:
return False
@@ -1305,75 +1231,16 @@ class Exposure(GraphNode):
and True
)
@property
def group(self):
return None
# ====================================
# Metric node
# ====================================
@dataclass
class WhereFilter(dbtClassMixin):
where_sql_template: str
@property
def call_parameter_sets(self) -> FilterCallParameterSets:
return WhereFilterParser.parse_call_parameter_sets(self.where_sql_template)
@dataclass
class MetricInputMeasure(dbtClassMixin):
name: str
filter: Optional[WhereFilter] = None
alias: Optional[str] = None
def measure_reference(self) -> MeasureReference:
return MeasureReference(element_name=self.name)
def post_aggregation_measure_reference(self) -> MeasureReference:
return MeasureReference(element_name=self.alias or self.name)
@dataclass
class MetricTimeWindow(dbtClassMixin):
count: int
granularity: TimeGranularity
@dataclass
class MetricInput(dbtClassMixin):
name: str
filter: Optional[WhereFilter] = None
alias: Optional[str] = None
offset_window: Optional[MetricTimeWindow] = None
offset_to_grain: Optional[TimeGranularity] = None
def as_reference(self) -> DSIMetricReference:
return DSIMetricReference(element_name=self.name)
def post_aggregation_reference(self) -> DSIMetricReference:
return DSIMetricReference(element_name=self.alias or self.name)
@dataclass
class MetricTypeParams(dbtClassMixin):
measure: Optional[MetricInputMeasure] = None
input_measures: List[MetricInputMeasure] = field(default_factory=list)
numerator: Optional[MetricInput] = None
denominator: Optional[MetricInput] = None
expr: Optional[str] = None
window: Optional[MetricTimeWindow] = None
grain_to_date: Optional[TimeGranularity] = None
metrics: Optional[List[MetricInput]] = None
@dataclass
class MetricReference(dbtClassMixin, Replaceable):
sql: Optional[Union[str, int]] = None
unique_id: Optional[str] = None
sql: Optional[Union[str, int]]
unique_id: Optional[str]
@dataclass
@@ -1381,11 +1248,16 @@ class Metric(GraphNode):
name: str
description: str
label: str
type: MetricType
type_params: MetricTypeParams
filter: Optional[WhereFilter] = None
metadata: Optional[SourceFileMetadata] = None
calculation_method: str
expression: str
filters: List[MetricFilter]
time_grains: List[str]
dimensions: List[str]
resource_type: NodeType = field(metadata={"restrict": [NodeType.Metric]})
timestamp: Optional[str] = None
window: Optional[MetricTime] = None
model: Optional[str] = None
model_unique_id: Optional[str] = None
meta: Dict[str, Any] = field(default_factory=dict)
tags: List[str] = field(default_factory=list)
config: MetricConfig = field(default_factory=MetricConfig)
@@ -1405,17 +1277,17 @@ class Metric(GraphNode):
def search_name(self):
return self.name
@property
def input_measures(self) -> List[MetricInputMeasure]:
return self.type_params.input_measures
def same_model(self, old: "Metric") -> bool:
return self.model == old.model
@property
def measure_references(self) -> List[MeasureReference]:
return [x.measure_reference() for x in self.input_measures]
def same_window(self, old: "Metric") -> bool:
return self.window == old.window
@property
def input_metrics(self) -> List[MetricInput]:
return self.type_params.metrics or []
def same_dimensions(self, old: "Metric") -> bool:
return self.dimensions == old.dimensions
def same_filters(self, old: "Metric") -> bool:
return self.filters == old.filters
def same_description(self, old: "Metric") -> bool:
return self.description == old.description
@@ -1423,24 +1295,24 @@ class Metric(GraphNode):
def same_label(self, old: "Metric") -> bool:
return self.label == old.label
def same_calculation_method(self, old: "Metric") -> bool:
return self.calculation_method == old.calculation_method
def same_expression(self, old: "Metric") -> bool:
return self.expression == old.expression
def same_timestamp(self, old: "Metric") -> bool:
return self.timestamp == old.timestamp
def same_time_grains(self, old: "Metric") -> bool:
return self.time_grains == old.time_grains
def same_config(self, old: "Metric") -> bool:
return self.config.same_contents(
self.unrendered_config,
old.unrendered_config,
)
def same_filter(self, old: "Metric") -> bool:
return True # TODO
def same_metadata(self, old: "Metric") -> bool:
return True # TODO
def same_type(self, old: "Metric") -> bool:
return self.type == old.type
def same_type_params(self, old: "Metric") -> bool:
return True # TODO
def same_contents(self, old: Optional["Metric"]) -> bool:
# existing when it didn't before is a change!
# metadata/tags changes are not "changes"
@@ -1448,12 +1320,16 @@ class Metric(GraphNode):
return True
return (
self.same_filter(old)
and self.same_metadata(old)
and self.same_type(old)
and self.same_type_params(old)
self.same_model(old)
and self.same_window(old)
and self.same_dimensions(old)
and self.same_filters(old)
and self.same_description(old)
and self.same_label(old)
and self.same_calculation_method(old)
and self.same_expression(old)
and self.same_timestamp(old)
and self.same_time_grains(old)
and self.same_config(old)
and True
)
@@ -1471,115 +1347,6 @@ class Group(BaseNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Group]})
# ====================================
# SemanticModel and related classes
# ====================================
@dataclass
class NodeRelation(dbtClassMixin):
alias: str
schema_name: str # TODO: Could this be called simply "schema" so we could reuse StateRelation?
database: Optional[str] = None
relation_name: Optional[str] = None
@dataclass
class SemanticModel(GraphNode):
model: str
node_relation: Optional[NodeRelation]
description: Optional[str] = None
defaults: Optional[Defaults] = None
entities: Sequence[Entity] = field(default_factory=list)
measures: Sequence[Measure] = field(default_factory=list)
dimensions: Sequence[Dimension] = field(default_factory=list)
metadata: Optional[SourceFileMetadata] = None
depends_on: DependsOn = field(default_factory=DependsOn)
refs: List[RefArgs] = field(default_factory=list)
created_at: float = field(default_factory=lambda: time.time())
config: SemanticModelConfig = field(default_factory=SemanticModelConfig)
@property
def entity_references(self) -> List[LinkableElementReference]:
return [entity.reference for entity in self.entities]
@property
def dimension_references(self) -> List[LinkableElementReference]:
return [dimension.reference for dimension in self.dimensions]
@property
def measure_references(self) -> List[MeasureReference]:
return [measure.reference for measure in self.measures]
@property
def has_validity_dimensions(self) -> bool:
return any([dim.validity_params is not None for dim in self.dimensions])
@property
def validity_start_dimension(self) -> Optional[Dimension]:
validity_start_dims = [
dim for dim in self.dimensions if dim.validity_params and dim.validity_params.is_start
]
if not validity_start_dims:
return None
return validity_start_dims[0]
@property
def validity_end_dimension(self) -> Optional[Dimension]:
validity_end_dims = [
dim for dim in self.dimensions if dim.validity_params and dim.validity_params.is_end
]
if not validity_end_dims:
return None
return validity_end_dims[0]
@property
def partitions(self) -> List[Dimension]: # noqa: D
return [dim for dim in self.dimensions or [] if dim.is_partition]
@property
def partition(self) -> Optional[Dimension]:
partitions = self.partitions
if not partitions:
return None
return partitions[0]
@property
def reference(self) -> SemanticModelReference:
return SemanticModelReference(semantic_model_name=self.name)
@property
def depends_on_nodes(self):
return self.depends_on.nodes
@property
def depends_on_macros(self):
return self.depends_on.macros
def checked_agg_time_dimension_for_measure(
self, measure_reference: MeasureReference
) -> TimeDimensionReference:
measure: Optional[Measure] = None
for measure in self.measures:
if measure.reference == measure_reference:
measure = measure
assert (
measure is not None
), f"No measure with name ({measure_reference.element_name}) in semantic_model with name ({self.name})"
if self.defaults is not None:
default_agg_time_dimesion = self.defaults.agg_time_dimension
agg_time_dimension_name = measure.agg_time_dimension or default_agg_time_dimesion
assert agg_time_dimension_name is not None, (
f"Aggregation time dimension for measure {measure.name} is not set! This should either be set directly on "
f"the measure specification in the model, or else defaulted to the primary time dimension in the data "
f"source containing the measure."
)
return TimeDimensionReference(element_name=agg_time_dimension_name)
# ====================================
# Patches
# ====================================
@@ -1603,8 +1370,6 @@ class ParsedNodePatch(ParsedPatch):
access: Optional[str]
version: Optional[NodeVersion]
latest_version: Optional[NodeVersion]
constraints: List[Dict[str, Any]]
deprecation_date: Optional[datetime]
@dataclass
@@ -1646,7 +1411,6 @@ GraphMemberNode = Union[
ResultNode,
Exposure,
Metric,
SemanticModel,
]
# All "nodes" (or node-like objects) in this file

View File

@@ -1,95 +0,0 @@
from dbt_semantic_interfaces.implementations.metric import PydanticMetric
from dbt_semantic_interfaces.implementations.project_configuration import (
PydanticProjectConfiguration,
)
from dbt_semantic_interfaces.implementations.semantic_manifest import PydanticSemanticManifest
from dbt_semantic_interfaces.implementations.semantic_model import PydanticSemanticModel
from dbt_semantic_interfaces.implementations.time_spine_table_configuration import (
PydanticTimeSpineTableConfiguration,
)
from dbt_semantic_interfaces.type_enums import TimeGranularity
from dbt_semantic_interfaces.validations.semantic_manifest_validator import (
SemanticManifestValidator,
)
from dbt.clients.system import write_file
from dbt.events.base_types import EventLevel
from dbt.events.functions import fire_event
from dbt.events.types import SemanticValidationFailure
from dbt.exceptions import ParsingError
class SemanticManifest:
def __init__(self, manifest):
self.manifest = manifest
def validate(self) -> bool:
# TODO: Enforce this check.
# if self.manifest.metrics and not self.manifest.semantic_models:
# fire_event(
# SemanticValidationFailure(
# msg="Metrics require semantic models, but none were found."
# ),
# EventLevel.ERROR,
# )
# return False
if not self.manifest.metrics or not self.manifest.semantic_models:
return True
semantic_manifest = self._get_pydantic_semantic_manifest()
validator = SemanticManifestValidator[PydanticSemanticManifest]()
validation_results = validator.validate_semantic_manifest(semantic_manifest)
for warning in validation_results.warnings:
fire_event(SemanticValidationFailure(msg=warning.message))
for error in validation_results.errors:
fire_event(SemanticValidationFailure(msg=error.message), EventLevel.ERROR)
return not validation_results.errors
def write_json_to_file(self, file_path: str):
semantic_manifest = self._get_pydantic_semantic_manifest()
json = semantic_manifest.json()
write_file(file_path, json)
def _get_pydantic_semantic_manifest(self) -> PydanticSemanticManifest:
project_config = PydanticProjectConfiguration(
time_spine_table_configurations=[],
)
pydantic_semantic_manifest = PydanticSemanticManifest(
metrics=[], semantic_models=[], project_configuration=project_config
)
for semantic_model in self.manifest.semantic_models.values():
pydantic_semantic_manifest.semantic_models.append(
PydanticSemanticModel.parse_obj(semantic_model.to_dict())
)
for metric in self.manifest.metrics.values():
pydantic_semantic_manifest.metrics.append(PydanticMetric.parse_obj(metric.to_dict()))
# Look for time-spine table model and create time spine table configuration
if self.manifest.semantic_models:
# Get model for time_spine_table
time_spine_model_name = "metricflow_time_spine"
model = self.manifest.ref_lookup.find(time_spine_model_name, None, None, self.manifest)
if not model:
raise ParsingError(
"The semantic layer requires a 'metricflow_time_spine' model in the project, but none was found. "
"Guidance on creating this model can be found on our docs site ("
"https://docs.getdbt.com/docs/build/metricflow-time-spine) "
)
# Create time_spine_table_config, set it in project_config, and add to semantic manifest
time_spine_table_config = PydanticTimeSpineTableConfiguration(
location=model.relation_name,
column_name="date_day",
grain=TimeGranularity.DAY,
)
pydantic_semantic_manifest.project_configuration.time_spine_table_configurations = [
time_spine_table_config
]
return pydantic_semantic_manifest

View File

@@ -1,147 +0,0 @@
from dataclasses import dataclass
from dbt.dataclass_schema import dbtClassMixin
from dbt_semantic_interfaces.references import (
DimensionReference,
EntityReference,
MeasureReference,
TimeDimensionReference,
)
from dbt_semantic_interfaces.type_enums import (
AggregationType,
DimensionType,
EntityType,
TimeGranularity,
)
from typing import List, Optional
@dataclass
class FileSlice(dbtClassMixin):
"""Provides file slice level context about what something was created from.
Implementation of the dbt-semantic-interfaces `FileSlice` protocol
"""
filename: str
content: str
start_line_number: int
end_line_number: int
@dataclass
class SourceFileMetadata(dbtClassMixin):
"""Provides file context about what something was created from.
Implementation of the dbt-semantic-interfaces `Metadata` protocol
"""
repo_file_path: str
file_slice: FileSlice
@dataclass
class Defaults(dbtClassMixin):
agg_time_dimension: Optional[str] = None
# ====================================
# Dimension objects
# ====================================
@dataclass
class DimensionValidityParams(dbtClassMixin):
is_start: bool = False
is_end: bool = False
@dataclass
class DimensionTypeParams(dbtClassMixin):
time_granularity: TimeGranularity
validity_params: Optional[DimensionValidityParams] = None
@dataclass
class Dimension(dbtClassMixin):
name: str
type: DimensionType
description: Optional[str] = None
is_partition: bool = False
type_params: Optional[DimensionTypeParams] = None
expr: Optional[str] = None
metadata: Optional[SourceFileMetadata] = None
@property
def reference(self) -> DimensionReference:
return DimensionReference(element_name=self.name)
@property
def time_dimension_reference(self) -> Optional[TimeDimensionReference]:
if self.type == DimensionType.TIME:
return TimeDimensionReference(element_name=self.name)
else:
return None
@property
def validity_params(self) -> Optional[DimensionValidityParams]:
if self.type_params:
return self.type_params.validity_params
else:
return None
# ====================================
# Entity objects
# ====================================
@dataclass
class Entity(dbtClassMixin):
name: str
type: EntityType
description: Optional[str] = None
role: Optional[str] = None
expr: Optional[str] = None
@property
def reference(self) -> EntityReference:
return EntityReference(element_name=self.name)
@property
def is_linkable_entity_type(self) -> bool:
return self.type in (EntityType.PRIMARY, EntityType.UNIQUE, EntityType.NATURAL)
# ====================================
# Measure objects
# ====================================
@dataclass
class MeasureAggregationParameters(dbtClassMixin):
percentile: Optional[float] = None
use_discrete_percentile: bool = False
use_approximate_percentile: bool = False
@dataclass
class NonAdditiveDimension(dbtClassMixin):
name: str
window_choice: AggregationType
window_groupings: List[str]
@dataclass
class Measure(dbtClassMixin):
name: str
agg: AggregationType
description: Optional[str] = None
create_metric: bool = False
expr: Optional[str] = None
agg_params: Optional[MeasureAggregationParameters] = None
non_additive_dimension: Optional[NonAdditiveDimension] = None
agg_time_dimension: Optional[str] = None
@property
def reference(self) -> MeasureReference:
return MeasureReference(element_name=self.name)

View File

@@ -1,18 +1,13 @@
import datetime
import re
from dbt import deprecations
from dbt.node_types import NodeType
from dbt.contracts.graph.semantic_models import (
Defaults,
DimensionValidityParams,
MeasureAggregationParameters,
)
from dbt.contracts.util import (
AdditionalPropertiesMixin,
Mergeable,
Replaceable,
)
from dbt.contracts.graph.manifest_upgrade import rename_metric_attr
# trigger the PathEncoder
import dbt.helper_types # noqa:F401
@@ -159,7 +154,6 @@ class UnparsedVersion(dbtClassMixin):
columns: Sequence[Union[dbt.helper_types.IncludeExclude, UnparsedColumn]] = field(
default_factory=list
)
deprecation_date: Optional[datetime.datetime] = None
def __lt__(self, other):
try:
@@ -198,8 +192,6 @@ class UnparsedVersion(dbtClassMixin):
else:
self._unparsed_columns.append(column)
self.deprecation_date = normalize_date(self.deprecation_date)
@dataclass
class UnparsedAnalysisUpdate(HasConfig, HasColumnDocs, HasColumnProps, HasYamlMetadata):
@@ -218,7 +210,6 @@ class UnparsedModelUpdate(UnparsedNodeUpdate):
access: Optional[str] = None
latest_version: Optional[NodeVersion] = None
versions: Sequence[UnparsedVersion] = field(default_factory=list)
deprecation_date: Optional[datetime.datetime] = None
def __post_init__(self):
if self.latest_version:
@@ -238,8 +229,6 @@ class UnparsedModelUpdate(UnparsedNodeUpdate):
self._version_map = {version.v: version for version in self.versions}
self.deprecation_date = normalize_date(self.deprecation_date)
def get_columns_for_version(self, version: NodeVersion) -> List[UnparsedColumn]:
if version not in self._version_map:
raise DbtInternalError(
@@ -598,47 +587,25 @@ class MetricTime(dbtClassMixin, Mergeable):
@dataclass
class UnparsedMetricInputMeasure(dbtClassMixin):
name: str
filter: Optional[str] = None
alias: Optional[str] = None
@dataclass
class UnparsedMetricInput(dbtClassMixin):
name: str
filter: Optional[str] = None
alias: Optional[str] = None
offset_window: Optional[str] = None
offset_to_grain: Optional[str] = None # str is really a TimeGranularity Enum
@dataclass
class UnparsedMetricTypeParams(dbtClassMixin):
measure: Optional[Union[UnparsedMetricInputMeasure, str]] = None
numerator: Optional[Union[UnparsedMetricInput, str]] = None
denominator: Optional[Union[UnparsedMetricInput, str]] = None
expr: Optional[Union[str, bool]] = None
window: Optional[str] = None
grain_to_date: Optional[str] = None # str is really a TimeGranularity Enum
metrics: Optional[List[Union[UnparsedMetricInput, str]]] = None
@dataclass
class UnparsedMetric(dbtClassMixin):
class UnparsedMetric(dbtClassMixin, Replaceable):
name: str
label: str
type: str
type_params: UnparsedMetricTypeParams
calculation_method: str
expression: str
description: str = ""
filter: Optional[str] = None
# metadata: Optional[Unparsedetadata] = None # TODO
timestamp: Optional[str] = None
time_grains: List[str] = field(default_factory=list)
dimensions: List[str] = field(default_factory=list)
window: Optional[MetricTime] = None
model: Optional[str] = None
filters: List[MetricFilter] = field(default_factory=list)
meta: Dict[str, Any] = field(default_factory=dict)
tags: List[str] = field(default_factory=list)
config: Dict[str, Any] = field(default_factory=dict)
@classmethod
def validate(cls, data):
data = rename_metric_attr(data, raise_deprecation_warning=True)
super(UnparsedMetric, cls).validate(data)
if "name" in data:
errors = []
@@ -658,6 +625,22 @@ class UnparsedMetric(dbtClassMixin):
f"The metric name '{data['name']}' is invalid. It {', '.join(e for e in errors)}"
)
if data.get("timestamp") is None and data.get("time_grains") is not None:
raise ValidationError(
f"The metric '{data['name']} has time_grains defined but is missing a timestamp dimension."
)
if data.get("timestamp") is None and data.get("window") is not None:
raise ValidationError(
f"The metric '{data['name']} has a window defined but is missing a timestamp dimension."
)
if data.get("model") is None and data.get("calculation_method") != "derived":
raise ValidationError("Non-derived metrics require a 'model' property")
if data.get("model") is not None and data.get("calculation_method") == "derived":
raise ValidationError("Derived metrics cannot have a 'model' property")
@dataclass
class UnparsedGroup(dbtClassMixin, Replaceable):
@@ -669,77 +652,3 @@ class UnparsedGroup(dbtClassMixin, Replaceable):
super(UnparsedGroup, cls).validate(data)
if data["owner"].get("name") is None and data["owner"].get("email") is None:
raise ValidationError("Group owner must have at least one of 'name' or 'email'.")
#
# semantic interfaces unparsed objects
#
@dataclass
class UnparsedEntity(dbtClassMixin):
name: str
type: str # EntityType enum
description: Optional[str] = None
role: Optional[str] = None
expr: Optional[str] = None
@dataclass
class UnparsedNonAdditiveDimension(dbtClassMixin):
name: str
window_choice: str # AggregationType enum
window_groupings: List[str]
@dataclass
class UnparsedMeasure(dbtClassMixin):
name: str
agg: str # actually an enum
description: Optional[str] = None
expr: Optional[Union[str, bool, int]] = None
agg_params: Optional[MeasureAggregationParameters] = None
non_additive_dimension: Optional[UnparsedNonAdditiveDimension] = None
agg_time_dimension: Optional[str] = None
@dataclass
class UnparsedDimensionTypeParams(dbtClassMixin):
time_granularity: str # TimeGranularity enum
validity_params: Optional[DimensionValidityParams] = None
@dataclass
class UnparsedDimension(dbtClassMixin):
name: str
type: str # actually an enum
description: Optional[str] = None
is_partition: bool = False
type_params: Optional[UnparsedDimensionTypeParams] = None
expr: Optional[str] = None
@dataclass
class UnparsedSemanticModel(dbtClassMixin):
name: str
model: str # looks like "ref(...)"
description: Optional[str] = None
defaults: Optional[Defaults] = None
entities: List[UnparsedEntity] = field(default_factory=list)
measures: List[UnparsedMeasure] = field(default_factory=list)
dimensions: List[UnparsedDimension] = field(default_factory=list)
def normalize_date(d: Optional[datetime.date]) -> Optional[datetime.datetime]:
"""Convert date to datetime (at midnight), and add local time zone if naive"""
if d is None:
return None
# convert date to datetime
dt = d if type(d) == datetime.datetime else datetime.datetime(d.year, d.month, d.day)
if not dt.tzinfo:
# date is naive, re-interpret as system time zone
dt = dt.astimezone()
return dt

View File

@@ -223,7 +223,7 @@ class Project(HyphenatedDbtClassMixin, Replaceable):
)
packages: List[PackageSpec] = field(default_factory=list)
query_comment: Optional[Union[QueryComment, NoValue, str]] = field(default_factory=NoValue)
restrict_access: bool = False
dbt_cloud: Optional[Dict[str, Any]] = None
@classmethod
def validate(cls, data):
@@ -240,6 +240,10 @@ class Project(HyphenatedDbtClassMixin, Replaceable):
or not isinstance(entry["search_order"], list)
):
raise ValidationError(f"Invalid project dispatch config: {entry}")
if "dbt_cloud" in data and not isinstance(data["dbt_cloud"], dict):
raise ValidationError(
f"Invalid dbt_cloud config. Expected a 'dict' but got '{type(data['dbt_cloud'])}'"
)
@dataclass

View File

@@ -17,7 +17,7 @@ class RelationType(StrEnum):
Table = "table"
View = "view"
CTE = "cte"
MaterializedView = "materialized_view"
MaterializedView = "materializedview"
External = "external"

View File

@@ -1,5 +1,3 @@
import threading
from dbt.contracts.graph.unparsed import FreshnessThreshold
from dbt.contracts.graph.nodes import SourceDefinition, ResultNode
from dbt.contracts.util import (
@@ -23,14 +21,13 @@ import agate
from dataclasses import dataclass, field
from datetime import datetime
from typing import (
Any,
Callable,
Union,
Dict,
List,
NamedTuple,
Optional,
Any,
NamedTuple,
Sequence,
Union,
)
from dbt.clients.system import write_json
@@ -59,16 +56,15 @@ class TimingInfo(dbtClassMixin):
# This is a context manager
class collect_timing_info:
def __init__(self, name: str, callback: Callable[[TimingInfo], None]):
def __init__(self, name: str):
self.timing_info = TimingInfo(name=name)
self.callback = callback
def __enter__(self):
self.timing_info.begin()
return self.timing_info
def __exit__(self, exc_type, exc_value, traceback):
self.timing_info.end()
self.callback(self.timing_info)
# Note: when legacy logger is removed, we can remove the following line
with TimingProcessor(self.timing_info):
fire_event(
@@ -163,20 +159,6 @@ class RunResult(NodeResult):
def skipped(self):
return self.status == RunStatus.Skipped
@classmethod
def from_node(cls, node: ResultNode, status: RunStatus, message: Optional[str]):
thread_id = threading.current_thread().name
return RunResult(
status=status,
thread_id=thread_id,
execution_time=0,
timing=[],
message=message,
node=node,
adapter_response={},
failures=None,
)
@dataclass
class ExecutionResult(dbtClassMixin):
@@ -263,6 +245,40 @@ class RunResultsArtifact(ExecutionResult, ArtifactMixin):
write_json(path, self.to_dict(omit_none=False))
@dataclass
class RunOperationResult(ExecutionResult):
success: bool
@dataclass
class RunOperationResultMetadata(BaseArtifactMetadata):
dbt_schema_version: str = field(
default_factory=lambda: str(RunOperationResultsArtifact.dbt_schema_version)
)
@dataclass
@schema_version("run-operation-result", 1)
class RunOperationResultsArtifact(RunOperationResult, ArtifactMixin):
@classmethod
def from_success(
cls,
success: bool,
elapsed_time: float,
generated_at: datetime,
):
meta = RunOperationResultMetadata(
dbt_schema_version=str(cls.dbt_schema_version),
generated_at=generated_at,
)
return cls(
metadata=meta,
results=[],
elapsed_time=elapsed_time,
success=success,
)
# due to issues with typing.Union collapsing subclasses, this can't subclass
# PartialResult
@@ -375,9 +391,6 @@ class FreshnessResult(ExecutionResult):
meta = FreshnessMetadata(generated_at=generated_at)
return cls(metadata=meta, results=results, elapsed_time=elapsed_time)
def write(self, path):
FreshnessExecutionResultArtifact.from_result(self).write(path)
@dataclass
@schema_version("sources", 3)

View File

@@ -1,12 +1,12 @@
from enum import Enum
import os
import threading
from typing import Optional
from dbt.events import types_pb2
import sys
from google.protobuf.json_format import ParseDict, MessageToDict, MessageToJson
from google.protobuf.message import Message
from dbt.events.helpers import get_json_string_utcnow
from typing import Optional
if sys.version_info >= (3, 8):
from typing import Protocol

View File

@@ -13,7 +13,7 @@ from uuid import uuid4
from dbt.events.format import timestamp_to_datetime_string
from dbt.events.base_types import BaseEvent, EventLevel, msg_from_base_event, EventMsg
import dbt.utils
# A Filter is a function which takes a BaseEvent and returns True if the event
# should be logged, False otherwise.
@@ -80,6 +80,7 @@ class LoggerConfig:
use_colors: bool = False
output_stream: Optional[TextIO] = None
output_file_name: Optional[str] = None
output_file_max_bytes: Optional[int] = 10 * 1024 * 1024 # 10 mb
logger: Optional[Any] = None
@@ -100,7 +101,7 @@ class _Logger:
file_handler = RotatingFileHandler(
filename=str(config.output_file_name),
encoding="utf8",
maxBytes=10 * 1024 * 1024, # 10 mb
maxBytes=config.output_file_max_bytes, # type: ignore
backupCount=5,
)
self._python_logger = self._get_python_log_for_handler(file_handler)
@@ -110,7 +111,6 @@ class _Logger:
log.setLevel(_log_level_map[self.level])
handler.setFormatter(logging.Formatter(fmt="%(message)s"))
log.handlers.clear()
log.propagate = False
log.addHandler(handler)
return log
@@ -175,7 +175,7 @@ class _JsonLogger(_Logger):
from dbt.events.functions import msg_to_dict
msg_dict = msg_to_dict(msg)
raw_log_line = json.dumps(msg_dict, sort_keys=True)
raw_log_line = json.dumps(msg_dict, sort_keys=True, cls=dbt.utils.ForgivingJSONEncoder)
line = self.scrubber(raw_log_line) # type: ignore
return line

View File

@@ -13,6 +13,7 @@ from typing import Callable, Dict, List, Optional, TextIO
import uuid
from google.protobuf.json_format import MessageToDict
import dbt.utils
LOG_VERSION = 3
metadata_vars: Optional[Dict[str, str]] = None
@@ -39,18 +40,14 @@ def setup_event_logger(flags, callbacks: List[Callable[[EventMsg], None]] = [])
else:
if flags.LOG_LEVEL != "none":
line_format = _line_format_from_str(flags.LOG_FORMAT, LineFormat.PlainText)
log_level = (
EventLevel.ERROR
if flags.QUIET
else EventLevel.DEBUG
if flags.DEBUG
else EventLevel(flags.LOG_LEVEL)
)
log_level = EventLevel.DEBUG if flags.DEBUG else EventLevel(flags.LOG_LEVEL)
console_config = _get_stdout_config(
line_format,
flags.DEBUG,
flags.USE_COLORS,
log_level,
flags.LOG_CACHE_EVENTS,
flags.QUIET,
)
EVENT_MANAGER.add_logger(console_config)
@@ -67,7 +64,11 @@ def setup_event_logger(flags, callbacks: List[Callable[[EventMsg], None]] = [])
log_level_file = EventLevel.DEBUG if flags.DEBUG else EventLevel(flags.LOG_LEVEL_FILE)
EVENT_MANAGER.add_logger(
_get_logfile_config(
log_file, flags.USE_COLORS_FILE, log_file_format, log_level_file
log_file,
flags.USE_COLORS_FILE,
log_file_format,
log_level_file,
flags.LOG_FILE_MAX_BYTES,
)
)
@@ -85,9 +86,11 @@ def _line_format_from_str(format_str: str, default: LineFormat) -> LineFormat:
def _get_stdout_config(
line_format: LineFormat,
debug: bool,
use_colors: bool,
level: EventLevel,
log_cache_events: bool,
quiet: bool,
) -> LoggerConfig:
return LoggerConfig(
@@ -99,6 +102,8 @@ def _get_stdout_config(
filter=partial(
_stdout_filter,
log_cache_events,
debug,
quiet,
line_format,
),
output_stream=sys.stdout,
@@ -107,16 +112,25 @@ def _get_stdout_config(
def _stdout_filter(
log_cache_events: bool,
debug_mode: bool,
quiet_mode: bool,
line_format: LineFormat,
msg: EventMsg,
) -> bool:
return (msg.info.name not in ["CacheAction", "CacheDumpGraph"] or log_cache_events) and not (
line_format == LineFormat.Json and type(msg.data) == Formatting
return (
(msg.info.name not in ["CacheAction", "CacheDumpGraph"] or log_cache_events)
and (EventLevel(msg.info.level) != EventLevel.DEBUG or debug_mode)
and (EventLevel(msg.info.level) == EventLevel.ERROR or not quiet_mode)
and not (line_format == LineFormat.Json and type(msg.data) == Formatting)
)
def _get_logfile_config(
log_path: str, use_colors: bool, line_format: LineFormat, level: EventLevel
log_path: str,
use_colors: bool,
line_format: LineFormat,
level: EventLevel,
log_file_max_bytes: int,
) -> LoggerConfig:
return LoggerConfig(
name="file_log",
@@ -126,6 +140,7 @@ def _get_logfile_config(
scrubber=env_scrubber,
filter=partial(_logfile_filter, bool(get_flags().LOG_CACHE_EVENTS), line_format),
output_file_name=log_path,
output_file_max_bytes=log_file_max_bytes,
)
@@ -142,9 +157,11 @@ def _get_logbook_log_config(
) -> LoggerConfig:
config = _get_stdout_config(
LineFormat.PlainText,
debug,
use_colors,
EventLevel.ERROR if quiet else EventLevel.DEBUG if debug else EventLevel.INFO,
EventLevel.DEBUG if debug else EventLevel.INFO,
log_cache_events,
quiet,
)
config.name = "logbook_log"
config.filter = (
@@ -176,7 +193,7 @@ EVENT_MANAGER: EventManager = EventManager()
EVENT_MANAGER.add_logger(
_get_logbook_log_config(False, True, False, False) # type: ignore
if ENABLE_LEGACY_LOGGER
else _get_stdout_config(LineFormat.PlainText, True, EventLevel.INFO, False)
else _get_stdout_config(LineFormat.PlainText, False, True, EventLevel.INFO, False, False)
)
# This global, and the following two functions for capturing stdout logs are
@@ -200,7 +217,7 @@ def stop_capture_stdout_logs():
# the message may contain secrets which must be scrubbed at the usage site.
def msg_to_json(msg: EventMsg) -> str:
msg_dict = msg_to_dict(msg)
raw_log_line = json.dumps(msg_dict, sort_keys=True)
raw_log_line = json.dumps(msg_dict, sort_keys=True, cls=dbt.utils.ForgivingJSONEncoder)
return raw_log_line
@@ -246,13 +263,6 @@ def fire_event_if(
fire_event(lazy_e(), level=level)
# a special case of fire_event_if, to only fire events in our unit/functional tests
def fire_event_if_test(
lazy_e: Callable[[], BaseEvent], level: Optional[EventLevel] = None
) -> None:
fire_event_if(conditional=("pytest" in sys.modules), lazy_e=lazy_e, level=level)
# top-level method for accessing the new eventing system
# this is where all the side effects happen branched by event type
# (i.e. - mutating the event history, printing to stdout, logging

View File

@@ -888,6 +888,26 @@ message ParsePerfInfoPathMsg {
ParsePerfInfoPath data = 2;
}
// I011
message GenericTestFileParse {
string path = 1;
}
message GenericTestFileParseMsg {
EventInfo info = 1;
GenericTestFileParse data = 2;
}
// I012
message MacroFileParse {
string path = 1;
}
message MacroFileParseMsg {
EventInfo info = 1;
MacroFileParse data = 2;
}
// Skipping I013
// I014
@@ -1175,48 +1195,6 @@ message UnpinnedRefNewVersionAvailableMsg {
UnpinnedRefNewVersionAvailable data = 2;
}
// I065
message DeprecatedModel {
string model_name = 1;
string model_version = 2;
string deprecation_date = 3;
}
message DeprecatedModelMsg {
EventInfo info = 1;
DeprecatedModel data = 2;
}
// I066
message UpcomingReferenceDeprecation {
string model_name = 1;
string ref_model_package = 2;
string ref_model_name = 3;
string ref_model_version = 4;
string ref_model_latest_version = 5;
string ref_model_deprecation_date = 6;
}
message UpcomingReferenceDeprecationMsg {
EventInfo info = 1;
UpcomingReferenceDeprecation data = 2;
}
// I067
message DeprecatedReference {
string model_name = 1;
string ref_model_package = 2;
string ref_model_name = 3;
string ref_model_version = 4;
string ref_model_latest_version = 5;
string ref_model_deprecation_date = 6;
}
message DeprecatedReferenceMsg {
EventInfo info = 1;
DeprecatedReference data = 2;
}
// I068
message UnsupportedConstraintMaterialization {
string materialized = 1;
@@ -1232,23 +1210,11 @@ message ParseInlineNodeError{
NodeInfo node_info = 1;
string exc = 2;
}
message ParseInlineNodeErrorMsg {
EventInfo info = 1;
ParseInlineNodeError data = 2;
}
// I070
message SemanticValidationFailure {
string msg = 2;
}
message SemanticValidationFailureMsg {
EventInfo info = 1;
SemanticValidationFailure data = 2;
}
// M - Deps generation
// M001
@@ -1650,6 +1616,7 @@ message LogSnapshotResult {
int32 total = 5;
float execution_time = 6;
map<string, string> cfg = 7;
string result_message = 8;
}
message LogSnapshotResultMsg {

View File

@@ -31,7 +31,6 @@ from dbt.node_types import NodeType
# | E | DB adapter |
# | I | Project parsing |
# | M | Deps generation |
# | P | Artifacts |
# | Q | Node execution |
# | W | Node testing |
# | Z | Misc |
@@ -810,7 +809,7 @@ class InputFileDiffError(DebugLevel):
return f"Error processing file diff: {self.category}, {self.file_id}"
# Skipping I003, I004, I005, I006, I007
# Skipping I002, I003, I004, I005, I006, I007
class InvalidValueForField(WarnLevel):
@@ -837,10 +836,20 @@ class ParsePerfInfoPath(InfoLevel):
return f"Performance info: {self.path}"
# Removed I011: GenericTestFileParse
class GenericTestFileParse(DebugLevel):
def code(self):
return "I011"
def message(self) -> str:
return f"Parsing {self.path}"
# Removed I012: MacroFileParse
class MacroFileParse(DebugLevel):
def code(self):
return "I012"
def message(self) -> str:
return f"Parsing {self.path}"
# Skipping I013
@@ -1147,63 +1156,6 @@ class UnpinnedRefNewVersionAvailable(InfoLevel):
return msg
class DeprecatedModel(WarnLevel):
def code(self):
return "I065"
def message(self) -> str:
version = ".v" + self.model_version if self.model_version else ""
msg = (
f"Model {self.model_name}{version} has passed its deprecation date of {self.deprecation_date}. "
"This model should be disabled or removed."
)
return warning_tag(msg)
class UpcomingReferenceDeprecation(WarnLevel):
def code(self):
return "I066"
def message(self) -> str:
ref_model_version = ".v" + self.ref_model_version if self.ref_model_version else ""
msg = (
f"While compiling '{self.model_name}': Found a reference to {self.ref_model_name}{ref_model_version}, "
f"which is slated for deprecation on '{self.ref_model_deprecation_date}'. "
)
if self.ref_model_version and self.ref_model_version != self.ref_model_latest_version:
coda = (
f"A new version of '{self.ref_model_name}' is available. Try it out: "
f"{{{{ ref('{self.ref_model_package}', '{self.ref_model_name}', "
f"v='{self.ref_model_latest_version}') }}}}."
)
msg = msg + coda
return warning_tag(msg)
class DeprecatedReference(WarnLevel):
def code(self):
return "I067"
def message(self) -> str:
ref_model_version = ".v" + self.ref_model_version if self.ref_model_version else ""
msg = (
f"While compiling '{self.model_name}': Found a reference to {self.ref_model_name}{ref_model_version}, "
f"which was deprecated on '{self.ref_model_deprecation_date}'. "
)
if self.ref_model_version and self.ref_model_version != self.ref_model_latest_version:
coda = (
f"A new version of '{self.ref_model_name}' is available. Migrate now: "
f"{{{{ ref('{self.ref_model_package}', '{self.ref_model_name}', "
f"v='{self.ref_model_latest_version}') }}}}."
)
msg = msg + coda
return warning_tag(msg)
class UnsupportedConstraintMaterialization(WarnLevel):
def code(self):
return "I068"
@@ -1225,14 +1177,6 @@ class ParseInlineNodeError(ErrorLevel):
return "Error while parsing node: " + self.node_info.node_name + "\n" + self.exc
class SemanticValidationFailure(WarnLevel):
def code(self):
return "I070"
def message(self) -> str:
return self.msg
# =======================================================
# M - Deps generation
# =======================================================
@@ -1614,7 +1558,7 @@ class LogSnapshotResult(DynamicLevel):
status = red(self.status.upper())
else:
info = "OK snapshotted"
status = green(self.status)
status = green(self.result_message)
msg = "{info} {description}".format(info=info, description=self.description, **self.cfg)
return format_fancy_output_line(

File diff suppressed because one or more lines are too long

View File

@@ -3,11 +3,11 @@ import json
import re
import io
import agate
from typing import Any, Dict, List, Mapping, Optional, Tuple, Union
from typing import Any, Dict, List, Mapping, Optional, Union
from dbt.dataclass_schema import ValidationError
from dbt.events.helpers import env_secrets, scrub_secrets
from dbt.node_types import NodeType, AccessType
from dbt.node_types import NodeType
from dbt.ui import line_wrap_message
import dbt.dataclass_schema
@@ -212,21 +212,11 @@ class ContractBreakingChangeError(DbtRuntimeError):
MESSAGE = "Breaking Change to Contract"
def __init__(
self,
contract_enforced_disabled: bool,
columns_removed: List[str],
column_type_changes: List[Tuple[str, str, str]],
enforced_column_constraint_removed: List[Tuple[str, str]],
enforced_model_constraint_removed: List[Tuple[str, List[str]]],
materialization_changed: List[str],
node=None,
self, contract_enforced_disabled, columns_removed, column_type_changes, node=None
):
self.contract_enforced_disabled = contract_enforced_disabled
self.columns_removed = columns_removed
self.column_type_changes = column_type_changes
self.enforced_column_constraint_removed = enforced_column_constraint_removed
self.enforced_model_constraint_removed = enforced_model_constraint_removed
self.materialization_changed = materialization_changed
super().__init__(self.message(), node)
@property
@@ -247,27 +237,6 @@ class ContractBreakingChangeError(DbtRuntimeError):
breaking_changes.append(
f"Columns with data_type changes: \n - {column_type_changes_str}"
)
if self.enforced_column_constraint_removed:
column_constraint_changes_str = "\n - ".join(
[f"{c[0]} ({c[1]})" for c in self.enforced_column_constraint_removed]
)
breaking_changes.append(
f"Enforced column level constraints were removed: \n - {column_constraint_changes_str}"
)
if self.enforced_model_constraint_removed:
model_constraint_changes_str = "\n - ".join(
[f"{c[0]} -> {c[1]}" for c in self.enforced_model_constraint_removed]
)
breaking_changes.append(
f"Enforced model level constraints were removed: \n - {model_constraint_changes_str}"
)
if self.materialization_changed:
materialization_changes_str = "\n - ".join(
f"{self.materialization_changed[0]} -> {self.materialization_changed[1]}"
)
breaking_changes.append(
f"Materialization changed with enforced constraints: \n - {materialization_changes_str}"
)
reasons = "\n\n".join(breaking_changes)
@@ -297,11 +266,6 @@ class ParsingError(DbtRuntimeError):
return "Parsing"
class dbtPluginError(DbtRuntimeError):
CODE = 10020
MESSAGE = "Plugin Error"
# TODO: this isn't raised in the core codebase. Is it raised elsewhere?
class JSONValidationError(DbtValidationError):
def __init__(self, typename, errors):
@@ -411,7 +375,7 @@ class DbtProfileError(DbtConfigError):
class SemverError(Exception):
def __init__(self, msg: Optional[str] = None):
def __init__(self, msg: str = None):
self.msg = msg
if msg is not None:
super().__init__(msg)
@@ -690,15 +654,6 @@ class UnknownGitCloningProblemError(DbtRuntimeError):
return msg
class NoAdaptersAvailableError(DbtRuntimeError):
def __init__(self):
super().__init__(msg=self.get_message())
def get_message(self) -> str:
msg = "No adapters available. Learn how to install an adapter by going to https://docs.getdbt.com/docs/connect-adapters#install-using-the-cli"
return msg
class BadSpecError(DbtInternalError):
def __init__(self, repo, revision, error):
self.repo = repo
@@ -1064,17 +1019,6 @@ class DuplicateMacroNameError(CompilationError):
return msg
class MacroResultAlreadyLoadedError(CompilationError):
def __init__(self, result_name):
self.result_name = result_name
super().__init__(msg=self.get_message())
def get_message(self) -> str:
msg = f"The 'statement' result named '{self.result_name}' has already been loaded into a variable"
return msg
# parser level exceptions
class DictParseError(ParsingError):
def __init__(self, exc: ValidationError, node):
@@ -1219,31 +1163,26 @@ class SnapshopConfigError(ParsingError):
class DbtReferenceError(ParsingError):
def __init__(self, unique_id: str, ref_unique_id: str, access: AccessType, scope: str):
def __init__(self, unique_id: str, ref_unique_id: str, group: str):
self.unique_id = unique_id
self.ref_unique_id = ref_unique_id
self.access = access
self.scope = scope
self.scope_type = "group" if self.access == AccessType.Private else "package"
self.group = group
super().__init__(msg=self.get_message())
def get_message(self) -> str:
return (
f"Node {self.unique_id} attempted to reference node {self.ref_unique_id}, "
f"which is not allowed because the referenced node is {self.access} to the '{self.scope}' {self.scope_type}."
f"which is not allowed because the referenced node is private to the {self.group} group."
)
class InvalidAccessTypeError(ParsingError):
def __init__(self, unique_id: str, field_value: str, materialization: Optional[str] = None):
def __init__(self, unique_id: str, field_value: str):
self.unique_id = unique_id
self.field_value = field_value
self.materialization = materialization
with_materialization = (
f"with '{self.materialization}' materialization " if self.materialization else ""
msg = (
f"Node {self.unique_id} has an invalid value ({self.field_value}) for the access field"
)
msg = f"Node {self.unique_id} {with_materialization}has an invalid value ({self.field_value}) for the access field"
super().__init__(msg=msg)
@@ -1410,7 +1349,7 @@ class TargetNotFoundError(CompilationError):
target_package_string = ""
if self.target_package is not None:
target_package_string = f"in package or project '{self.target_package}' "
target_package_string = f"in package '{self.target_package}' "
msg = (
f"{resource_type_title} '{unique_id}' ({original_file_path}) depends on a "
@@ -1824,19 +1763,17 @@ class UninstalledPackagesFoundError(CompilationError):
self,
count_packages_specified: int,
count_packages_installed: int,
packages_specified_path: str,
packages_install_path: str,
):
self.count_packages_specified = count_packages_specified
self.count_packages_installed = count_packages_installed
self.packages_specified_path = packages_specified_path
self.packages_install_path = packages_install_path
super().__init__(msg=self.get_message())
def get_message(self) -> str:
msg = (
f"dbt found {self.count_packages_specified} package(s) "
f"specified in {self.packages_specified_path}, but only "
"specified in packages.yml, but only "
f"{self.count_packages_installed} package(s) installed "
f'in {self.packages_install_path}. Run "dbt deps" to '
"install package dependencies."
@@ -2013,23 +1950,6 @@ class AmbiguousAliasError(CompilationError):
return msg
class AmbiguousResourceNameRefError(CompilationError):
def __init__(self, duped_name, unique_ids, node=None):
self.duped_name = duped_name
self.unique_ids = unique_ids
self.packages = [unique_id.split(".")[1] for unique_id in unique_ids]
super().__init__(msg=self.get_message(), node=node)
def get_message(self) -> str:
formatted_unique_ids = "'{0}'".format("', '".join(self.unique_ids))
formatted_packages = "'{0}'".format("' or '".join(self.packages))
msg = (
f"When referencing '{self.duped_name}', dbt found nodes in multiple packages: {formatted_unique_ids}"
f"\nTo fix this, use two-argument 'ref', with the package name first: {formatted_packages}"
)
return msg
class AmbiguousCatalogMatchError(CompilationError):
def __init__(self, unique_id: str, match_1, match_2):
self.unique_id = unique_id
@@ -2407,7 +2327,7 @@ class RPCCompiling(DbtRuntimeError):
CODE = 10010
MESSAGE = 'RPC server is compiling the project, call the "status" method for' " compile status"
def __init__(self, msg: Optional[str] = None, node=None):
def __init__(self, msg: str = None, node=None):
if msg is None:
msg = "compile in progress"
super().__init__(msg, node)

View File

@@ -87,7 +87,6 @@ def get_flag_dict():
"introspect",
"target_path",
"log_path",
"invocation_command",
}
return {key: getattr(GLOBAL_FLAGS, key.upper(), None) for key in flag_attr}

View File

@@ -28,29 +28,16 @@ class Graph:
"""Returns all nodes having a path to `node` in `graph`"""
if not self.graph.has_node(node):
raise DbtInternalError(f"Node {node} not found in the graph!")
filtered_graph = self.exclude_edge_type("parent_test")
return {
child
for _, child in nx.bfs_edges(filtered_graph, node, reverse=True, depth_limit=max_depth)
for _, child in nx.bfs_edges(self.graph, node, reverse=True, depth_limit=max_depth)
}
def descendants(self, node: UniqueId, max_depth: Optional[int] = None) -> Set[UniqueId]:
"""Returns all nodes reachable from `node` in `graph`"""
if not self.graph.has_node(node):
raise DbtInternalError(f"Node {node} not found in the graph!")
filtered_graph = self.exclude_edge_type("parent_test")
return {child for _, child in nx.bfs_edges(filtered_graph, node, depth_limit=max_depth)}
def exclude_edge_type(self, edge_type_to_exclude):
return nx.restricted_view(
self.graph,
nodes=[],
edges=(
(a, b)
for a, b in self.graph.edges
if self.graph[a][b].get("edge_type") == edge_type_to_exclude
),
)
return {child for _, child in nx.bfs_edges(self.graph, node, depth_limit=max_depth)}
def select_childrens_parents(self, selected: Set[UniqueId]) -> Set[UniqueId]:
ancestors_for = self.select_children(selected) | selected

View File

@@ -36,18 +36,16 @@ def can_select_indirectly(node):
class NodeSelector(MethodManager):
"""The node selector is aware of the graph and manifest"""
"""The node selector is aware of the graph and manifest,"""
def __init__(
self,
graph: Graph,
manifest: Manifest,
previous_state: Optional[PreviousState] = None,
include_empty_nodes: bool = False,
):
super().__init__(manifest, previous_state)
self.full_graph = graph
self.include_empty_nodes = include_empty_nodes
# build a subgraph containing only non-empty, enabled nodes and enabled
# sources.
@@ -168,14 +166,8 @@ class NodeSelector(MethodManager):
elif unique_id in self.manifest.metrics:
metric = self.manifest.metrics[unique_id]
return metric.config.enabled
elif unique_id in self.manifest.semantic_models:
return True
node = self.manifest.nodes[unique_id]
if self.include_empty_nodes:
return node.config.enabled
else:
return not node.empty and node.config.enabled
return not node.empty and node.config.enabled
def node_is_match(self, node: GraphMemberNode) -> bool:
"""Determine if a node is a match for the selector. Non-match nodes
@@ -193,8 +185,6 @@ class NodeSelector(MethodManager):
node = self.manifest.exposures[unique_id]
elif unique_id in self.manifest.metrics:
node = self.manifest.metrics[unique_id]
elif unique_id in self.manifest.semantic_models:
node = self.manifest.semantic_models[unique_id]
else:
raise DbtInternalError(f"Node {unique_id} not found in the manifest!")
return self.node_is_match(node)
@@ -323,13 +313,11 @@ class ResourceTypeSelector(NodeSelector):
manifest: Manifest,
previous_state: Optional[PreviousState],
resource_types: List[NodeType],
include_empty_nodes: bool = False,
):
super().__init__(
graph=graph,
manifest=manifest,
previous_state=previous_state,
include_empty_nodes=include_empty_nodes,
)
self.resource_types: Set[NodeType] = set(resource_types)

View File

@@ -61,8 +61,8 @@ def is_selected_node(fqn: List[str], node_selector: str, is_versioned: bool) ->
flat_node_selector = node_selector.split(".")
if fqn[-2] == node_selector:
return True
# If this is a versioned model, then the last two segments should be allowed to exactly match on either the '.' or '_' delimiter
elif "_".join(fqn[-2:]) == "_".join(flat_node_selector[-2:]):
# If this is a versioned model, then the last two segments should be allowed to exactly match
elif fqn[-2:] == flat_node_selector[-2:]:
return True
else:
if fqn[-1] == node_selector:
@@ -351,8 +351,6 @@ class FileSelectorMethod(SelectorMethod):
for node, real_node in self.all_nodes(included_nodes):
if fnmatch(Path(real_node.original_file_path).name, selector):
yield node
elif fnmatch(Path(real_node.original_file_path).stem, selector):
yield node
class PackageSelectorMethod(SelectorMethod):
@@ -437,8 +435,6 @@ class ResourceTypeSelectorMethod(SelectorMethod):
class TestNameSelectorMethod(SelectorMethod):
__test__ = False
def search(self, included_nodes: Set[UniqueId], selector: str) -> Iterator[UniqueId]:
for node, real_node in self.parsed_nodes(included_nodes):
if real_node.resource_type == NodeType.Test and hasattr(real_node, "test_metadata"):
@@ -447,8 +443,6 @@ class TestNameSelectorMethod(SelectorMethod):
class TestTypeSelectorMethod(SelectorMethod):
__test__ = False
def search(self, included_nodes: Set[UniqueId], selector: str) -> Iterator[UniqueId]:
search_type: Type
# continue supporting 'schema' + 'data' for backwards compatibility
@@ -536,24 +530,12 @@ class StateSelectorMethod(SelectorMethod):
return self.recursively_check_macros_modified(node, visited_macros)
# TODO check modifed_content and check_modified macro seems a bit redundent
def check_modified_content(
self, old: Optional[SelectorTarget], new: SelectorTarget, adapter_type: str
) -> bool:
if isinstance(new, (SourceDefinition, Exposure, Metric)):
# these all overwrite `same_contents`
different_contents = not new.same_contents(old) # type: ignore
else:
different_contents = not new.same_contents(old, adapter_type) # type: ignore
def check_modified_content(self, old: Optional[SelectorTarget], new: SelectorTarget) -> bool:
different_contents = not new.same_contents(old) # type: ignore
upstream_macro_change = self.check_macros_modified(new)
return different_contents or upstream_macro_change
def check_unmodified_content(
self, old: Optional[SelectorTarget], new: SelectorTarget, adapter_type: str
) -> bool:
return not self.check_modified_content(old, new, adapter_type)
def check_modified_macros(self, old, new: SelectorTarget) -> bool:
def check_modified_macros(self, _, new: SelectorTarget) -> bool:
return self.check_macros_modified(new)
@staticmethod
@@ -570,21 +552,6 @@ class StateSelectorMethod(SelectorMethod):
return check_modified_things
@staticmethod
def check_modified_contract(
compare_method: str,
adapter_type: Optional[str],
) -> Callable[[Optional[SelectorTarget], SelectorTarget], bool]:
# get a function that compares two selector target based on compare method provided
def check_modified_contract(old: Optional[SelectorTarget], new: SelectorTarget) -> bool:
if hasattr(new, compare_method):
# when old body does not exist or old and new are not the same
return not old or not getattr(new, compare_method)(old, adapter_type) # type: ignore
else:
return False
return check_modified_contract
def check_new(self, old: Optional[SelectorTarget], new: SelectorTarget) -> bool:
return old is None
@@ -592,15 +559,11 @@ class StateSelectorMethod(SelectorMethod):
if self.previous_state is None or self.previous_state.manifest is None:
raise DbtRuntimeError("Got a state selector method, but no comparison manifest")
adapter_type = self.manifest.metadata.adapter_type
state_checks = {
# it's new if there is no old version
"new": lambda old, new: old is None,
"old": lambda old, new: old is not None,
"new": lambda old, _: old is None,
# use methods defined above to compare properties of old + new
"modified": self.check_modified_content,
"unmodified": self.check_unmodified_content,
"modified.body": self.check_modified_factory("same_body"),
"modified.configs": self.check_modified_factory("same_config"),
"modified.persisted_descriptions": self.check_modified_factory(
@@ -608,7 +571,7 @@ class StateSelectorMethod(SelectorMethod):
),
"modified.relation": self.check_modified_factory("same_database_representation"),
"modified.macros": self.check_modified_macros,
"modified.contract": self.check_modified_contract("same_contract", adapter_type),
"modified.contract": self.check_modified_factory("same_contract"),
}
if selector in state_checks:
checker = state_checks[selector]
@@ -621,7 +584,6 @@ class StateSelectorMethod(SelectorMethod):
for node, real_node in self.all_nodes(included_nodes):
previous_node: Optional[SelectorTarget] = None
if node in manifest.nodes:
previous_node = manifest.nodes[node]
elif node in manifest.sources:
@@ -631,15 +593,7 @@ class StateSelectorMethod(SelectorMethod):
elif node in manifest.metrics:
previous_node = manifest.metrics[node]
keyword_args = {}
if checker.__name__ in [
"same_contract",
"check_modified_content",
"check_unmodified_content",
]:
keyword_args["adapter_type"] = adapter_type # type: ignore
if checker(previous_node, real_node, **keyword_args): # type: ignore
if checker(previous_node, real_node):
yield node

View File

@@ -1,44 +0,0 @@
{% macro drop_relation(relation) -%}
{{ return(adapter.dispatch('drop_relation', 'dbt')(relation)) }}
{% endmacro %}
{% macro default__drop_relation(relation) -%}
{% call statement('drop_relation', auto_begin=False) -%}
{%- if relation.is_table -%}
{{- drop_table(relation) -}}
{%- elif relation.is_view -%}
{{- drop_view(relation) -}}
{%- elif relation.is_materialized_view -%}
{{- drop_materialized_view(relation) -}}
{%- else -%}
drop {{ relation.type }} if exists {{ relation }} cascade
{%- endif -%}
{%- endcall %}
{% endmacro %}
{% macro drop_table(relation) -%}
{{ return(adapter.dispatch('drop_table', 'dbt')(relation)) }}
{%- endmacro %}
{% macro default__drop_table(relation) -%}
drop table if exists {{ relation }} cascade
{%- endmacro %}
{% macro drop_view(relation) -%}
{{ return(adapter.dispatch('drop_view', 'dbt')(relation)) }}
{%- endmacro %}
{% macro default__drop_view(relation) -%}
drop view if exists {{ relation }} cascade
{%- endmacro %}
{% macro drop_materialized_view(relation) -%}
{{ return(adapter.dispatch('drop_materialized_view', 'dbt')(relation)) }}
{%- endmacro %}
{% macro default__drop_materialized_view(relation) -%}
drop materialized view if exists {{ relation }} cascade
{%- endmacro %}

View File

@@ -21,21 +21,3 @@
{% endif %}
{% endfor %}
{% endmacro %}
{% macro get_drop_index_sql(relation, index_name) -%}
{{ adapter.dispatch('get_drop_index_sql', 'dbt')(relation, index_name) }}
{%- endmacro %}
{% macro default__get_drop_index_sql(relation, index_name) -%}
{{ exceptions.raise_compiler_error("`get_drop_index_sql has not been implemented for this adapter.") }}
{%- endmacro %}
{% macro get_show_indexes_sql(relation) -%}
{{ adapter.dispatch('get_show_indexes_sql', 'dbt')(relation) }}
{%- endmacro %}
{% macro default__get_show_indexes_sql(relation) -%}
{{ exceptions.raise_compiler_error("`get_show_indexes_sql has not been implemented for this adapter.") }}
{%- endmacro %}

View File

@@ -31,6 +31,16 @@
{{ return(backup_relation) }}
{% endmacro %}
{% macro drop_relation(relation) -%}
{{ return(adapter.dispatch('drop_relation', 'dbt')(relation)) }}
{% endmacro %}
{% macro default__drop_relation(relation) -%}
{% call statement('drop_relation', auto_begin=False) -%}
drop {{ relation.type }} if exists {{ relation }} cascade
{%- endcall %}
{% endmacro %}
{% macro truncate_relation(relation) -%}
{{ return(adapter.dispatch('truncate_relation', 'dbt')(relation)) }}

View File

@@ -0,0 +1,22 @@
{% macro get_show_sql(compiled_code, sql_header, limit) -%}
{%- if sql_header -%}
{{ sql_header }}
{%- endif -%}
{%- if limit is not none -%}
{{ get_limit_subquery_sql(compiled_code, limit) }}
{%- else -%}
{{ compiled_code }}
{%- endif -%}
{% endmacro %}
{% macro get_limit_subquery_sql(sql, limit) %}
{{ adapter.dispatch('get_limit_subquery_sql', 'dbt')(sql, limit) }}
{% endmacro %}
{% macro default__get_limit_subquery_sql(sql, limit) %}
select *
from (
{{ sql }}
) as model_limit_subq
limit {{ limit }}
{% endmacro %}

View File

@@ -1,10 +0,0 @@
{% macro validate_sql(sql) -%}
{{ return(adapter.dispatch('validate_sql', 'dbt')(sql)) }}
{% endmacro %}
{% macro default__validate_sql(sql) -%}
{% call statement('validate_sql') -%}
explain {{ sql }}
{% endcall %}
{{ return(load_result('validate_sql')) }}
{% endmacro %}

Some files were not shown because too many files have changed in this diff Show More