Compare commits

...

19 Commits

Author SHA1 Message Date
Emily Rockman
734b6429c7 update to use Docs object 2022-07-27 13:34:23 -05:00
Benoit Perigaud
a75b2c0a90 Make docs a dataclass instead of a Dict 2022-07-26 19:38:22 +02:00
Benoit Perigaud
6b6ae22434 Merge branch 'feature/custom-node-colors-dbt_project' of github.com:dbt-labs/dbt-core into feature/custom-node-colors-dbt_project 2022-07-20 12:51:29 +02:00
Benoit Perigaud
287f443ec9 Make docs a Dict to avoid parsing errors 2022-07-20 12:44:01 +02:00
Benoit Perigaud
aea2c4a29b Add node_color to Docs 2022-07-20 12:43:09 +02:00
Benoit Perigaud
21ffe31270 Handle when docs is both under docs and config.docs 2022-07-20 12:42:25 +02:00
Sung Won Chung
70c9074625 Merge branch 'main' of https://github.com/dbt-labs/dbt into feature/custom-node-colors-dbt_project 2022-07-19 14:27:18 -05:00
Tomás Farías Santana
2048a1af6f [CT-472] feat: Retrying method for acquiring connection handles (#5432)
Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift).
2022-07-19 14:55:55 -04:00
Emily Rockman
71223dc253 add timeouts to integration tests (#5481)
* add timeout to integration tests

* add timeout to all jobs in workflow
2022-07-19 10:31:46 -05:00
Emily Rockman
e03d35a9fc Fix: Rename try methods to strict (#5477) (#5478)
* rename strict methods

* add changelog

Co-authored-by: Jeremy Yeo <jeremyyeo@users.noreply.github.com>
2022-07-15 09:38:21 -05:00
github-actions[bot]
f988f76fcc Bumping version to 1.3.0a1 (#5463)
* Bumping version to 1.3.0a1

* Remove whitespace

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-07-12 14:17:06 -04:00
Emily Rockman
0cacfd0f88 cleanup changelog after cutting 1.2.latest branch (#5462) 2022-07-12 11:05:50 -05:00
Matthew McKnight
c25260e5dd Add Grant SQL Macros (#5369)
* init push or ct-660 work

* changes to default versions of get_show_grant_sql and get_grant_sql

* completing init default versions of all macros being called for look over and collaboration

* minor update to should_revoke

* post pairing push up (does have log statements to make sure we remove)

* minor spacing changes

* minor changes, and removal of logs so people can have clean grab of code

* minor changes to how get_revoke_sql works

* init attempt at applying apply_grants to all materialzations

* name change from recipents -> grantee

* minor changes

* working on making a context to handle the diff gathering between grant_config and curreent_grants to see what needs to be revoked, I know if we assign a role, and a model becomes dependent on it we can't drop the role now still not seeing the diff appear in log

* removing logs from most materializations to better track diff of grants generation logs

* starting to build out postgres get_show_grant_sql getting empty query errors hopefully will clear up as we add the other postgres versions of macros and isn't a psycopg2 issue as indicated by searching

* 6/27 eod update looking into diff_grants variable not getting passed into get_revoke_sql

* changes to loop cases

* changes after pairing meeting

* adding apply_grants to create_or_replace_view.sql

* models are building but testing out small issues around revoke statement never building

* postgrest must fixes from jeremy's feedback

* postgres minor change to standarize_grants_dict

* updating after pairing with dough and jeremey incorporating the new version of should revoke logic.

* adding  ref of diff_of_two_dicts to base keys ref

* change of method type for standardize_grants_dict

* minor update trying to fix unit test

* changes based on morning feedback

* change log message in default_apply_grants macro

* CT-808 grant adapter tests (#5447)

* Create initial test for grants

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

* rename grant[privilege] -> grant_config[privilege]

* postgres macro rename to copy_grants

* CT-808 more grant adapter tests (#5452)

* Add tests for invalid user and invalid privilege

* Add more grant tests

* Macro touch ups

* Many more tests

* Allow easily replacing privilege names

* Keep adding tests

* Refactor macros to return lists, fix test

* Code checks

* Keep tweaking tests

* Revert cool grantees join bc Snowflake isnt happy

* Use Postgres/BQ as standard for standardize_grants_dict

* Code checks

* add missing replace

* small replace tweak,  add additional dict diffs

* All tests passing on BQ

* Add type cast to test_snapshot_grants

* Refactor for DRYer apply_grants macros

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* update to main, create changelog, whitespace fixes

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-07-11 11:58:41 -05:00
Batuhan Taskaya
c521fa6b74 Add --target-path as a CLI option. (#5402) 2022-07-07 12:49:25 -04:00
Emily Rockman
f304b4b2da add new index file for dbt-docs for 1.2.0 release (#5446)
* add new index file for dbt-docs for 1.2.0 release

* add PR number to changelogs
2022-07-06 16:12:38 -05:00
Drew Banin
064d890172 (#4884) Add support for ratio metrics (#5027)
* wip

* More support for ratio metrics

* Formatting and linting

* Fix unit tests

* Support disabling metrics

* mypy

* address all TODOs

* make pypy happy

* wip

* checkpoint

* refactor, remove ratio_terms

* flake8 and unit tests

* remove debugger

* quickfix for filters

* Experiment with functional testing for 'expression' metrics

* reformatting slightly

* make file and mypy fix

* remove config from metrics - wip

* add metrics back to context

* adding test changes

* fixing test metrics

* revert name audit

* pre-commit fixes

* add changelog

* Bumping manifest version to v6 (#5430)

* Bumping manifest version to v6

* Adding manifest file for tests

* Reverting unneeded changes

* Updating v6

* Updating test to add metrics field

* Adding changelog

* add v5 to backwards compatibility

* Clean up test_previous_version_state, update for v6 (#5440)

* Update test_previous_version_state for v6. Cleanup

* Regenerate, rm breakpoint

* Code checks

* Add assertion that will fail when we bump manifest version

* update tests to automatically tests all previous versions

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Callum McCann <cmccann51@gmail.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-07-06 16:01:14 -05:00
Benoit Perigaud
9fca33cb29 Add docs config and input validation 2022-06-21 09:10:15 +02:00
Benoit Perigaud
6360247d39 Remove node_color from the original docs config 2022-06-21 09:09:35 +02:00
Matt Winkler
f0fbb0e551 add Optional node_color config in Docs dataclass 2022-06-07 09:17:15 -06:00
138 changed files with 8926 additions and 728 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.2.0b1
current_version = 1.3.0a1
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)

View File

@@ -2,6 +2,8 @@
For information on prior major and minor releases, see their changelogs:
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)
* [1.0](https://github.com/dbt-labs/dbt-core/blob/1.0.latest/CHANGELOG.md)
* [0.21](https://github.com/dbt-labs/dbt-core/blob/0.21.latest/CHANGELOG.md)

View File

@@ -1,69 +0,0 @@
## dbt-core 1.2.0-b1 - June 24, 2022
### Features
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
- Adds itertools to modules Jinja namespace ([#5130](https://github.com/dbt-labs/dbt-core/issues/5130), [#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- allow target as an option in profile_template.yml ([#5179](https://github.com/dbt-labs/dbt-core/issues/5179), [#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- seed: Add new macro get_csv_sql ([#5206](https://github.com/dbt-labs/dbt-core/issues/5206), [#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- Grants as Node Configs ([#5189](https://github.com/dbt-labs/dbt-core/issues/5189), [#5230](https://github.com/dbt-labs/dbt-core/pull/5230))
- Adds file selectors and support for file selectors in the default method selector ([#5240](https://github.com/dbt-labs/dbt-core/issues/5240), [#5241](https://github.com/dbt-labs/dbt-core/pull/5241))
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
### Fixes
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- Use yaml renderer (with target context) for rendering selectors ([#5131](https://github.com/dbt-labs/dbt-core/issues/5131), [#5136](https://github.com/dbt-labs/dbt-core/pull/5136))
- Fix retry logic to return values after initial try ([#5023](https://github.com/dbt-labs/dbt-core/issues/5023), [#5137](https://github.com/dbt-labs/dbt-core/pull/5137))
- Scrub secret env vars from CommandError in exception stacktrace ([#5151](https://github.com/dbt-labs/dbt-core/issues/5151), [#5152](https://github.com/dbt-labs/dbt-core/pull/5152))
- Ensure the metric name does not contain spaces ([#4572](https://github.com/dbt-labs/dbt-core/issues/4572), [#5173](https://github.com/dbt-labs/dbt-core/pull/5173))
- When parsing 'all_sources' should be a list of unique dirs ([#5120](https://github.com/dbt-labs/dbt-core/issues/5120), [#5176](https://github.com/dbt-labs/dbt-core/pull/5176))
- Add warning if yaml contains duplicate keys ([#5114](https://github.com/dbt-labs/dbt-core/issues/5114), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146))
- Modifying the drop_test_schema to work better with Redshift issues around locked tables and current transactions ([#5200](https://github.com/dbt-labs/dbt-core/issues/5200), [#5198](https://github.com/dbt-labs/dbt-core/pull/5198))
- Fix column comparison in snapshot_check_all_get_existing_columns for check-strategy snapshots with explicit check_cols defined ([#5222](https://github.com/dbt-labs/dbt-core/issues/5222), [#5223](https://github.com/dbt-labs/dbt-core/pull/5223))
- Changed how `--select state:modified` detects changes for macros nodes depend on ([#5202](https://github.com/dbt-labs/dbt-core/issues/5202), [#5224](https://github.com/dbt-labs/dbt-core/pull/5224))
- Fix column comparison in snapshot_check_all_get_existing_columns to use adapter.get_columns_in_relation ([#5222](https://github.com/dbt-labs/dbt-core/issues/5222), [#5232](https://github.com/dbt-labs/dbt-core/pull/5232))
- Remove docs file from manifest when removing doc node ([#4146](https://github.com/dbt-labs/dbt-core/issues/4146), [#5270](https://github.com/dbt-labs/dbt-core/pull/5270))
- Remove duplicate dbt script entry ([#5314](https://github.com/dbt-labs/dbt-core/issues/5314), [#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
### Under the Hood
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
- Move package deprecation check outside of package cache ([#5068](https://github.com/dbt-labs/dbt-core/issues/5068), [#5069](https://github.com/dbt-labs/dbt-core/pull/5069))
- removal of scaffold first attempt and create_adapter_plugin.py as they are deprecated new scaffold can be found https://github.com/dbt-labs/dbt-database-adapter-scaffold ([#4980](https://github.com/dbt-labs/dbt-core/issues/4980), [#5117](https://github.com/dbt-labs/dbt-core/pull/5117))
- Mypy -> 0.942 + fixed import logic to allow for full mypy coverage ([#4805](https://github.com/dbt-labs/dbt-core/issues/4805), [#5171](https://github.com/dbt-labs/dbt-core/pull/5171))
- Converted dbt list tests to pytest ([#5049](https://github.com/dbt-labs/dbt-core/issues/5049), [#5178](https://github.com/dbt-labs/dbt-core/pull/5178))
- Fix: Call str and repr for UnsetProfileConfig without a RuntimeException ([#5081](https://github.com/dbt-labs/dbt-core/issues/5081), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- Improve tracking error logging message ([#5197](https://github.com/dbt-labs/dbt-core/issues/5197), [#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- Clean up materialization logic: more consistent relation names, loading from cache ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- Use the default Python version for local dev and test instead of requiring Python 3.8 ([#5257](https://github.com/dbt-labs/dbt-core/issues/5257), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- Fix test for context set function ([#5266](https://github.com/dbt-labs/dbt-core/issues/5266), [#5272](https://github.com/dbt-labs/dbt-core/pull/5272))
- Fix pip upgrade step in CI for Windows ([#5321](https://github.com/dbt-labs/dbt-core/issues/5321), [#5320](https://github.com/dbt-labs/dbt-core/pull/5320))
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
### Dependencies
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
### Contributors
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@dependabot[bot]](https://github.com/dependabot[bot]) ([#5141](https://github.com/dbt-labs/dbt-core/pull/5141), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- [@tomasfarias](https://github.com/tomasfarias) ([#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump ubuntu from 20.04 to 22.04"
time: 2022-04-27T19:51:28.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5141"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bumping hologram version"
time: 2022-05-06T16:09:07.000000-05:00
custom:
Author: leahwicz
Issue: "5219"
PR: "5218"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker"
time: 2022-06-13T00:14:56.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5367"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add selector method when reading selector definitions
time: 2022-04-08T11:26:10.713088+10:00
custom:
Author: danieldiamond
Issue: "4821"
PR: "4827"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add set and zip function to contexts
time: 2022-04-23T23:17:56.851793+12:00
custom:
Author: jeremyyeo
Issue: "2345"
PR: "5107"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adds itertools to modules Jinja namespace
time: 2022-04-24T13:26:55.008246+01:00
custom:
Author: bd3dowling
Issue: "5130"
PR: "5140"

View File

@@ -1,7 +0,0 @@
kind: Features
body: allow target as an option in profile_template.yml
time: 2022-04-28T06:56:44.511519-04:00
custom:
Author: alexrosenfeld10
Issue: "5179"
PR: "5184"

View File

@@ -1,7 +0,0 @@
kind: Features
body: 'seed: Add new macro get_csv_sql'
time: 2022-05-03T14:29:34.847959075Z
custom:
Author: adamantike
Issue: "5206"
PR: "5207"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Grants as Node Configs
time: 2022-05-10T20:49:49.197999-04:00
custom:
Author: gshank
Issue: "5189"
PR: "5230"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adds file selectors and support for file selectors in the default method selector
time: 2022-05-12T21:57:48.289674-07:00
custom:
Author: jwills
Issue: "5240"
PR: "5241"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Move cross-db macros from dbt-utils into dbt-core global project
time: 2022-05-18T11:46:04.557104+02:00
custom:
Author: jtcohen6 dbeatty10
Issue: "4813"
PR: "5265"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Prettify duration message at the end of execution
time: 2022-06-11T16:39:31.725960083+01:00
custom:
Author: jared-rimmer
Issue: "5253"
PR: "5364"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Early return from dbt init if no available adapters
time: 2022-06-14T08:20:51.096872718+02:00
custom:
Author: ulisesojeda
Issue: "5365"
PR: "5366"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Adding new cols to check_cols in snapshots
time: 2022-03-17T21:09:16.977086+01:00
custom:
Author: GtheSheep
Issue: "3146"
PR: "4893"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Truncate relation names when appending a suffix that will result in len > 63
characters using make_temp_relation and make_backup_relation macros
time: 2022-03-22T17:37:53.320082-07:00
custom:
Author: epapineau
Issue: "2869"
PR: "4921"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Restore ability to utilize `updated_at` for check_cols snapshots
time: 2022-04-15T11:29:27.063462-06:00
custom:
Author: dbeatty10
Issue: "5076"
PR: "5077"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix retry logic to return values after initial try
time: 2022-04-22T13:12:27.239055-05:00
custom:
Author: emmyoop
Issue: "5023"
PR: "5137"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Use yaml renderer (with target context) for rendering selectors
time: 2022-04-22T13:56:45.147893-04:00
custom:
Author: gshank
Issue: "5131"
PR: "5136"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Scrub secret env vars from CommandError in exception stacktrace
time: 2022-04-25T20:39:24.365495+02:00
custom:
Author: jtcohen6
Issue: "5151"
PR: "5152"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Ensure the metric name does not contain spaces
time: 2022-04-26T20:21:04.360693-04:00
custom:
Author: gshank
Issue: "4572"
PR: "5173"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: When parsing 'all_sources' should be a list of unique dirs
time: 2022-04-27T10:26:48.648388-04:00
custom:
Author: gshank
Issue: "5120"
PR: "5176"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add warning if yaml contains duplicate keys
time: 2022-04-28T10:01:57.893956+12:00
custom:
Author: jeremyyeo
Issue: "5114"
PR: "5146"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Modifying the drop_test_schema to work better with Redshift issues around locked
tables and current transactions
time: 2022-04-29T16:07:42.750046-05:00
custom:
Author: Mcknight-42
Issue: "5200"
PR: "5198"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Fix column comparison in snapshot_check_all_get_existing_columns for check-strategy
snapshots with explicit check_cols defined
time: 2022-05-09T13:00:21.649028+02:00
custom:
Author: jtcohen6
Issue: "5222"
PR: "5223"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Changed how `--select state:modified` detects changes for macros nodes depend
on
time: 2022-05-09T13:13:12.889074-05:00
custom:
Author: stu-k
Issue: "5202"
PR: "5224"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix column comparison in snapshot_check_all_get_existing_columns to use adapter.get_columns_in_relation
time: 2022-05-11T12:32:38.313321+02:00
custom:
Author: jtcohen6
Issue: "5222"
PR: "5232"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove docs file from manifest when removing doc node
time: 2022-05-18T13:46:10.167143-04:00
custom:
Author: gshank
Issue: "4146"
PR: "5270"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove duplicate dbt script entry
time: 2022-06-01T08:13:14.067001+10:00
custom:
Author: groodt
Issue: "5314"
PR: "5304"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Change node ancestor/descendant algo, fixes issue where downstream models aren't
run when using networkx >= 2.8.1
time: 2022-06-01T13:59:08.886215-05:00
custom:
Author: iknox-fa
Issue: "5286"
PR: "5326"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fixing Windows color regression
time: 2022-06-01T19:42:34.263009-04:00
custom:
Author: leahwicz
Issue: "5191"
PR: "5327"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: 'Define compatibility for older manifest versions when using state: selection
methods'
time: 2022-06-08T08:09:14.321735+02:00
custom:
Author: jtcohen6
Issue: "5213"
PR: "5346"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove duplicate key checking introduced in 1.2.0a1
time: 2022-06-23T08:24:31.900647+12:00
custom:
Author: jeremyyeo
Issue: "5331"
PR: "5403"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Migrating 005_simple_seed to the new test framework.
time: 2022-04-09T04:05:39.20045-07:00
custom:
Author: versusfacit
Issue: "200"
PR: "5013"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert 029_docs_generate tests to new framework
time: 2022-04-13T18:30:14.706391-04:00
custom:
Author: gshank
Issue: "5035"
PR: "5058"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Move package deprecation check outside of package cache
time: 2022-04-14T13:22:06.157579-05:00
custom:
Author: emmyoop
Issue: "5068"
PR: "5069"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: removal of scaffold first attempt and create_adapter_plugin.py as they are deprecated new scaffold can be found https://github.com/dbt-labs/dbt-database-adapter-scaffold
time: 2022-04-20T12:00:25.171923-05:00
custom:
Author: McKnight-42
Issue: "4980"
PR: "5117"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Mypy -> 0.942 + fixed import logic to allow for full mypy coverage
time: 2022-04-27T11:21:27.499359-05:00
custom:
Author: iknox-fa
Issue: "4805"
PR: "5171"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Converted dbt list tests to pytest
time: 2022-04-27T14:06:28.882908-05:00
custom:
Author: stu-k
Issue: "5049"
PR: "5178"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: 'Fix: Call str and repr for UnsetProfileConfig without a RuntimeException'
time: 2022-05-03T19:52:12.793729384+02:00
custom:
Author: tomasfarias
Issue: "5081"
PR: "5209"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Improve tracking error logging message
time: 2022-05-04T01:00:31.60387036+02:00
custom:
Author: NicolasPA
Issue: "5197"
PR: "5211"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: 'Clean up materialization logic: more consistent relation names, loading from
cache'
time: 2022-05-09T09:26:28.551068+02:00
custom:
Author: jtcohen6
Issue: "2869"
PR: "4921"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Use the default Python version for local dev and test instead of requiring Python
3.8
time: 2022-05-18T09:51:44.603193-07:00
custom:
Author: jwills
Issue: "5257"
PR: "5269"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix test for context set function
time: 2022-05-18T14:55:22.554316-04:00
custom:
Author: gshank
Issue: "5266"
PR: "5272"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix pip upgrade step in CI for Windows
time: 2022-06-01T10:52:45.872931-04:00
custom:
Author: gshank
Issue: "5321"
PR: "5320"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix unit test test_graph_selection
time: 2022-06-01T11:26:48.725831-04:00
custom:
Author: gshank
Issue: "5323"
PR: "5324"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Update context readme + clean up context code"
time: 2022-06-06T23:03:53.022568+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5334"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: removed script meant for snowflake to snowflake
time: 2022-06-10T11:10:56.080236-05:00
custom:
Author: McKnight-42
Issue: "5361"
PR: "5362"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump mypy from 0.942 to 0.961"
time: 2022-06-07T00:09:25.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5337"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core"
time: 2022-06-17T00:18:49.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5388"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump black from 22.3.0 to 22.6.0"
time: 2022-06-29T00:08:31.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5420"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Move type_* macros from dbt-utils into dbt-core, with tests
time: 2022-06-30T13:54:52.165139+02:00
custom:
Author: jtcohen6
Issue: "5317"
PR: "5428"

View File

@@ -0,0 +1,8 @@
kind: Features
body: Add reusable function for retrying adapter connections. Utilize said function
to add retries for Postgres (and Redshift).
time: 2022-07-15T03:55:55.270637265+02:00
custom:
Author: tomasfarias
Issue: "5022"
PR: "5432"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add inheritance to materialization macro resolution
time: 2022-06-08T10:42:51.839945-04:00
custom:
Author: volkangurel
Issue: "4646"
PR: "5348"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Improve pluralizations for Documentation and SqlOperation NodeTypes
time: 2022-06-09T11:52:46.578469-05:00
custom:
Author: pdebelak
Issue: "5352"
PR: "5356"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Properly use quotes for Snowflake snapshots when checking all columns
time: 2022-06-17T11:44:43.978834+02:00
custom:
Author: pquadri
Issue: "2975"
PR: "5389"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: fixes handling of RESET color code with USE_COLORS=False
time: 2022-06-17T16:00:27.038058-04:00
custom:
Author: darin-reify
Issue: "5288"
PR: "5394"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Rename try to strict for more intuitiveness
time: 2022-07-15T23:11:48.327928+12:00
custom:
Author: jeremyyeo
Issue: "5475"
PR: "5477"

View File

@@ -1,7 +0,0 @@
kind: Security
body: Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README
time: 2022-06-06T23:03:53.022568+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5334"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Added the suggested RegEx to check the SemVer string within a package dependency
and improved invalid version error handling.
time: 2022-06-13T17:07:03.773668-05:00
custom:
Author: fivetran-joemarkiewicz
Issue: "5201"
PR: "5370"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: 'Add annotation to render_value method reimplemented in #5334'
time: 2022-06-16T12:05:16.474078+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5382"

View File

@@ -38,6 +38,7 @@ jobs:
name: code-quality
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Check out the repository
@@ -65,6 +66,7 @@ jobs:
name: unit test / python ${{ matrix.python-version }}
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
@@ -109,6 +111,7 @@ jobs:
name: integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 45
strategy:
fail-fast: false
@@ -125,6 +128,9 @@ jobs:
TOXENV: integration
PYTEST_ADDOPTS: "-v --color=yes -n4 --csv integration_results.csv"
DBT_INVOCATION_ENV: github-actions
DBT_TEST_USER_1: dbt_test_user_1
DBT_TEST_USER_2: dbt_test_user_2
DBT_TEST_USER_3: dbt_test_user_3
steps:
- name: Check out the repository

View File

@@ -30,6 +30,11 @@ jobs:
LOG_DIR: "/home/runner/work/dbt-core/dbt-core/logs"
# tells integration tests to output into json format
DBT_LOG_FORMAT: "json"
# Additional test users
DBT_TEST_USER_1: dbt_test_user_1
DBT_TEST_USER_2: dbt_test_user_2
DBT_TEST_USER_3: dbt_test_user_3
steps:
- name: checkout dev
uses: actions/checkout@v2

View File

@@ -6,81 +6,12 @@
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.2.0-b1 - June 24, 2022
### Features
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
- Adds itertools to modules Jinja namespace ([#5130](https://github.com/dbt-labs/dbt-core/issues/5130), [#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- allow target as an option in profile_template.yml ([#5179](https://github.com/dbt-labs/dbt-core/issues/5179), [#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- seed: Add new macro get_csv_sql ([#5206](https://github.com/dbt-labs/dbt-core/issues/5206), [#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- Grants as Node Configs ([#5189](https://github.com/dbt-labs/dbt-core/issues/5189), [#5230](https://github.com/dbt-labs/dbt-core/pull/5230))
- Adds file selectors and support for file selectors in the default method selector ([#5240](https://github.com/dbt-labs/dbt-core/issues/5240), [#5241](https://github.com/dbt-labs/dbt-core/pull/5241))
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
### Fixes
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- Use yaml renderer (with target context) for rendering selectors ([#5131](https://github.com/dbt-labs/dbt-core/issues/5131), [#5136](https://github.com/dbt-labs/dbt-core/pull/5136))
- Fix retry logic to return values after initial try ([#5023](https://github.com/dbt-labs/dbt-core/issues/5023), [#5137](https://github.com/dbt-labs/dbt-core/pull/5137))
- Scrub secret env vars from CommandError in exception stacktrace ([#5151](https://github.com/dbt-labs/dbt-core/issues/5151), [#5152](https://github.com/dbt-labs/dbt-core/pull/5152))
- Ensure the metric name does not contain spaces ([#4572](https://github.com/dbt-labs/dbt-core/issues/4572), [#5173](https://github.com/dbt-labs/dbt-core/pull/5173))
- When parsing 'all_sources' should be a list of unique dirs ([#5120](https://github.com/dbt-labs/dbt-core/issues/5120), [#5176](https://github.com/dbt-labs/dbt-core/pull/5176))
- Add warning if yaml contains duplicate keys ([#5114](https://github.com/dbt-labs/dbt-core/issues/5114), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146))
- Modifying the drop_test_schema to work better with Redshift issues around locked tables and current transactions ([#5200](https://github.com/dbt-labs/dbt-core/issues/5200), [#5198](https://github.com/dbt-labs/dbt-core/pull/5198))
- Fix column comparison in snapshot_check_all_get_existing_columns for check-strategy snapshots with explicit check_cols defined ([#5222](https://github.com/dbt-labs/dbt-core/issues/5222), [#5223](https://github.com/dbt-labs/dbt-core/pull/5223))
- Changed how `--select state:modified` detects changes for macros nodes depend on ([#5202](https://github.com/dbt-labs/dbt-core/issues/5202), [#5224](https://github.com/dbt-labs/dbt-core/pull/5224))
- Fix column comparison in snapshot_check_all_get_existing_columns to use adapter.get_columns_in_relation ([#5222](https://github.com/dbt-labs/dbt-core/issues/5222), [#5232](https://github.com/dbt-labs/dbt-core/pull/5232))
- Remove docs file from manifest when removing doc node ([#4146](https://github.com/dbt-labs/dbt-core/issues/4146), [#5270](https://github.com/dbt-labs/dbt-core/pull/5270))
- Remove duplicate dbt script entry ([#5314](https://github.com/dbt-labs/dbt-core/issues/5314), [#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
### Under the Hood
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
- Move package deprecation check outside of package cache ([#5068](https://github.com/dbt-labs/dbt-core/issues/5068), [#5069](https://github.com/dbt-labs/dbt-core/pull/5069))
- removal of scaffold first attempt and create_adapter_plugin.py as they are deprecated new scaffold can be found https://github.com/dbt-labs/dbt-database-adapter-scaffold ([#4980](https://github.com/dbt-labs/dbt-core/issues/4980), [#5117](https://github.com/dbt-labs/dbt-core/pull/5117))
- Mypy -> 0.942 + fixed import logic to allow for full mypy coverage ([#4805](https://github.com/dbt-labs/dbt-core/issues/4805), [#5171](https://github.com/dbt-labs/dbt-core/pull/5171))
- Converted dbt list tests to pytest ([#5049](https://github.com/dbt-labs/dbt-core/issues/5049), [#5178](https://github.com/dbt-labs/dbt-core/pull/5178))
- Fix: Call str and repr for UnsetProfileConfig without a RuntimeException ([#5081](https://github.com/dbt-labs/dbt-core/issues/5081), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- Improve tracking error logging message ([#5197](https://github.com/dbt-labs/dbt-core/issues/5197), [#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- Clean up materialization logic: more consistent relation names, loading from cache ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- Use the default Python version for local dev and test instead of requiring Python 3.8 ([#5257](https://github.com/dbt-labs/dbt-core/issues/5257), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- Fix test for context set function ([#5266](https://github.com/dbt-labs/dbt-core/issues/5266), [#5272](https://github.com/dbt-labs/dbt-core/pull/5272))
- Fix pip upgrade step in CI for Windows ([#5321](https://github.com/dbt-labs/dbt-core/issues/5321), [#5320](https://github.com/dbt-labs/dbt-core/pull/5320))
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
### Dependencies
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
### Contributors
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@dependabot[bot]](https://github.com/dependabot[bot]) ([#5141](https://github.com/dbt-labs/dbt-core/pull/5141), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- [@tomasfarias](https://github.com/tomasfarias) ([#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
## Previous Releases
For information on prior major and minor releases, see their changelogs:
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)
* [1.0](https://github.com/dbt-labs/dbt-core/blob/1.0.latest/CHANGELOG.md)
* [0.21](https://github.com/dbt-labs/dbt-core/blob/0.21.latest/CHANGELOG.md)

View File

@@ -1,10 +1,24 @@
import abc
import os
from time import sleep
import sys
# multiprocessing.RLock is a function returning this type
from multiprocessing.synchronize import RLock
from threading import get_ident
from typing import Dict, Tuple, Hashable, Optional, ContextManager, List
from typing import (
Any,
Dict,
Tuple,
Hashable,
Optional,
ContextManager,
List,
Type,
Union,
Iterable,
Callable,
)
import agate
@@ -21,6 +35,7 @@ from dbt.contracts.graph.manifest import Manifest
from dbt.adapters.base.query_headers import (
MacroQueryStringSetter,
)
from dbt.events import AdapterLogger
from dbt.events.functions import fire_event
from dbt.events.types import (
NewConnection,
@@ -34,6 +49,9 @@ from dbt.events.types import (
)
from dbt import flags
SleepTime = Union[int, float] # As taken by time.sleep.
AdapterHandle = Any # Adapter connection handle objects can be any class.
class BaseConnectionManager(metaclass=abc.ABCMeta):
"""Methods to implement:
@@ -159,6 +177,94 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
conn.name = conn_name
return conn
@classmethod
def retry_connection(
cls,
connection: Connection,
connect: Callable[[], AdapterHandle],
logger: AdapterLogger,
retryable_exceptions: Iterable[Type[Exception]],
retry_limit: int = 1,
retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1,
_attempts: int = 0,
) -> Connection:
"""Given a Connection, set its handle by calling connect.
The calls to connect will be retried up to retry_limit times to deal with transient
connection errors. By default, one retry will be attempted if retryable_exceptions is set.
:param Connection connection: An instance of a Connection that needs a handle to be set,
usually when attempting to open it.
:param connect: A callable that returns the appropiate connection handle for a
given adapter. This callable will be retried retry_limit times if a subclass of any
Exception in retryable_exceptions is raised by connect.
:type connect: Callable[[], AdapterHandle]
:param AdapterLogger logger: A logger to emit messages on retry attempts or errors. When
handling expected errors, we call debug, and call warning on unexpected errors or when
all retry attempts have been exhausted.
:param retryable_exceptions: An iterable of exception classes that if raised by
connect should trigger a retry.
:type retryable_exceptions: Iterable[Type[Exception]]
:param int retry_limit: How many times to retry the call to connect. If this limit
is exceeded before a successful call, a FailedToConnectException will be raised.
Must be non-negative.
:param retry_timeout: Time to wait between attempts to connect. Can also take a
Callable that takes the number of attempts so far, beginning at 0, and returns an int
or float to be passed to time.sleep.
:type retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1
:param int _attempts: Parameter used to keep track of the number of attempts in calling the
connect function across recursive calls. Passed as an argument to retry_timeout if it
is a Callable. This parameter should not be set by the initial caller.
:raises dbt.exceptions.FailedToConnectException: Upon exhausting all retry attempts without
successfully acquiring a handle.
:return: The given connection with its appropriate state and handle attributes set
depending on whether we successfully acquired a handle or not.
"""
timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout
if timeout < 0:
raise dbt.exceptions.FailedToConnectException(
"retry_timeout cannot be negative or return a negative time."
)
if retry_limit < 0 or retry_limit > sys.getrecursionlimit():
# This guard is not perfect others may add to the recursion limit (e.g. built-ins).
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException("retry_limit cannot be negative")
try:
connection.handle = connect()
connection.state = ConnectionState.OPEN
return connection
except tuple(retryable_exceptions) as e:
if retry_limit <= 0:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException(str(e))
logger.debug(
f"Got a retryable error when attempting to open a {cls.TYPE} connection.\n"
f"{retry_limit} attempts remaining. Retrying in {timeout} seconds.\n"
f"Error:\n{e}"
)
sleep(timeout)
return cls.retry_connection(
connection=connection,
connect=connect,
logger=logger,
retry_limit=retry_limit - 1,
retry_timeout=retry_timeout,
retryable_exceptions=retryable_exceptions,
_attempts=_attempts + 1,
)
except Exception as e:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException(str(e))
@abc.abstractmethod
def cancel_open(self) -> Optional[List[str]]:
"""Cancel all open connections on the adapter. (passable)"""

View File

@@ -159,6 +159,7 @@ class BaseAdapter(metaclass=AdapterMeta):
- convert_datetime_type
- convert_date_type
- convert_time_type
- standardize_grants_dict
Macros:
- get_catalog
@@ -538,6 +539,33 @@ class BaseAdapter(metaclass=AdapterMeta):
"`list_relations_without_caching` is not implemented for this " "adapter!"
)
###
# Methods about grants
###
@available
def standardize_grants_dict(self, grants_table: agate.Table) -> dict:
"""Translate the result of `show grants` (or equivalent) to match the
grants which a user would configure in their project.
Ideally, the SQL to show grants should also be filtering:
filter OUT any grants TO the current user/role (e.g. OWNERSHIP).
If that's not possible in SQL, it can be done in this method instead.
:param grants_table: An agate table containing the query result of
the SQL returned by get_show_grant_sql
:return: A standardized dictionary matching the `grants` config
:rtype: dict
"""
grants_dict: Dict[str, List[str]] = {}
for row in grants_table:
grantee = row["grantee"]
privilege = row["privilege_type"]
if privilege in grants_dict.keys():
grants_dict[privilege].append(grantee)
else:
grants_dict.update({privilege: [grantee]})
return grants_dict
###
# Provided methods about relations
###

View File

@@ -397,6 +397,8 @@ class Compiler:
linker.dependency(node.unique_id, (manifest.nodes[dependency].unique_id))
elif dependency in manifest.sources:
linker.dependency(node.unique_id, (manifest.sources[dependency].unique_id))
elif dependency in manifest.metrics:
linker.dependency(node.unique_id, (manifest.metrics[dependency].unique_id))
else:
dependency_not_found(node, dependency)

View File

@@ -15,7 +15,7 @@ from typing_extensions import Protocol, runtime_checkable
import hashlib
import os
from dbt import deprecations
from dbt import flags, deprecations
from dbt.clients.system import resolve_path_from_base
from dbt.clients.system import path_exists
from dbt.clients.system import load_file_contents
@@ -142,6 +142,13 @@ def _all_source_paths(
T = TypeVar("T")
def flag_or(flag: Optional[T], value: Optional[T], default: T) -> T:
if flag is None:
return value_or(value, default)
else:
return flag
def value_or(value: Optional[T], default: T) -> T:
if value is None:
return default
@@ -356,9 +363,9 @@ class PartialProject(RenderComponents):
docs_paths: List[str] = value_or(cfg.docs_paths, all_source_paths)
asset_paths: List[str] = value_or(cfg.asset_paths, [])
target_path: str = value_or(cfg.target_path, "target")
target_path: str = flag_or(flags.TARGET_PATH, cfg.target_path, "target")
clean_targets: List[str] = value_or(cfg.clean_targets, [target_path])
log_path: str = value_or(cfg.log_path, "logs")
log_path: str = flag_or(flags.LOG_PATH, cfg.log_path, "logs")
packages_install_path: str = value_or(cfg.packages_install_path, "dbt_packages")
# in the default case we'll populate this once we know the adapter type
# It would be nice to just pass along a Quoting here, but that would

View File

@@ -1,6 +1,6 @@
import json
import os
from typing import Any, Dict, NoReturn, Optional, Mapping, Iterable, Set
from typing import Any, Dict, NoReturn, Optional, Mapping, Iterable, Set, List
from dbt import flags
from dbt import tracking
@@ -474,19 +474,17 @@ class BaseContext(metaclass=ContextMeta):
@contextmember
@staticmethod
def try_set(value: Iterable[Any]) -> Set[Any]:
"""The `try_set` context method can be used to convert any iterable
def set_strict(value: Iterable[Any]) -> Set[Any]:
"""The `set_strict` context method can be used to convert any iterable
to a sequence of iterable elements that are unique (a set). The
difference to the `set` context method is that the `try_set` method
difference to the `set` context method is that the `set_strict` method
will raise an exception on a TypeError.
:param value: The iterable
:param default: A default value to return if the `value` argument
is not an iterable
Usage:
{% set my_list = [1, 2, 2, 3] %}
{% set my_set = try_set(my_list) %}
{% set my_set = set_strict(my_list) %}
{% do log(my_set) %} {# {1, 2, 3} #}
"""
try:
@@ -497,7 +495,7 @@ class BaseContext(metaclass=ContextMeta):
@contextmember("zip")
@staticmethod
def _zip(*args: Iterable[Any], default: Any = None) -> Optional[Iterable[Any]]:
"""The `try_zip` context method can be used to used to return
"""The `zip` context method can be used to used to return
an iterator of tuples, where the i-th tuple contains the i-th
element from each of the argument iterables.
@@ -518,21 +516,19 @@ class BaseContext(metaclass=ContextMeta):
@contextmember
@staticmethod
def try_zip(*args: Iterable[Any]) -> Iterable[Any]:
"""The `try_zip` context method can be used to used to return
def zip_strict(*args: Iterable[Any]) -> Iterable[Any]:
"""The `zip_strict` context method can be used to used to return
an iterator of tuples, where the i-th tuple contains the i-th
element from each of the argument iterables. The difference to the
`zip` context method is that the `try_zip` method will raise an
`zip` context method is that the `zip_strict` method will raise an
exception on a TypeError.
:param *args: Any number of iterables
:param default: A default value to return if `*args` is not
iterable
Usage:
{% set my_list_a = [1, 2] %}
{% set my_list_b = ['alice', 'bob'] %}
{% set my_zip = try_zip(my_list_a, my_list_b) | list %}
{% set my_zip = zip_strict(my_list_a, my_list_b) | list %}
{% do log(my_set) %} {# [(1, 'alice'), (2, 'bob')] #}
"""
try:
@@ -657,6 +653,35 @@ class BaseContext(metaclass=ContextMeta):
print(msg)
return ""
@contextmember
@staticmethod
def diff_of_two_dicts(
dict_a: Dict[str, List[str]], dict_b: Dict[str, List[str]]
) -> Dict[str, List[str]]:
"""
Given two dictionaries of type Dict[str, List[str]]:
dict_a = {'key_x': ['value_1', 'VALUE_2'], 'KEY_Y': ['value_3']}
dict_b = {'key_x': ['value_1'], 'key_z': ['value_4']}
Return the same dictionary representation of dict_a MINUS dict_b,
performing a case-insensitive comparison between the strings in each.
All keys returned will be in the original case of dict_a.
returns {'key_x': ['VALUE_2'], 'KEY_Y': ['value_3']}
"""
dict_diff = {}
dict_b_lowered = {k.casefold(): [x.casefold() for x in v] for k, v in dict_b.items()}
for k in dict_a:
if k.casefold() in dict_b_lowered.keys():
diff = []
for v in dict_a[k]:
if v.casefold() not in dict_b_lowered[k.casefold()]:
diff.append(v)
if diff:
dict_diff.update({k: diff})
else:
dict_diff.update({k: dict_a[k]})
return dict_diff
def generate_base_context(cli_vars: Dict[str, Any]) -> Dict[str, Any]:
ctx = BaseContext(cli_vars)

View File

@@ -40,6 +40,7 @@ from dbt.contracts.graph.parsed import (
ParsedSeedNode,
ParsedSourceDefinition,
)
from dbt.contracts.graph.metrics import MetricReference, ResolvedMetricReference
from dbt.exceptions import (
CompilationException,
ParsingException,
@@ -50,7 +51,9 @@ from dbt.exceptions import (
missing_config,
raise_compiler_error,
ref_invalid_args,
metric_invalid_args,
ref_target_not_found,
metric_target_not_found,
ref_bad_context,
source_target_not_found,
wrapped_exports,
@@ -199,7 +202,7 @@ class BaseResolver(metaclass=abc.ABCMeta):
return self.db_wrapper.Relation
@abc.abstractmethod
def __call__(self, *args: str) -> Union[str, RelationProxy]:
def __call__(self, *args: str) -> Union[str, RelationProxy, MetricReference]:
pass
@@ -265,6 +268,41 @@ class BaseSourceResolver(BaseResolver):
return self.resolve(args[0], args[1])
class BaseMetricResolver(BaseResolver):
def resolve(self, name: str, package: Optional[str] = None) -> MetricReference:
...
def _repack_args(self, name: str, package: Optional[str]) -> List[str]:
if package is None:
return [name]
else:
return [package, name]
def validate_args(self, name: str, package: Optional[str]):
if not isinstance(name, str):
raise CompilationException(
f"The name argument to metric() must be a string, got {type(name)}"
)
if package is not None and not isinstance(package, str):
raise CompilationException(
f"The package argument to metric() must be a string or None, got {type(package)}"
)
def __call__(self, *args: str) -> MetricReference:
name: str
package: Optional[str] = None
if len(args) == 1:
name = args[0]
elif len(args) == 2:
package, name = args
else:
metric_invalid_args(self.model, args)
self.validate_args(name, package)
return self.resolve(name, package)
class Config(Protocol):
def __init__(self, model, context_config: Optional[ContextConfig]):
...
@@ -511,6 +549,34 @@ class RuntimeSourceResolver(BaseSourceResolver):
return self.Relation.create_from_source(target_source)
# metric` implementations
class ParseMetricResolver(BaseMetricResolver):
def resolve(self, name: str, package: Optional[str] = None) -> MetricReference:
self.model.metrics.append(self._repack_args(name, package))
return MetricReference(name, package)
class RuntimeMetricResolver(BaseMetricResolver):
def resolve(self, target_name: str, target_package: Optional[str] = None) -> MetricReference:
target_metric = self.manifest.resolve_metric(
target_name,
target_package,
self.current_project,
self.model.package_name,
)
if target_metric is None or isinstance(target_metric, Disabled):
# TODO : Use a different exception!!
metric_target_not_found(
self.model,
target_name,
target_package,
)
return ResolvedMetricReference(target_metric, self.manifest, self.Relation)
# `var` implementations.
class ModelConfiguredVar(Var):
def __init__(
@@ -568,6 +634,7 @@ class Provider(Protocol):
Var: Type[ModelConfiguredVar]
ref: Type[BaseRefResolver]
source: Type[BaseSourceResolver]
metric: Type[BaseMetricResolver]
class ParseProvider(Provider):
@@ -577,6 +644,7 @@ class ParseProvider(Provider):
Var = ParseVar
ref = ParseRefResolver
source = ParseSourceResolver
metric = ParseMetricResolver
class GenerateNameProvider(Provider):
@@ -586,6 +654,7 @@ class GenerateNameProvider(Provider):
Var = RuntimeVar
ref = ParseRefResolver
source = ParseSourceResolver
metric = ParseMetricResolver
class RuntimeProvider(Provider):
@@ -595,6 +664,7 @@ class RuntimeProvider(Provider):
Var = RuntimeVar
ref = RuntimeRefResolver
source = RuntimeSourceResolver
metric = RuntimeMetricResolver
class OperationProvider(RuntimeProvider):
@@ -778,6 +848,10 @@ class ProviderContext(ManifestContext):
def source(self) -> Callable:
return self.provider.source(self.db_wrapper, self.model, self.config, self.manifest)
@contextproperty
def metric(self) -> Callable:
return self.provider.metric(self.db_wrapper, self.model, self.config, self.manifest)
@contextproperty("config")
def ctx_config(self) -> Config:
"""The `config` variable exists to handle end-user configuration for
@@ -1355,7 +1429,7 @@ class MetricRefResolver(BaseResolver):
if not isinstance(name, str):
raise ParsingException(
f"In a metrics section in {self.model.original_file_path} "
f"the name argument to ref() must be a string"
"the name argument to ref() must be a string"
)
@@ -1373,6 +1447,12 @@ def generate_parse_metrics(
project,
manifest,
),
"metric": ParseMetricResolver(
None,
metric,
project,
manifest,
),
}

View File

@@ -183,6 +183,39 @@ class RefableLookup(dbtClassMixin):
return manifest.nodes[unique_id]
class MetricLookup(dbtClassMixin):
def __init__(self, manifest: "Manifest"):
self.storage: Dict[str, Dict[PackageName, UniqueID]] = {}
self.populate(manifest)
def get_unique_id(self, search_name, package: Optional[PackageName]):
return find_unique_id_for_package(self.storage, search_name, package)
def find(self, search_name, package: Optional[PackageName], manifest: "Manifest"):
unique_id = self.get_unique_id(search_name, package)
if unique_id is not None:
return self.perform_lookup(unique_id, manifest)
return None
def add_metric(self, metric: ParsedMetric):
if metric.search_name not in self.storage:
self.storage[metric.search_name] = {}
self.storage[metric.search_name][metric.package_name] = metric.unique_id
def populate(self, manifest):
for metric in manifest.metrics.values():
if hasattr(metric, "name"):
self.add_metric(metric)
def perform_lookup(self, unique_id: UniqueID, manifest: "Manifest") -> ParsedMetric:
if unique_id not in manifest.metrics:
raise dbt.exceptions.InternalException(
f"Metric {unique_id} found in cache but not found in manifest"
)
return manifest.metrics[unique_id]
# This handles both models/seeds/snapshots and sources
class DisabledLookup(dbtClassMixin):
def __init__(self, manifest: "Manifest"):
@@ -431,6 +464,9 @@ class Disabled(Generic[D]):
target: D
MaybeMetricNode = Optional[ParsedMetric]
MaybeDocumentation = Optional[ParsedDocumentation]
@@ -592,6 +628,9 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
_ref_lookup: Optional[RefableLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
)
_metric_lookup: Optional[MetricLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
)
_disabled_lookup: Optional[DisabledLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
)
@@ -837,6 +876,12 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self._ref_lookup = RefableLookup(self)
return self._ref_lookup
@property
def metric_lookup(self) -> MetricLookup:
if self._metric_lookup is None:
self._metric_lookup = MetricLookup(self)
return self._metric_lookup
def rebuild_ref_lookup(self):
self._ref_lookup = RefableLookup(self)
@@ -912,6 +957,22 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
return Disabled(disabled[0])
return None
def resolve_metric(
self,
target_metric_name: str,
target_metric_package: Optional[str],
current_project: str,
node_package: str,
) -> MaybeMetricNode:
metric: Optional[ParsedMetric] = None
candidates = _search_packages(current_project, node_package, target_metric_package)
for pkg in candidates:
metric = self.metric_lookup.find(target_metric_name, pkg, self)
if metric is not None:
return metric
return None
# Called by DocsRuntimeContext.doc
def resolve_doc(
self,
@@ -1076,6 +1137,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self._doc_lookup,
self._source_lookup,
self._ref_lookup,
self._metric_lookup,
self._disabled_lookup,
self._analysis_lookup,
)
@@ -1095,7 +1157,7 @@ AnyManifest = Union[Manifest, MacroManifest]
@dataclass
@schema_version("manifest", 5)
@schema_version("manifest", 6)
class WritableManifest(ArtifactMixin):
nodes: Mapping[UniqueID, ManifestNode] = field(
metadata=dict(description=("The nodes defined in the dbt project and its dependencies"))
@@ -1141,7 +1203,7 @@ class WritableManifest(ArtifactMixin):
@classmethod
def compatible_previous_versions(self):
return [("manifest", 4)]
return [("manifest", 4), ("manifest", 5)]
def __post_serialize__(self, dct):
for unique_id, node in dct["nodes"].items():

View File

@@ -0,0 +1,70 @@
from dbt.node_types import NodeType
class MetricReference(object):
def __init__(self, metric_name, package_name=None):
self.metric_name = metric_name
self.package_name = package_name
def __str__(self):
return f"{self.metric_name}"
class ResolvedMetricReference(MetricReference):
"""
Simple proxy over a ParsedMetric which delegates property
lookups to the underlying node. Also adds helper functions
for working with metrics (ie. __str__ and templating functions)
"""
def __init__(self, node, manifest, Relation):
super().__init__(node.name, node.package_name)
self.node = node
self.manifest = manifest
self.Relation = Relation
def __getattr__(self, key):
return getattr(self.node, key)
def __str__(self):
return f"{self.node.name}"
@classmethod
def parent_metrics(cls, metric_node, manifest):
yield metric_node
for parent_unique_id in metric_node.depends_on.nodes:
node = manifest.metrics.get(parent_unique_id)
if node and node.resource_type == NodeType.Metric:
yield from cls.parent_metrics(node, manifest)
def parent_models(self):
in_scope_metrics = list(self.parent_metrics(self.node, self.manifest))
to_return = {
"base": [],
"derived": [],
}
for metric in in_scope_metrics:
if metric.type == "expression":
to_return["derived"].append(
{"metric_source": None, "metric": metric, "is_derived": True}
)
else:
for node_unique_id in metric.depends_on.nodes:
node = self.manifest.nodes.get(node_unique_id)
if node and node.resource_type in NodeType.refable():
to_return["base"].append(
{
"metric_relation_node": node,
"metric_relation": self.Relation.create(
database=node.database,
schema=node.schema,
identifier=node.alias,
),
"metric": metric,
"is_derived": False,
}
)
return to_return

View File

@@ -7,7 +7,8 @@ from dbt.dataclass_schema import (
ValidationError,
register_pattern,
)
from dbt.contracts.graph.unparsed import AdditionalPropertiesAllowed
from dbt.contracts.graph.unparsed import AdditionalPropertiesAllowed, Docs
from dbt.contracts.graph.utils import validate_color
from dbt.exceptions import InternalException, CompilationException
from dbt.contracts.util import Replaceable, list_str
from dbt import hooks
@@ -285,7 +286,7 @@ class BaseConfig(AdditionalPropertiesAllowed, Replaceable):
# 'meta' moved here from node
mergebehavior = {
"append": ["pre-hook", "pre_hook", "post-hook", "post_hook", "tags"],
"update": ["quoting", "column_types", "meta"],
"update": ["quoting", "column_types", "meta", "docs"],
"dict_key_append": ["grants"],
}
@@ -460,6 +461,20 @@ class NodeConfig(NodeAndTestConfig):
grants: Dict[str, Any] = field(
default_factory=dict, metadata=MergeBehavior.DictKeyAppend.meta()
)
docs: Docs = field(
default_factory=lambda: Docs(show=True),
metadata=MergeBehavior.Update.meta(),
)
# we validate that node_color has a suitable value to prevent dbt-docs from crashing
def __post_init__(self):
if self.docs.node_color:
node_color = self.docs.node_color
if not validate_color(node_color):
raise ValidationError(
f"Invalid color name for docs.node_color: {node_color}. "
"It is neither a valid HTML color name nor a valid HEX code."
)
@classmethod
def __pre_deserialize__(cls, data):

View File

@@ -157,7 +157,6 @@ class ParsedNodeMixins(dbtClassMixin):
self.created_at = time.time()
self.description = patch.description
self.columns = patch.columns
self.docs = patch.docs
def get_materialization(self):
return self.config.materialized
@@ -198,11 +197,12 @@ class ParsedNodeDefaults(NodeInfoMixin, ParsedNodeMandatory):
tags: List[str] = field(default_factory=list)
refs: List[List[str]] = field(default_factory=list)
sources: List[List[str]] = field(default_factory=list)
metrics: List[List[str]] = field(default_factory=list)
depends_on: DependsOn = field(default_factory=DependsOn)
description: str = field(default="")
columns: Dict[str, ColumnInfo] = field(default_factory=dict)
meta: Dict[str, Any] = field(default_factory=dict)
docs: Docs = field(default_factory=Docs)
docs: Docs = field(default_factory=lambda: Docs(show=True))
patch_path: Optional[str] = None
compiled_path: Optional[str] = None
build_path: Optional[str] = None
@@ -793,24 +793,32 @@ class ParsedExposure(UnparsedBaseNode, HasUniqueID, HasFqn):
)
@dataclass
class MetricReference(dbtClassMixin, Replaceable):
sql: Optional[Union[str, int]]
unique_id: Optional[str]
@dataclass
class ParsedMetric(UnparsedBaseNode, HasUniqueID, HasFqn):
model: str
name: str
description: str
label: str
type: str
sql: Optional[str]
sql: str
timestamp: Optional[str]
filters: List[MetricFilter]
time_grains: List[str]
dimensions: List[str]
model: Optional[str] = None
model_unique_id: Optional[str] = None
resource_type: NodeType = NodeType.Metric
meta: Dict[str, Any] = field(default_factory=dict)
tags: List[str] = field(default_factory=list)
sources: List[List[str]] = field(default_factory=list)
depends_on: DependsOn = field(default_factory=DependsOn)
refs: List[List[str]] = field(default_factory=list)
metrics: List[List[str]] = field(default_factory=list)
created_at: float = field(default_factory=lambda: time.time())
@property

View File

@@ -1,15 +1,11 @@
from dbt.node_types import NodeType
from dbt.contracts.util import (
AdditionalPropertiesMixin,
Mergeable,
Replaceable,
)
from dbt.contracts.util import AdditionalPropertiesMixin, Mergeable, Replaceable
# trigger the PathEncoder
import dbt.helper_types # noqa:F401
from dbt.exceptions import CompilationException, ParsingException
from dbt.dataclass_schema import dbtClassMixin, StrEnum, ExtensibleDbtClassMixin
from dbt.dataclass_schema import dbtClassMixin, StrEnum, ExtensibleDbtClassMixin, ValidationError
from dataclasses import dataclass, field
from datetime import timedelta
@@ -80,6 +76,7 @@ class UnparsedRunHook(UnparsedNode):
@dataclass
class Docs(dbtClassMixin, Replaceable):
show: bool = True
node_color: Optional[str] = None
@dataclass
@@ -448,12 +445,15 @@ class MetricFilter(dbtClassMixin, Replaceable):
@dataclass
class UnparsedMetric(dbtClassMixin, Replaceable):
model: str
# TODO : verify that this disallows metric names with spaces
# TODO: fix validation that you broke :p
# name: Identifier
name: str
label: str
type: str
model: Optional[str] = None
description: str = ""
sql: Optional[str] = None
sql: Union[str, int] = ""
timestamp: Optional[str] = None
time_grains: List[str] = field(default_factory=list)
dimensions: List[str] = field(default_factory=list)
@@ -463,6 +463,15 @@ class UnparsedMetric(dbtClassMixin, Replaceable):
@classmethod
def validate(cls, data):
# super().validate(data)
# TODO: putting this back for now to get tests passing. Do we want to implement name: Identifier?
super(UnparsedMetric, cls).validate(data)
if "name" in data and " " in data["name"]:
raise ParsingException(f"Metrics name '{data['name']}' cannot contain spaces")
# TODO: Expressions _cannot_ have `model` properties
if data.get("model") is None and data.get("type") != "expression":
raise ValidationError("Non-expression metrics require a 'model' property")
if data.get("model") is not None and data.get("type") == "expression":
raise ValidationError("Expression metrics cannot have a 'model' property")

View File

@@ -0,0 +1,153 @@
import re
HTML_COLORS = [
"aliceblue",
"antiquewhite",
"aqua",
"aquamarine",
"azure",
"beige",
"bisque",
"black",
"blanchedalmond",
"blue",
"blueviolet",
"brown",
"burlywood",
"cadetblue",
"chartreuse",
"chocolate",
"coral",
"cornflowerblue",
"cornsilk",
"crimson",
"cyan",
"darkblue",
"darkcyan",
"darkgoldenrod",
"darkgray",
"darkgreen",
"darkkhaki",
"darkmagenta",
"darkolivegreen",
"darkorange",
"darkorchid",
"darkred",
"darksalmon",
"darkseagreen",
"darkslateblue",
"darkslategray",
"darkturquoise",
"darkviolet",
"deeppink",
"deepskyblue",
"dimgray",
"dodgerblue",
"firebrick",
"floralwhite",
"forestgreen",
"fuchsia",
"gainsboro",
"ghostwhite",
"gold",
"goldenrod",
"gray",
"green",
"greenyellow",
"honeydew",
"hotpink",
"indianred",
"indigo",
"ivory",
"khaki",
"lavender",
"lavenderblush",
"lawngreen",
"lemonchiffon",
"lightblue",
"lightcoral",
"lightcyan",
"lightgoldenrodyellow",
"lightgray",
"lightgreen",
"lightpink",
"lightsalmon",
"lightsalmon",
"lightseagreen",
"lightskyblue",
"lightslategray",
"lightsteelblue",
"lightyellow",
"lime",
"limegreen",
"linen",
"magenta",
"maroon",
"mediumaquamarine",
"mediumblue",
"mediumorchid",
"mediumpurple",
"mediumseagreen",
"mediumslateblue",
"mediumslateblue",
"mediumspringgreen",
"mediumturquoise",
"mediumvioletred",
"midnightblue",
"mintcream",
"mistyrose",
"moccasin",
"navajowhite",
"navy",
"oldlace",
"olive",
"olivedrab",
"orange",
"orangered",
"orchid",
"palegoldenrod",
"palegreen",
"paleturquoise",
"palevioletred",
"papayawhip",
"peachpuff",
"peru",
"pink",
"plum",
"powderblue",
"purple",
"rebeccapurple",
"red",
"rosybrown",
"royalblue",
"saddlebrown",
"salmon",
"sandybrown",
"seagreen",
"seashell",
"sienna",
"silver",
"skyblue",
"slateblue",
"slategray",
"snow",
"springgreen",
"steelblue",
"tan",
"teal",
"thistle",
"tomato",
"turquoise",
"violet",
"wheat",
"white",
"whitesmoke",
"yellow",
"yellowgreen",
]
def validate_color(color: str) -> bool:
match_hex = re.search(r"^#(?:[0-9a-f]{3}){1,2}$", color.lower())
match_html_color_name = color.lower() in HTML_COLORS
return bool(match_hex or match_html_color_name)

View File

@@ -1,4 +1,4 @@
from dbt.contracts.util import Replaceable, Mergeable, list_str
from dbt.contracts.util import Replaceable, Mergeable, list_str, Identifier
from dbt.contracts.connection import QueryComment, UserConfigContract
from dbt.helper_types import NoValue
from dbt.dataclass_schema import (
@@ -7,7 +7,6 @@ from dbt.dataclass_schema import (
HyphenatedDbtClassMixin,
ExtensibleDbtClassMixin,
register_pattern,
ValidatedStringMixin,
)
from dataclasses import dataclass, field
from typing import Optional, List, Dict, Union, Any
@@ -19,25 +18,6 @@ PIN_PACKAGE_URL = (
DEFAULT_SEND_ANONYMOUS_USAGE_STATS = True
class Name(ValidatedStringMixin):
ValidationRegex = r"^[^\d\W]\w*$"
@classmethod
def is_valid(cls, value: Any) -> bool:
if not isinstance(value, str):
return False
try:
cls.validate(value)
except ValidationError:
return False
return True
register_pattern(Name, r"^[^\d\W]\w*$")
class SemverString(str, SerializableType):
def _serialize(self) -> str:
return self
@@ -182,7 +162,7 @@ BANNED_PROJECT_NAMES = {
@dataclass
class Project(HyphenatedDbtClassMixin, Replaceable):
name: Name
name: Identifier
version: Union[SemverString, float]
config_version: int
project_root: Optional[str] = None

View File

@@ -9,6 +9,13 @@ from dbt.version import __version__
from dbt.events.functions import get_invocation_id
from dbt.dataclass_schema import dbtClassMixin
from dbt.dataclass_schema import (
ValidatedStringMixin,
ValidationError,
register_pattern,
)
SourceKey = Tuple[str, str]
@@ -250,3 +257,22 @@ class ArtifactMixin(VersionedSchema, Writable, Readable):
super().validate(data)
if cls.dbt_schema_version is None:
raise InternalException("Cannot call from_dict with no schema version!")
class Identifier(ValidatedStringMixin):
ValidationRegex = r"^[^\d\W]\w*$"
@classmethod
def is_valid(cls, value: Any) -> bool:
if not isinstance(value, str):
return False
try:
cls.validate(value)
except ValidationError:
return False
return True
register_pattern(Identifier, r"^[^\d\W]\w*$")

View File

@@ -520,6 +520,12 @@ def ref_invalid_args(model, args) -> NoReturn:
raise_compiler_error("ref() takes at most two arguments ({} given)".format(len(args)), model)
def metric_invalid_args(model, args) -> NoReturn:
raise_compiler_error(
"metric() takes at most two arguments ({} given)".format(len(args)), model
)
def ref_bad_context(model, args) -> NoReturn:
ref_args = ", ".join("'{}'".format(a) for a in args)
ref_string = "{{{{ ref({}) }}}}".format(ref_args)
@@ -650,6 +656,23 @@ def source_target_not_found(
raise_compiler_error(msg, model)
def get_metric_not_found_msg(
model,
target_name: str,
target_package: Optional[str],
) -> str:
reason = "was not found"
return _get_target_failure_msg(
model, target_name, target_package, include_path=True, reason=reason, target_kind="metric"
)
def metric_target_not_found(metric, target_name: str, target_package: Optional[str]) -> NoReturn:
msg = get_metric_not_found_msg(metric, target_name, target_package)
raise_compiler_error(msg, metric)
def dependency_not_found(model, target_model_name):
raise_compiler_error(
"'{}' depends on '{}' which is not in the graph!".format(

View File

@@ -37,6 +37,8 @@ EVENT_BUFFER_SIZE = 100000
QUIET = None
NO_PRINT = None
CACHE_SELECTED_ONLY = None
TARGET_PATH = None
LOG_PATH = None
_NON_BOOLEAN_FLAGS = [
"LOG_FORMAT",
@@ -44,6 +46,8 @@ _NON_BOOLEAN_FLAGS = [
"PROFILES_DIR",
"INDIRECT_SELECTION",
"EVENT_BUFFER_SIZE",
"TARGET_PATH",
"LOG_PATH",
]
_NON_DBT_ENV_FLAGS = ["DO_NOT_TRACK"]
@@ -71,6 +75,8 @@ flag_defaults = {
"QUIET": False,
"NO_PRINT": False,
"CACHE_SELECTED_ONLY": False,
"TARGET_PATH": None,
"LOG_PATH": None,
}
@@ -121,6 +127,7 @@ def set_from_args(args, user_config):
global WRITE_JSON, PARTIAL_PARSE, USE_COLORS, STORE_FAILURES, PROFILES_DIR, DEBUG, LOG_FORMAT
global INDIRECT_SELECTION, VERSION_CHECK, FAIL_FAST, SEND_ANONYMOUS_USAGE_STATS
global PRINTER_WIDTH, WHICH, LOG_CACHE_EVENTS, EVENT_BUFFER_SIZE, QUIET, NO_PRINT, CACHE_SELECTED_ONLY
global TARGET_PATH, LOG_PATH
STRICT_MODE = False # backwards compatibility
# cli args without user_config or env var option
@@ -148,6 +155,8 @@ def set_from_args(args, user_config):
QUIET = get_flag_value("QUIET", args, user_config)
NO_PRINT = get_flag_value("NO_PRINT", args, user_config)
CACHE_SELECTED_ONLY = get_flag_value("CACHE_SELECTED_ONLY", args, user_config)
TARGET_PATH = get_flag_value("TARGET_PATH", args, user_config)
LOG_PATH = get_flag_value("LOG_PATH", args, user_config)
_set_overrides_from_env()

View File

@@ -0,0 +1,167 @@
{# ------- BOOLEAN MACROS --------- #}
{#
-- COPY GRANTS
-- When a relational object (view or table) is replaced in this database,
-- do previous grants carry over to the new object? This may depend on:
-- whether we use alter-rename-swap versus CREATE OR REPLACE
-- user-supplied configuration (e.g. copy_grants on Snowflake)
-- By default, play it safe, assume TRUE: that grants ARE copied over.
-- This means dbt will first "show" current grants and then calculate diffs.
-- It may require an additional query than is strictly necessary,
-- but better safe than sorry.
#}
{% macro copy_grants() %}
{{ return(adapter.dispatch('copy_grants', 'dbt')()) }}
{% endmacro %}
{% macro default__copy_grants() %}
{{ return(True) }}
{% endmacro %}
{#
-- SUPPORT MULTIPLE GRANTEES PER DCL STATEMENT
-- Does this database support 'grant {privilege} to {grantee_1}, {grantee_2}, ...'
-- Or must these be separate statements:
-- `grant {privilege} to {grantee_1}`;
-- `grant {privilege} to {grantee_2}`;
-- By default, pick the former, because it's what we prefer when available.
#}
{% macro support_multiple_grantees_per_dcl_statement() %}
{{ return(adapter.dispatch('support_multiple_grantees_per_dcl_statement', 'dbt')()) }}
{% endmacro %}
{%- macro default__support_multiple_grantees_per_dcl_statement() -%}
{{ return(True) }}
{%- endmacro -%}
{% macro should_revoke(existing_relation, full_refresh_mode=True) %}
{% if not existing_relation %}
{#-- The table doesn't already exist, so no grants to copy over --#}
{{ return(False) }}
{% elif full_refresh_mode %}
{#-- The object is being REPLACED -- whether grants are copied over depends on the value of user config --#}
{{ return(copy_grants()) }}
{% else %}
{#-- The table is being merged/upserted/inserted -- grants will be carried over --#}
{{ return(True) }}
{% endif %}
{% endmacro %}
{# ------- DCL STATEMENT TEMPLATES --------- #}
{% macro get_show_grant_sql(relation) %}
{{ return(adapter.dispatch("get_show_grant_sql", "dbt")(relation)) }}
{% endmacro %}
{% macro default__get_show_grant_sql(relation) %}
show grants on {{ relation }}
{% endmacro %}
{% macro get_grant_sql(relation, privilege, grantees) %}
{{ return(adapter.dispatch('get_grant_sql', 'dbt')(relation, privilege, grantees)) }}
{% endmacro %}
{%- macro default__get_grant_sql(relation, privilege, grantees) -%}
grant {{ privilege }} on {{ relation }} to {{ grantees | join(', ') }}
{%- endmacro -%}
{% macro get_revoke_sql(relation, privilege, grantees) %}
{{ return(adapter.dispatch('get_revoke_sql', 'dbt')(relation, privilege, grantees)) }}
{% endmacro %}
{%- macro default__get_revoke_sql(relation, privilege, grantees) -%}
revoke {{ privilege }} on {{ relation }} from {{ grantees | join(', ') }}
{%- endmacro -%}
{# ------- RUNTIME APPLICATION --------- #}
{% macro get_dcl_statement_list(relation, grant_config, get_dcl_macro) %}
{{ return(adapter.dispatch('get_dcl_statement_list', 'dbt')(relation, grant_config, get_dcl_macro)) }}
{% endmacro %}
{%- macro default__get_dcl_statement_list(relation, grant_config, get_dcl_macro) -%}
{#
-- Unpack grant_config into specific privileges and the set of users who need them granted/revoked.
-- Depending on whether this database supports multiple grantees per statement, pass in the list of
-- all grantees per privilege, or (if not) template one statement per privilege-grantee pair.
-- `get_dcl_macro` will be either `get_grant_sql` or `get_revoke_sql`
#}
{%- set dcl_statements = [] -%}
{%- for privilege, grantees in grant_config.items() %}
{%- if support_multiple_grantees_per_dcl_statement() and grantees -%}
{%- set dcl = get_dcl_macro(relation, privilege, grantees) -%}
{%- do dcl_statements.append(dcl) -%}
{%- else -%}
{%- for grantee in grantees -%}
{% set dcl = get_dcl_macro(relation, privilege, [grantee]) %}
{%- do dcl_statements.append(dcl) -%}
{% endfor -%}
{%- endif -%}
{%- endfor -%}
{{ return(dcl_statements) }}
{%- endmacro %}
{% macro call_dcl_statements(dcl_statement_list) %}
{{ return(adapter.dispatch("call_dcl_statements", "dbt")(dcl_statement_list)) }}
{% endmacro %}
{% macro default__call_dcl_statements(dcl_statement_list) %}
{#
-- By default, supply all grant + revoke statements in a single semicolon-separated block,
-- so that they're all processed together.
-- Some databases do not support this. Those adapters will need to override this macro
-- to run each statement individually.
#}
{% call statement('grants') %}
{% for dcl_statement in dcl_statement_list %}
{{ dcl_statement }};
{% endfor %}
{% endcall %}
{% endmacro %}
{% macro apply_grants(relation, grant_config, should_revoke) %}
{{ return(adapter.dispatch("apply_grants", "dbt")(relation, grant_config, should_revoke)) }}
{% endmacro %}
{% macro default__apply_grants(relation, grant_config, should_revoke=True) %}
{#-- If grant_config is {} or None, this is a no-op --#}
{% if grant_config %}
{% if should_revoke %}
{#-- We think previous grants may have carried over --#}
{#-- Show current grants and calculate diffs --#}
{% set current_grants_table = run_query(get_show_grant_sql(relation)) %}
{% set current_grants_dict = adapter.standardize_grants_dict(current_grants_table) %}
{% set needs_granting = diff_of_two_dicts(grant_config, current_grants_dict) %}
{% set needs_revoking = diff_of_two_dicts(current_grants_dict, grant_config) %}
{% if not (needs_granting or needs_revoking) %}
{{ log('On ' ~ relation ~': All grants are in place, no revocation or granting needed.')}}
{% endif %}
{% else %}
{#-- We don't think there's any chance of previous grants having carried over. --#}
{#-- Jump straight to granting what the user has configured. --#}
{% set needs_revoking = {} %}
{% set needs_granting = grant_config %}
{% endif %}
{% if needs_granting or needs_revoking %}
{% set revoke_statement_list = get_dcl_statement_list(relation, needs_revoking, get_revoke_sql) %}
{% set grant_statement_list = get_dcl_statement_list(relation, needs_granting, get_grant_sql) %}
{% set dcl_statement_list = revoke_statement_list + grant_statement_list %}
{% if dcl_statement_list %}
{{ call_dcl_statements(dcl_statement_list) }}
{% endif %}
{% endif %}
{% endif %}
{% endmacro %}

View File

@@ -20,6 +20,8 @@
-- BEGIN, in a separate transaction
{%- set preexisting_intermediate_relation = load_cached_relation(intermediate_relation)-%}
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
-- grab current tables grants config for comparision later on
{% set grant_config = config.get('grants') %}
{{ drop_relation_if_exists(preexisting_intermediate_relation) }}
{{ drop_relation_if_exists(preexisting_backup_relation) }}
@@ -59,6 +61,9 @@
{% do to_drop.append(backup_relation) %}
{% endif %}
{% set should_revoke = should_revoke(existing_relation, full_refresh_mode) %}
{% do apply_grants(target_relation, grant_config, should_revoke=should_revoke) %}
{% do persist_docs(target_relation, model) %}
{% if existing_relation is none or existing_relation.is_view or should_full_refresh() %}

View File

@@ -14,6 +14,8 @@
{%- set backup_relation = make_backup_relation(target_relation, backup_relation_type) -%}
-- as above, the backup_relation should not already exist
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
-- grab current tables grants config for comparision later on
{% set grant_config = config.get('grants') %}
-- drop the temp relations if they exist already in the database
{{ drop_relation_if_exists(preexisting_intermediate_relation) }}
@@ -40,6 +42,9 @@
{{ run_hooks(post_hooks, inside_transaction=True) }}
{% set should_revoke = should_revoke(existing_relation, full_refresh_mode=True) %}
{% do apply_grants(target_relation, grant_config, should_revoke=should_revoke) %}
{% do persist_docs(target_relation, model) %}
-- `COMMIT` happens here

View File

@@ -13,12 +13,12 @@
{%- set identifier = model['alias'] -%}
{%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}
{%- set exists_as_view = (old_relation is not none and old_relation.is_view) -%}
{%- set target_relation = api.Relation.create(
identifier=identifier, schema=schema, database=database,
type='view') -%}
{% set grant_config = config.get('grants') %}
{{ run_hooks(pre_hooks) }}
@@ -34,6 +34,9 @@
{{ get_create_view_as_sql(target_relation, sql) }}
{%- endcall %}
{% set should_revoke = should_revoke(exists_as_view, full_refresh_mode=True) %}
{% do apply_grants(target_relation, grant_config, should_revoke=True) %}
{{ run_hooks(post_hooks) }}
{{ return({'relations': [target_relation]}) }}

View File

@@ -25,6 +25,8 @@
{%- set backup_relation = make_backup_relation(target_relation, backup_relation_type) -%}
-- as above, the backup_relation should not already exist
{%- set preexisting_backup_relation = load_cached_relation(backup_relation) -%}
-- grab current tables grants config for comparision later on
{% set grant_config = config.get('grants') %}
{{ run_hooks(pre_hooks, inside_transaction=False) }}
@@ -47,6 +49,9 @@
{% endif %}
{{ adapter.rename_relation(intermediate_relation, target_relation) }}
{% set should_revoke = should_revoke(existing_relation, full_refresh_mode=True) %}
{% do apply_grants(target_relation, grant_config, should_revoke=should_revoke) %}
{% do persist_docs(target_relation, model) %}
{{ run_hooks(post_hooks, inside_transaction=True) }}

View File

@@ -8,7 +8,10 @@
{%- set exists_as_table = (old_relation is not none and old_relation.is_table) -%}
{%- set exists_as_view = (old_relation is not none and old_relation.is_view) -%}
{%- set grant_config = config.get('grants') -%}
{%- set agate_table = load_agate_table() -%}
-- grab current tables grants config for comparision later on
{%- do store_result('agate_table', response='OK', agate_table=agate_table) -%}
{{ run_hooks(pre_hooks, inside_transaction=False) }}
@@ -35,6 +38,10 @@
{% endcall %}
{% set target_relation = this.incorporate(type='table') %}
{% set should_revoke = should_revoke(old_relation, full_refresh_mode) %}
{% do apply_grants(target_relation, grant_config, should_revoke=should_revoke) %}
{% do persist_docs(target_relation, model) %}
{% if full_refresh_mode or not exists_as_table %}

View File

@@ -5,6 +5,8 @@
{%- set strategy_name = config.get('strategy') -%}
{%- set unique_key = config.get('unique_key') %}
-- grab current tables grants config for comparision later on
{%- set grant_config = config.get('grants') -%}
{% set target_relation_exists, target_relation = get_or_create_relation(
database=model.database,
@@ -73,6 +75,9 @@
{{ final_sql }}
{% endcall %}
{% set should_revoke = should_revoke(target_relation_exists, full_refresh_mode=False) %}
{% do apply_grants(target_relation, grant_config, should_revoke=should_revoke) %}
{% do persist_docs(target_relation, model) %}
{% if not target_relation_exists %}

File diff suppressed because one or more lines are too long

View File

@@ -142,6 +142,7 @@ def main(args=None):
exit_code = e.code
except BaseException as e:
traceback.print_exc()
fire_event(MainEncounteredError(e=str(e)))
fire_event(MainStackTrace(stack_trace=traceback.format_exc()))
exit_code = ExitCodes.UnhandledError.value
@@ -651,6 +652,22 @@ def _add_common_arguments(*subparsers):
settings in profiles.yml.
""",
)
sub.add_argument(
"--target-path",
required=False,
help="""
Configure the 'target-path'. Only applies this setting for the
current run. Overrides the 'DBT_TARGET_PATH' if it is set.
""",
)
sub.add_argument(
"--log-path",
required=False,
help="""
Configure the 'log-path'. Only applies this setting for the
current run. Overrides the 'DBT_LOG_PATH' if it is set.
""",
)
_add_version_check(sub)

View File

@@ -17,7 +17,7 @@ from dbt.config import Project, RuntimeConfig
from dbt.context.context_config import ContextConfig
from dbt.contracts.graph.manifest import Manifest
from dbt.contracts.graph.parsed import HasUniqueID, ManifestNodes
from dbt.contracts.graph.unparsed import UnparsedNode
from dbt.contracts.graph.unparsed import UnparsedNode, Docs
from dbt.exceptions import ParsingException, validator_error_message, InternalException
from dbt import hooks
from dbt.node_types import NodeType
@@ -287,6 +287,11 @@ class ConfiguredParser(
if "meta" in config_dict and config_dict["meta"]:
parsed_node.meta = config_dict["meta"]
# If we have docs in the config, copy to node level, for backwards
# compatibility with earlier node-only config.
if "docs" in config_dict and config_dict["docs"]:
parsed_node.docs = Docs(config_dict["docs"])
# unrendered_config is used to compare the original database/schema/alias
# values and to handle 'same_config' and 'same_contents' calls
parsed_node.unrendered_config = config.build_config_dict(rendered=False)

View File

@@ -73,6 +73,7 @@ from dbt.exceptions import (
ref_target_not_found,
get_target_not_found_or_disabled_msg,
source_target_not_found,
metric_target_not_found,
get_source_not_found_or_disabled_msg,
warn_or_error,
)
@@ -389,6 +390,7 @@ class ManifestLoader:
self.process_sources(self.root_project.project_name)
self.process_refs(self.root_project.project_name)
self.process_docs(self.root_project)
self.process_metrics(self.root_project)
# update tracking data
self._perf_info.process_manifest_elapsed = time.perf_counter() - start_process
@@ -634,7 +636,9 @@ class ManifestLoader:
if not flags.PARTIAL_PARSE:
fire_event(PartialParsingNotEnabled())
return None
path = os.path.join(self.root_project.target_path, PARTIAL_PARSE_FILE_NAME)
path = os.path.join(
self.root_project.project_root, self.root_project.target_path, PARTIAL_PARSE_FILE_NAME
)
reparse_reason = None
@@ -833,6 +837,21 @@ class ManifestLoader:
continue
_process_refs_for_metric(self.manifest, current_project, metric)
# Takes references in 'metrics' array of nodes and exposures, finds the target
# node, and updates 'depends_on.nodes' with the unique id
def process_metrics(self, config: RuntimeConfig):
current_project = config.project_name
for node in self.manifest.nodes.values():
if node.created_at < self.started_at:
continue
_process_metrics_for_node(self.manifest, current_project, node)
for metric in self.manifest.metrics.values():
# TODO: Can we do this if the metric is derived & depends on
# some other metric for its definition? Maybe....
if metric.created_at < self.started_at:
continue
_process_metrics_for_node(self.manifest, current_project, metric)
# nodes: node and column descriptions
# sources: source and table descriptions, column descriptions
# macros: macro argument descriptions
@@ -936,6 +955,19 @@ def invalid_source_fail_unless_test(node, target_name, target_table_name, disabl
source_target_not_found(node, target_name, target_table_name, disabled=disabled)
def invalid_metric_fail_unless_test(node, target_metric_name, target_metric_package):
if node.resource_type == NodeType.Test:
msg = get_target_not_found_or_disabled_msg(node, target_metric_name, target_metric_package)
warn_or_error(msg, log_fmt=warning_tag("{}"))
else:
metric_target_not_found(
node,
target_metric_name,
target_metric_package,
)
def _check_resource_uniqueness(
manifest: Manifest,
config: RuntimeConfig,
@@ -1039,6 +1071,11 @@ def _process_docs_for_metrics(context: Dict[str, Any], metric: ParsedMetric) ->
metric.description = get_rendered(metric.description, context)
# TODO: this isn't actually referenced anywhere?
def _process_derived_metrics(context: Dict[str, Any], metric: ParsedMetric) -> None:
metric.description = get_rendered(metric.description, context)
def _process_refs_for_exposure(manifest: Manifest, current_project: str, exposure: ParsedExposure):
"""Given a manifest and exposure in that manifest, process its refs"""
for ref in exposure.refs:
@@ -1121,6 +1158,48 @@ def _process_refs_for_metric(manifest: Manifest, current_project: str, metric: P
manifest.update_metric(metric)
def _process_metrics_for_node(
manifest: Manifest, current_project: str, node: Union[ManifestNode, ParsedMetric]
):
"""Given a manifest and a node in that manifest, process its metrics"""
for metric in node.metrics:
target_metric: Optional[ParsedMetric] = None
target_metric_name: str
target_metric_package: Optional[str] = None
if len(metric) == 1:
target_metric_name = metric[0]
elif len(metric) == 2:
target_metric_package, target_metric_name = metric
else:
raise dbt.exceptions.InternalException(
f"Metric references should always be 1 or 2 arguments - got {len(metric)}"
)
# Resolve_ref
target_metric = manifest.resolve_metric(
target_metric_name,
target_metric_package,
current_project,
node.package_name,
)
if target_metric is None:
# This may raise. Even if it doesn't, we don't want to add
# this node to the graph b/c there is no destination node
invalid_metric_fail_unless_test(
node,
target_metric_name,
target_metric_package,
)
continue
target_metric_id = target_metric.unique_id
node.depends_on.nodes.append(target_metric_id)
def _process_refs_for_node(manifest: Manifest, current_project: str, node: ManifestNode):
"""Given a manifest and a node in that manifest, process its refs"""
for ref in node.refs:

View File

@@ -66,6 +66,9 @@ class SchemaYamlRenderer(BaseRenderer):
return False
elif self._is_norender_key(keypath[0:]):
return False
elif self.key == "metrics":
if keypath[0] == "sql":
return False
else: # models, seeds, snapshots, analyses
if self._is_norender_key(keypath[0:]):
return False

View File

@@ -797,6 +797,7 @@ class NonSourceParser(YamlDocsReader, Generic[NonSourceTarget, Parsed]):
if self.key != "macros":
# macros don't have the 'config' key support yet
self.normalize_meta_attribute(data, path)
self.normalize_docs_attribute(data, path)
node = self._target_type().from_dict(data)
except (ValidationError, JSONValidationException) as exc:
msg = error_context(path, self.key, data, exc)
@@ -804,21 +805,27 @@ class NonSourceParser(YamlDocsReader, Generic[NonSourceTarget, Parsed]):
else:
yield node
# We want to raise an error if 'meta' is in two places, and move 'meta'
# We want to raise an error if some attributes are in two places, and move them
# from toplevel to config if necessary
def normalize_meta_attribute(self, data, path):
if "meta" in data:
if "config" in data and "meta" in data["config"]:
def normalize_attribute(self, data, path, attribute):
if attribute in data:
if "config" in data and attribute in data["config"]:
raise ParsingException(
f"""
In {path}: found meta dictionary in 'config' dictionary and as top-level key.
In {path}: found {attribute} dictionary in 'config' dictionary and as top-level key.
Remove the top-level key and define it under 'config' dictionary only.
""".strip()
)
else:
if "config" not in data:
data["config"] = {}
data["config"]["meta"] = data.pop("meta")
data["config"][attribute] = data.pop(attribute)
def normalize_meta_attribute(self, data, path):
return self.normalize_attribute(data, path, "meta")
def normalize_docs_attribute(self, data, path):
return self.normalize_attribute(data, path, "docs")
def patch_node_config(self, node, patch):
# Get the ContextConfig that's used in calculating the config
@@ -1029,7 +1036,7 @@ class MetricParser(YamlReader):
description=unparsed.description,
label=unparsed.label,
type=unparsed.type,
sql=unparsed.sql,
sql=str(unparsed.sql),
timestamp=unparsed.timestamp,
dimensions=unparsed.dimensions,
time_grains=unparsed.time_grains,
@@ -1044,8 +1051,17 @@ class MetricParser(YamlReader):
self.schema_parser.manifest,
package_name,
)
model_ref = "{{ " + unparsed.model + " }}"
get_rendered(model_ref, ctx, parsed, capture_macros=True)
if parsed.model is not None:
model_ref = "{{ " + parsed.model + " }}"
get_rendered(model_ref, ctx, parsed)
parsed.sql = get_rendered(
parsed.sql,
ctx,
node=parsed,
)
return parsed
def parse(self) -> Iterable[ParsedMetric]:

View File

@@ -14,7 +14,7 @@ from dbt import flags
from dbt.version import _get_adapter_plugin_names
from dbt.adapters.factory import load_plugin, get_include_paths
from dbt.contracts.project import Name as ProjectName
from dbt.contracts.util import Identifier as ProjectName
from dbt.events.functions import fire_event
from dbt.events.types import (

View File

@@ -73,7 +73,10 @@ def run_dbt(args: List[str] = None, expect_pass=True):
return res
# Use this if you need to capture the command logs in a test
# Use this if you need to capture the command logs in a test.
# If you want the logs that are normally written to a file, you must
# start with the "--debug" flag. The structured schema log CI test
# will turn the logs into json, so you have to be prepared for that.
def run_dbt_and_capture(args: List[str] = None, expect_pass=True):
try:
stringbuf = capture_stdout_logs()
@@ -83,6 +86,11 @@ def run_dbt_and_capture(args: List[str] = None, expect_pass=True):
finally:
stop_capture_stdout_logs()
# Json logs will have lots of escape characters which will
# make checks for strings in the logs fail, so remove those.
if '{"code":' in stdout:
stdout = stdout.replace("\\", "")
return res, stdout

View File

@@ -235,5 +235,5 @@ def _get_adapter_plugin_names() -> Iterator[str]:
yield plugin_name
__version__ = "1.2.0b1"
__version__ = "1.3.0a1"
installed = get_installed_version()

View File

@@ -25,7 +25,7 @@ with open(os.path.join(this_directory, "README.md")) as f:
package_name = "dbt-core"
package_version = "1.2.0b1"
package_version = "1.3.0a1"
description = """With dbt, data analysts and engineers can build analytics \
the way engineers build applications."""

View File

@@ -14,8 +14,8 @@ FROM --platform=$build_for python:3.10.5-slim-bullseye as base
# N.B. The refs updated automagically every release via bumpversion
# N.B. dbt-postgres is currently found in the core codebase so a value of dbt-core@<some_version> is correct
ARG dbt_core_ref=dbt-core@v1.2.0b1
ARG dbt_postgres_ref=dbt-core@v1.2.0b1
ARG dbt_core_ref=dbt-core@v1.3.0a1
ARG dbt_postgres_ref=dbt-core@v1.3.0a1
ARG dbt_redshift_ref=dbt-redshift@v1.0.0
ARG dbt_bigquery_ref=dbt-bigquery@v1.0.0
ARG dbt_snowflake_ref=dbt-snowflake@v1.0.0

Some files were not shown because too many files have changed in this diff Show More