Compare commits

..

22 Commits

Author SHA1 Message Date
github-actions[bot]
d21f2cd090 Bumping version to 1.2.3 and generate changelog (#6262)
* Bumping version to 1.2.3rc1 and generate CHANGELOG

* Updating version to remove rc1

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-11-16 09:47:22 -05:00
github-actions[bot]
d6b5147566 Updates lib to use new profile name functionality (#6202) (#6220)
* Updates lib to use new profile name functionality

* Adds changie entry

* Fixes formatting

(cherry picked from commit d0543c9242)

Co-authored-by: Rachel <41338402+racheldaniel@users.noreply.github.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-11-15 22:09:55 -05:00
github-actions[bot]
e4cbc9790b s/gitlab/github for flake8 precommit repo (#6252) (#6255)
(cherry picked from commit eae98677b9)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2022-11-15 11:11:52 -05:00
Emily Rockman
b25a09d374 add python version and upgrade action (#6204) (#6210)
(cherry picked from commit c3ccbe3357)
2022-11-03 12:44:12 -05:00
github-actions[bot]
111c639f03 more ergonomic profile name handling (#6157) (#6158) 2022-11-02 13:35:51 -05:00
github-actions[bot]
abee0f0dc1 Bumping version to 1.2.2 and generate changelog (#5996)
* Bumping version to 1.2.2rc1 and generate CHANGELOG

* Bumping version to 1.2.2 and generate CHANGELOG (#5997)

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-10-03 14:16:51 -04:00
leahwicz
f903aed3e0 Update Version Bump to include Homebrew in PATH (#5963) (#5993) 2022-10-03 13:37:56 -04:00
github-actions[bot]
1aee7b3700 Runtime: Prevent introspective queries at compile (SL only) (#5926) (#5943)
* Preliminary changes to keep compile from connecting to the warehouse for runtime calls

* Adds option to lib to skip connecting to warehouse for compile; adds prelim tests

* Removes unused imports

* Simplifies test and renames to SqlCompileRunnerNoIntrospection

* Updates name in tests

* Spacing

* Updates test to check for adapter connection call instead of compile and execute

* Removes commented line

* Fixes test names

* Updates plugin to postgres type as snowflake isn't available

* Fixes docstring

* Fixes formatting

* Moves conditional logic out of class

* Fixes formatting

* Removes commented line

* Moves import

* Unmoves import

* Updates changelog

* Adds further info to method docstring

(cherry picked from commit f1326f526c)

Co-authored-by: Rachel <41338402+racheldaniel@users.noreply.github.com>
2022-09-29 15:47:22 +02:00
github-actions[bot]
6278880fa0 Fix adapter reset race condition in lib.py (#5921) (#5931)
* (#5919) Fix adapter reset race condition in lib.py

* run black

* changie

(cherry picked from commit 4e8aa007cf)

Co-authored-by: Drew Banin <drew@dbtlabs.com>
2022-09-27 14:28:54 -04:00
github-actions[bot]
07415ca3e7 Bumping version to 1.2.1 and generate changelog (#5713)
* Bumping version to 1.2.1 and generate CHANGELOG

* Fix Docker file version

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-08-25 13:03:30 -04:00
github-actions[bot]
1d6eb7dcd5 Bumping version to 1.2.1rc2 and generate changelog (#5682)
* Bumping version to 1.2.1rc2 and generate CHANGELOG

* Remove newline

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-08-18 13:02:13 -04:00
github-actions[bot]
7955fe7c3b Change postgres name truncation logic to be overridable. (#5656) (#5681)
* Change postgres name truncation logic to be overridable. Add exception with debugging instructions.

* Add changelog.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
(cherry picked from commit 0d02446e07)

Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com>
2022-08-18 12:10:58 -04:00
github-actions[bot]
f82cddbefe Bumping version to 1.2.1rc1 and generate CHANGELOG (#5641)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-08-11 13:02:45 +02:00
Jeremy Cohen
39901c5155 Use sys.exit instead of exit (#5627) (#5640)
* Use sys.exit instead of exit

* Add changelog

Co-authored-by: varun-dc <91232924+varun-dc@users.noreply.github.com>
2022-08-10 20:49:17 +02:00
github-actions[bot]
6c21d10469 Do not render metrics description field when doing render_data (#5603) (#5619)
(cherry picked from commit eb72dbf32a)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-08-05 15:54:19 -04:00
github-actions[bot]
9a10de926e Add changelog and whitespace fix to version bump Action (#5563) (#5572)
* Add changelog and whitespace fix to version bump Action

* Fixing whitespace

* Remove tabs

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Updating per comments

* Fix whitespace

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
(cherry picked from commit 2547e4f55e)

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-07-29 14:01:32 -04:00
github-actions[bot]
9c32441d05 Fix handling of top-level exceptions (#5560) (#5570)
(cherry picked from commit b43fc76701)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-07-28 16:15:37 -04:00
github-actions[bot]
c7652f6a33 Bumping version to 1.2.0 (#5557)
* Bumping version to 1.2.0

* Whitespace fix

* Changlog update

* Docker file version update

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-07-26 08:28:05 -04:00
github-actions[bot]
a2a531722e Bumping version to 1.2.0rc2 (#5497)
* Bumping version to 1.2.0rc2

* Removing whitespace

* Update Dockerfile

* Changelog update

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-07-20 09:01:13 -04:00
leahwicz
7656ffb9e6 [CT-472] feat: Retrying method for acquiring connection handles (#5432) (#5494)
Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift).

Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
2022-07-19 21:36:04 -04:00
Jeremy Yeo
3f3792882c Fix: Rename try methods to strict (#5477)
* rename strict methods

* add changelog
2022-07-15 08:17:01 -05:00
github-actions[bot]
0db634d12f Bumping version to 1.2.0rc1 (#5458)
* Bumping version to 1.2.0rc1

* Remove whitespace

* Update Changelog

* Update Dockerfile

* Removing dependabot as contributor

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Removing dependabot as contributor

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-07-11 16:58:51 -04:00
845 changed files with 29199 additions and 41485 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion] [bumpversion]
current_version = 1.5.0a1 current_version = 1.2.3
parse = (?P<major>\d+) parse = (?P<major>\d+)
\.(?P<minor>\d+) \.(?P<minor>\d+)
\.(?P<patch>\d+) \.(?P<patch>\d+)

View File

@@ -2,10 +2,6 @@
For information on prior major and minor releases, see their changelogs: For information on prior major and minor releases, see their changelogs:
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md) * [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)
* [1.0](https://github.com/dbt-labs/dbt-core/blob/1.0.latest/CHANGELOG.md) * [1.0](https://github.com/dbt-labs/dbt-core/blob/1.0.latest/CHANGELOG.md)
* [0.21](https://github.com/dbt-labs/dbt-core/blob/0.21.latest/CHANGELOG.md) * [0.21](https://github.com/dbt-labs/dbt-core/blob/0.21.latest/CHANGELOG.md)

103
.changes/1.2.0.md Normal file
View File

@@ -0,0 +1,103 @@
## dbt-core 1.2.0 - July 26, 2022
### Features
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
- Adds itertools to modules Jinja namespace ([#5130](https://github.com/dbt-labs/dbt-core/issues/5130), [#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- allow target as an option in profile_template.yml ([#5179](https://github.com/dbt-labs/dbt-core/issues/5179), [#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- seed: Add new macro get_csv_sql ([#5206](https://github.com/dbt-labs/dbt-core/issues/5206), [#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- Grants as Node Configs ([#5189](https://github.com/dbt-labs/dbt-core/issues/5189), [#5230](https://github.com/dbt-labs/dbt-core/pull/5230))
- Adds file selectors and support for file selectors in the default method selector ([#5240](https://github.com/dbt-labs/dbt-core/issues/5240), [#5241](https://github.com/dbt-labs/dbt-core/pull/5241))
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- Allow customizing `target-path` and `log-path` through environment variables and CLI flags. ([#5399](https://github.com/dbt-labs/dbt-core/issues/5399), [#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- Move type_* macros from dbt-utils into dbt-core, with tests ([#5317](https://github.com/dbt-labs/dbt-core/issues/5317), [#5428](https://github.com/dbt-labs/dbt-core/pull/5428))
- Add support for ratio metrics ([#4884](https://github.com/dbt-labs/dbt-core/issues/4884), [#5027](https://github.com/dbt-labs/dbt-core/pull/5027))
- Allow users to define grants as a reasonable default in the dbt_project.yml or within each model sql or yml file combined. ([#5263](https://github.com/dbt-labs/dbt-core/issues/5263), [#5369](https://github.com/dbt-labs/dbt-core/pull/5369))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
### Fixes
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- Use yaml renderer (with target context) for rendering selectors ([#5131](https://github.com/dbt-labs/dbt-core/issues/5131), [#5136](https://github.com/dbt-labs/dbt-core/pull/5136))
- Fix retry logic to return values after initial try ([#5023](https://github.com/dbt-labs/dbt-core/issues/5023), [#5137](https://github.com/dbt-labs/dbt-core/pull/5137))
- Scrub secret env vars from CommandError in exception stacktrace ([#5151](https://github.com/dbt-labs/dbt-core/issues/5151), [#5152](https://github.com/dbt-labs/dbt-core/pull/5152))
- Ensure the metric name does not contain spaces ([#4572](https://github.com/dbt-labs/dbt-core/issues/4572), [#5173](https://github.com/dbt-labs/dbt-core/pull/5173))
- When parsing 'all_sources' should be a list of unique dirs ([#5120](https://github.com/dbt-labs/dbt-core/issues/5120), [#5176](https://github.com/dbt-labs/dbt-core/pull/5176))
- Add warning if yaml contains duplicate keys ([#5114](https://github.com/dbt-labs/dbt-core/issues/5114), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146))
- Modifying the drop_test_schema to work better with Redshift issues around locked tables and current transactions ([#5200](https://github.com/dbt-labs/dbt-core/issues/5200), [#5198](https://github.com/dbt-labs/dbt-core/pull/5198))
- Fix column comparison in snapshot_check_all_get_existing_columns for check-strategy snapshots with explicit check_cols defined ([#5222](https://github.com/dbt-labs/dbt-core/issues/5222), [#5223](https://github.com/dbt-labs/dbt-core/pull/5223))
- Changed how `--select state:modified` detects changes for macros nodes depend on ([#5202](https://github.com/dbt-labs/dbt-core/issues/5202), [#5224](https://github.com/dbt-labs/dbt-core/pull/5224))
- Fix column comparison in snapshot_check_all_get_existing_columns to use adapter.get_columns_in_relation ([#5222](https://github.com/dbt-labs/dbt-core/issues/5222), [#5232](https://github.com/dbt-labs/dbt-core/pull/5232))
- Remove docs file from manifest when removing doc node ([#4146](https://github.com/dbt-labs/dbt-core/issues/4146), [#5270](https://github.com/dbt-labs/dbt-core/pull/5270))
- Remove duplicate dbt script entry ([#5314](https://github.com/dbt-labs/dbt-core/issues/5314), [#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
- Add inheritance to materialization macro resolution ([#4646](https://github.com/dbt-labs/dbt-core/issues/4646), [#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
- Improve pluralizations for Documentation and SqlOperation NodeTypes ([#5352](https://github.com/dbt-labs/dbt-core/issues/5352), [#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- Properly use quotes for Snowflake snapshots when checking all columns ([#2975](https://github.com/dbt-labs/dbt-core/issues/2975), [#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- fixes handling of RESET color code with USE_COLORS=False ([#5288](https://github.com/dbt-labs/dbt-core/issues/5288), [#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
### Docs
- Fixed sample SQL Code for sources when no database is defined ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Add support for `file:` selector in DAG viz ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [Snyk] Upgrade prismjs from 1.27.0 to 1.28.0 ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Run build and tests in CI checks ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Improve metrics DAG viz and documentation page ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Upgrade cytoscape.js fork ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
### Under the Hood
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
- Move package deprecation check outside of package cache ([#5068](https://github.com/dbt-labs/dbt-core/issues/5068), [#5069](https://github.com/dbt-labs/dbt-core/pull/5069))
- removal of scaffold first attempt and create_adapter_plugin.py as they are deprecated new scaffold can be found https://github.com/dbt-labs/dbt-database-adapter-scaffold ([#4980](https://github.com/dbt-labs/dbt-core/issues/4980), [#5117](https://github.com/dbt-labs/dbt-core/pull/5117))
- Mypy -> 0.942 + fixed import logic to allow for full mypy coverage ([#4805](https://github.com/dbt-labs/dbt-core/issues/4805), [#5171](https://github.com/dbt-labs/dbt-core/pull/5171))
- Converted dbt list tests to pytest ([#5049](https://github.com/dbt-labs/dbt-core/issues/5049), [#5178](https://github.com/dbt-labs/dbt-core/pull/5178))
- Fix: Call str and repr for UnsetProfileConfig without a RuntimeException ([#5081](https://github.com/dbt-labs/dbt-core/issues/5081), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- Improve tracking error logging message ([#5197](https://github.com/dbt-labs/dbt-core/issues/5197), [#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- Clean up materialization logic: more consistent relation names, loading from cache ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- Use the default Python version for local dev and test instead of requiring Python 3.8 ([#5257](https://github.com/dbt-labs/dbt-core/issues/5257), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- Fix test for context set function ([#5266](https://github.com/dbt-labs/dbt-core/issues/5266), [#5272](https://github.com/dbt-labs/dbt-core/pull/5272))
- Fix pip upgrade step in CI for Windows ([#5321](https://github.com/dbt-labs/dbt-core/issues/5321), [#5320](https://github.com/dbt-labs/dbt-core/pull/5320))
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
- Added the suggested RegEx to check the SemVer string within a package dependency and improved invalid version error handling. ([#5201](https://github.com/dbt-labs/dbt-core/issues/5201), [#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
- Bump manifest version to v6 ([#5417](https://github.com/dbt-labs/dbt-core/issues/5417), [#5430](https://github.com/dbt-labs/dbt-core/pull/5430))
- Add tests for SQL grants ([#5437](https://github.com/dbt-labs/dbt-core/issues/5437), [#5447](https://github.com/dbt-labs/dbt-core/pull/5447))
### Dependencies
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Bump mypy from 0.942 to 0.961 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5337](https://github.com/dbt-labs/dbt-core/pull/5337))
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5388](https://github.com/dbt-labs/dbt-core/pull/5388))
- Bump black from 22.3.0 to 22.6.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5420](https://github.com/dbt-labs/dbt-core/pull/5420))
### Security
- Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Contributors
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- [@b-per](https://github.com/b-per) ([#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- [@darin-reify](https://github.com/darin-reify) ([#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@drewbanin](https://github.com/drewbanin) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) ([#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- [@isidentical](https://github.com/isidentical) ([#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- [@pdebelak](https://github.com/pdebelak) ([#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- [@pquadri](https://github.com/pquadri) ([#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- [@volkangurel](https://github.com/volkangurel) ([#5348](https://github.com/dbt-labs/dbt-core/pull/5348))

9
.changes/1.2.1.md Normal file
View File

@@ -0,0 +1,9 @@
## dbt-core 1.2.1 - August 25, 2022
### Fixes
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
### Contributors
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))

9
.changes/1.2.2.md Normal file
View File

@@ -0,0 +1,9 @@
## dbt-core 1.2.2 - October 03, 2022
### Features
- This conditionally no-ops warehouse connection at compile depending on an env var, disabling introspection/queries during compilation only. This is a temporary solution to more complex permissions requirements for the semantic layer. ([#5936](https://github.com/dbt-labs/dbt-core/issues/5936), [#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
### Fixes
- Fix race condition when invoking dbt via lib.py concurrently ([#5919](https://github.com/dbt-labs/dbt-core/issues/5919), [#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
### Contributors
- [@drewbanin](https://github.com/drewbanin) ([#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
- [@racheldaniel](https://github.com/racheldaniel) ([#5926](https://github.com/dbt-labs/dbt-core/pull/5926))

6
.changes/1.2.3.md Normal file
View File

@@ -0,0 +1,6 @@
## dbt-core 1.2.3 - November 16, 2022
### Features
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201), [#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
### Contributors
- [@racheldaniel](https://github.com/racheldaniel) ([#6202](https://github.com/dbt-labs/dbt-core/pull/6202))

View File

@@ -1,6 +0,0 @@
kind: Features
body: Adding the entity node
time: 2023-01-18T13:48:04.487817-06:00
custom:
Author: callum-mcdata
Issue: "6627"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Fix use of ConnectionReused logging event
time: 2023-01-13T13:25:13.023168-05:00
custom:
Author: gshank
Issue: "6168"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Update deprecated github action command
time: 2023-01-17T11:17:37.046095-06:00
custom:
Author: davidbloss
Issue: "6153"

124
.changie.yaml Normal file → Executable file
View File

@@ -6,122 +6,56 @@ changelogPath: CHANGELOG.md
versionExt: md versionExt: md
versionFormat: '## dbt-core {{.Version}} - {{.Time.Format "January 02, 2006"}}' versionFormat: '## dbt-core {{.Version}} - {{.Time.Format "January 02, 2006"}}'
kindFormat: '### {{.Kind}}' kindFormat: '### {{.Kind}}'
changeFormat: |- changeFormat: '- {{.Body}} ([#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), [#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
kinds: kinds:
- label: Breaking Changes - label: Breaking Changes
- label: Features - label: Features
- label: Fixes - label: Fixes
- label: Docs - label: Docs
changeFormat: |- - label: Under the Hood
{{- $IssueList := list }} - label: Dependencies
{{- $changes := splitList " " $.Custom.Issue }} - label: Security
{{- range $issueNbr := $changes }}
{{- $changeLink := "[dbt-docs/#nbr](https://github.com/dbt-labs/dbt-docs/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
afterKind: 1
afterChangelogVersion: 1
beforeKind: 1
endOfVersion: 1
custom: custom:
- key: Author - key: Author
label: GitHub Username(s) (separated by a single space if multiple) label: GitHub Username(s) (separated by a single space if multiple)
type: string type: string
minLength: 3 minLength: 3
- key: Issue - key: Issue
label: GitHub Issue Number (separated by a single space if multiple) label: GitHub Issue Number
type: string type: int
minLength: 1 minLength: 4
- key: PR
label: GitHub Pull Request Number
type: int
minLength: 4
footerFormat: | footerFormat: |
{{- $contributorDict := dict }} {{- $contributorDict := dict }}
{{- /* any names added to this list should be all lowercase for later matching purposes */}} {{- /* any names added to this list should be all lowercase for later matching purposes */}}
{{- $core_team := list "michelleark" "peterallenwebb" "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "aranke" "dependabot[bot]" "snyk-bot" "colin-rogers-dbt" }} {{- $core_team := list "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "dependabot" }}
{{- range $change := .Changes }} {{- range $change := .Changes }}
{{- $authorList := splitList " " $change.Custom.Author }} {{- $authorList := splitList " " $change.Custom.Author }}
{{- /* loop through all authors for a single changelog */}} {{- /* loop through all authors for a PR */}}
{{- range $author := $authorList }} {{- range $author := $authorList }}
{{- $authorLower := lower $author }} {{- $authorLower := lower $author }}
{{- /* we only want to include non-core team contributors */}} {{- /* we only want to include non-core team contributors */}}
{{- if not (has $authorLower $core_team)}} {{- if not (has $authorLower $core_team)}}
{{- $changeList := splitList " " $change.Custom.Author }} {{- $pr := $change.Custom.PR }}
{{- /* Docs kind link back to dbt-docs instead of dbt-core issues */}} {{- /* check if this contributor has other PRs associated with them already */}}
{{- $changeLink := $change.Kind }} {{- if hasKey $contributorDict $author }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }} {{- $prList := get $contributorDict $author }}
{{- $changeLink = "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $change.Custom.PR }} {{- $prList = append $prList $pr }}
{{- else if eq $change.Kind "Docs"}} {{- $contributorDict := set $contributorDict $author $prList }}
{{- $changeLink = "[dbt-docs/#nbr](https://github.com/dbt-labs/dbt-docs/issues/nbr)" | replace "nbr" $change.Custom.Issue }} {{- else }}
{{- else }} {{- $prList := list $change.Custom.PR }}
{{- $changeLink = "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $change.Custom.Issue }} {{- $contributorDict := set $contributorDict $author $prList }}
{{- end }} {{- end }}
{{- /* check if this contributor has other changes associated with them already */}} {{- end}}
{{- if hasKey $contributorDict $author }}
{{- $contributionList := get $contributorDict $author }}
{{- $contributionList = append $contributionList $changeLink }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- else }}
{{- $contributionList := list $changeLink }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- end }}
{{- end}}
{{- end}} {{- end}}
{{- end }} {{- end }}
{{- /* no indentation here for formatting so the final markdown doesn't have unneeded indentations */}} {{- /* no indentation here for formatting so the final markdown doesn't have unneeded indentations */}}
{{- if $contributorDict}} {{- if $contributorDict}}
### Contributors ### Contributors
{{- range $k,$v := $contributorDict }} {{- range $k,$v := $contributorDict }}
- [@{{$k}}](https://github.com/{{$k}}) ({{ range $index, $element := $v }}{{if $index}}, {{end}}{{$element}}{{end}}) - [@{{$k}}](https://github.com/{{$k}}) ({{ range $index, $element := $v }}{{if $index}}, {{end}}[#{{$element}}](https://github.com/dbt-labs/dbt-core/pull/{{$element}}){{end}})
{{- end }} {{- end }}
{{- end }} {{- end }}

View File

@@ -9,4 +9,4 @@ ignore =
E203 # makes Flake8 work like black E203 # makes Flake8 work like black
E741 E741
E501 # long line checking is done in black E501 # long line checking is done in black
exclude = test/ exclude = test

2
.gitattributes vendored
View File

@@ -1,2 +0,0 @@
core/dbt/include/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary

60
.github/CODEOWNERS vendored
View File

@@ -16,57 +16,25 @@
# Changes to GitHub configurations including Actions # Changes to GitHub configurations including Actions
/.github/ @leahwicz /.github/ @leahwicz
### LANGUAGE
# Language core modules # Language core modules
/core/dbt/config/ @dbt-labs/core-language /core/dbt/config/ @dbt-labs/core-language
/core/dbt/context/ @dbt-labs/core-language /core/dbt/context/ @dbt-labs/core-language
/core/dbt/contracts/ @dbt-labs/core-language /core/dbt/contracts/ @dbt-labs/core-language
/core/dbt/deps/ @dbt-labs/core-language /core/dbt/deps/ @dbt-labs/core-language
/core/dbt/events/ @dbt-labs/core-language # structured logging /core/dbt/parser/ @dbt-labs/core-language
/core/dbt/parser/ @dbt-labs/core-language
# Language misc files
/core/dbt/dataclass_schema.py @dbt-labs/core-language
/core/dbt/hooks.py @dbt-labs/core-language
/core/dbt/node_types.py @dbt-labs/core-language
/core/dbt/semver.py @dbt-labs/core-language
### EXECUTION
# Execution core modules # Execution core modules
/core/dbt/graph/ @dbt-labs/core-execution /core/dbt/events/ @dbt-labs/core-execution @dbt-labs/core-language # eventually remove language but they have knowledge here now
/core/dbt/task/ @dbt-labs/core-execution /core/dbt/graph/ @dbt-labs/core-execution
/core/dbt/task/ @dbt-labs/core-execution
# Execution misc files # Adapter interface, scaffold, Postgres plugin
/core/dbt/compilation.py @dbt-labs/core-execution /core/dbt/adapters @dbt-labs/core-adapters
/core/dbt/flags.py @dbt-labs/core-execution /core/scripts/create_adapter_plugin.py @dbt-labs/core-adapters
/core/dbt/lib.py @dbt-labs/core-execution /plugins/ @dbt-labs/core-adapters
/core/dbt/main.py @dbt-labs/core-execution
/core/dbt/profiler.py @dbt-labs/core-execution
/core/dbt/selected_resources.py @dbt-labs/core-execution
/core/dbt/tracking.py @dbt-labs/core-execution
/core/dbt/version.py @dbt-labs/core-execution
# Global project: default macros, including generic tests + materializations
### ADAPTERS /core/dbt/include/global_project @dbt-labs/core-execution @dbt-labs/core-adapters
# Adapter interface ("base" + "sql" adapter defaults, cache)
/core/dbt/adapters @dbt-labs/core-adapters
# Global project (default macros + materializations), starter project
/core/dbt/include @dbt-labs/core-adapters
# Postgres plugin
/plugins/ @dbt-labs/core-adapters
# Functional tests for adapter plugins
/tests/adapter @dbt-labs/core-adapters
### TESTS
# Overlapping ownership for vast majority of unit + functional tests
# Perf regression testing framework # Perf regression testing framework
# This excludes the test project files itself since those aren't specific # This excludes the test project files itself since those aren't specific

View File

@@ -9,33 +9,23 @@ body:
Thanks for taking the time to fill out this bug report! Thanks for taking the time to fill out this bug report!
- type: checkboxes - type: checkboxes
attributes: attributes:
label: Is this a new bug in dbt-core? label: Is there an existing issue for this?
description: > description: Please search to see if an issue already exists for the bug you encountered.
In other words, is this an error, flaw, failure or fault in our software?
If this is a bug that broke existing functionality that used to work, please open a regression issue.
If this is a bug in an adapter plugin, please open an issue in the adapter's repository.
If this is a bug experienced while using dbt Cloud, please report to [support](mailto:support@getdbt.com).
If this is a request for help or troubleshooting code in your own dbt project, please join our [dbt Community Slack](https://www.getdbt.com/community/join-the-community/) or open a [Discussion question](https://github.com/dbt-labs/docs.getdbt.com/discussions).
Please search to see if an issue already exists for the bug you encountered.
options: options:
- label: I believe this is a new bug in dbt-core - label: I have searched the existing issues
required: true
- label: I have searched the existing issues, and I could not find an existing issue for this bug
required: true required: true
- type: textarea - type: textarea
attributes: attributes:
label: Current Behavior label: Current Behavior
description: A concise description of what you're experiencing. description: A concise description of what you're experiencing.
validations: validations:
required: true required: false
- type: textarea - type: textarea
attributes: attributes:
label: Expected Behavior label: Expected Behavior
description: A concise description of what you expected to happen. description: A concise description of what you expected to happen.
validations: validations:
required: true required: false
- type: textarea - type: textarea
attributes: attributes:
label: Steps To Reproduce label: Steps To Reproduce
@@ -46,7 +36,7 @@ body:
3. Run '...' 3. Run '...'
4. See error... 4. See error...
validations: validations:
required: true required: false
- type: textarea - type: textarea
id: logs id: logs
attributes: attributes:
@@ -62,8 +52,8 @@ body:
description: | description: |
examples: examples:
- **OS**: Ubuntu 20.04 - **OS**: Ubuntu 20.04
- **Python**: 3.9.12 (`python3 --version`) - **Python**: 3.7.2 (`python --version`)
- **dbt-core**: 1.1.1 (`dbt --version`) - **dbt**: 0.21.0 (`dbt --version`)
value: | value: |
- OS: - OS:
- Python: - Python:
@@ -74,15 +64,13 @@ body:
- type: dropdown - type: dropdown
id: database id: database
attributes: attributes:
label: Which database adapter are you using with dbt? label: What database are you using dbt with?
description: If the bug is specific to the database or adapter, please open the issue in that adapter's repository instead
multiple: true multiple: true
options: options:
- postgres - postgres
- redshift - redshift
- snowflake - snowflake
- bigquery - bigquery
- spark
- other (mention it in "Additional Context") - other (mention it in "Additional Context")
validations: validations:
required: false required: false

View File

@@ -1,14 +1,4 @@
blank_issues_enabled: false
contact_links: contact_links:
- name: Ask the community for help
url: https://github.com/dbt-labs/docs.getdbt.com/discussions
about: Need help troubleshooting? Check out our guide on how to ask
- name: Contact dbt Cloud support
url: mailto:support@getdbt.com
about: Are you using dbt Cloud? Contact our support team for help!
- name: Participate in Discussions
url: https://github.com/dbt-labs/dbt-core/discussions
about: Do you have a Big Idea for dbt? Read open discussions, or start a new one
- name: Create an issue for dbt-redshift - name: Create an issue for dbt-redshift
url: https://github.com/dbt-labs/dbt-redshift/issues/new/choose url: https://github.com/dbt-labs/dbt-redshift/issues/new/choose
about: Report a bug or request a feature for dbt-redshift about: Report a bug or request a feature for dbt-redshift
@@ -18,6 +8,9 @@ contact_links:
- name: Create an issue for dbt-snowflake - name: Create an issue for dbt-snowflake
url: https://github.com/dbt-labs/dbt-snowflake/issues/new/choose url: https://github.com/dbt-labs/dbt-snowflake/issues/new/choose
about: Report a bug or request a feature for dbt-snowflake about: Report a bug or request a feature for dbt-snowflake
- name: Create an issue for dbt-spark - name: Ask a question or get support
url: https://github.com/dbt-labs/dbt-spark/issues/new/choose url: https://docs.getdbt.com/docs/guides/getting-help
about: Report a bug or request a feature for dbt-spark about: Ask a question or request support
- name: Questions on Stack Overflow
url: https://stackoverflow.com/questions/tagged/dbt
about: Look at questions/answers at Stack Overflow

View File

@@ -1,5 +1,5 @@
name: ✨ Feature name: ✨ Feature
description: Propose a straightforward extension of dbt functionality description: Suggest an idea for dbt
title: "[Feature] <title>" title: "[Feature] <title>"
labels: ["enhancement", "triage"] labels: ["enhancement", "triage"]
body: body:
@@ -9,24 +9,18 @@ body:
Thanks for taking the time to fill out this feature request! Thanks for taking the time to fill out this feature request!
- type: checkboxes - type: checkboxes
attributes: attributes:
label: Is this your first time submitting a feature request? label: Is there an existing feature request for this?
description: > description: Please search to see if an issue already exists for the feature you would like.
We want to make sure that features are distinct and discoverable, options:
so that other members of the community can find them and offer their thoughts. - label: I have searched the existing issues
required: true
Issues are the right place to request straightforward extensions of existing dbt functionality. label: Is this your first time opening an issue?
For "big ideas" about future capabilities of dbt, we ask that you open a
[discussion](https://github.com/dbt-labs/dbt-core/discussions) in the "Ideas" category instead.
options: options:
- label: I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations) - label: I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
required: true required: true
- label: I have searched the existing issues, and I could not find an existing issue for this feature
required: true
- label: I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
required: true
- type: textarea - type: textarea
attributes: attributes:
label: Describe the feature label: Describe the Feature
description: A clear and concise description of what you want to happen. description: A clear and concise description of what you want to happen.
validations: validations:
required: true required: true

View File

@@ -1,93 +0,0 @@
name: ☣️ Regression
description: Report a regression you've observed in a newer version of dbt
title: "[Regression] <title>"
labels: ["bug", "regression", "triage"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this regression report!
- type: checkboxes
attributes:
label: Is this a regression in a recent version of dbt-core?
description: >
A regression is when documented functionality works as expected in an older version of dbt-core,
and no longer works after upgrading to a newer version of dbt-core
options:
- label: I believe this is a regression in dbt-core functionality
required: true
- label: I have searched the existing issues, and I could not find an existing issue for this regression
required: true
- type: textarea
attributes:
label: Current Behavior
description: A concise description of what you're experiencing.
validations:
required: true
- type: textarea
attributes:
label: Expected/Previous Behavior
description: A concise description of what you expected to happen.
validations:
required: true
- type: textarea
attributes:
label: Steps To Reproduce
description: Steps to reproduce the behavior.
placeholder: |
1. In this environment...
2. With this config...
3. Run '...'
4. See error...
validations:
required: true
- type: textarea
id: logs
attributes:
label: Relevant log output
description: |
If applicable, log output to help explain your problem.
render: shell
validations:
required: false
- type: textarea
attributes:
label: Environment
description: |
examples:
- **OS**: Ubuntu 20.04
- **Python**: 3.9.12 (`python3 --version`)
- **dbt-core (working version)**: 1.1.1 (`dbt --version`)
- **dbt-core (regression version)**: 1.2.0 (`dbt --version`)
value: |
- OS:
- Python:
- dbt (working version):
- dbt (regression version):
render: markdown
validations:
required: true
- type: dropdown
id: database
attributes:
label: Which database adapter are you using with dbt?
description: If the regression is specific to the database or adapter, please open the issue in that adapter's repository instead
multiple: true
options:
- postgres
- redshift
- snowflake
- bigquery
- spark
- other (mention it in "Additional Context")
validations:
required: false
- type: textarea
attributes:
label: Additional Context
description: |
Links? References? Anything that will give us more context about the issue you are encountering!
Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in.
validations:
required: false

216
.github/_README.md vendored
View File

@@ -1,216 +0,0 @@
<!-- GitHub will publish this readme on the main repo page if the name is `README.md` so we've added the leading underscore to prevent this -->
<!-- Do not rename this file `README.md` -->
<!-- See https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-readmes -->
## What are GitHub Actions?
GitHub Actions are used for many different purposes. We use them to run tests in CI, validate PRs are in an expected state, and automate processes.
- [Overview of GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions)
- [What's a workflow?](https://docs.github.com/en/actions/using-workflows/about-workflows)
- [GitHub Actions guides](https://docs.github.com/en/actions/guides)
___
## Where do actions and workflows live
We try to maintain actions that are shared across repositories in a single place so that necesary changes can be made in a single place.
[dbt-labs/actions](https://github.com/dbt-labs/actions/) is the central repository of actions and workflows we use across repositories.
GitHub Actions also live locally within a repository. The workflows can be found at `.github/workflows` from the root of the repository. These should be specific to that code base.
Note: We are actively moving actions into the central Action repository so there is currently some duplication across repositories.
___
## Basics of Using Actions
### Viewing Output
- View the detailed action output for your PR in the **Checks** tab of the PR. This only shows the most recent run. You can also view high level **Checks** output at the bottom on the PR.
- View _all_ action output for a repository from the [**Actions**](https://github.com/dbt-labs/dbt-core/actions) tab. Workflow results last 1 year. Artifacts last 90 days, unless specified otherwise in individual workflows.
This view often shows what seem like duplicates of the same workflow. This occurs when files are renamed but the workflow name has not changed. These are in fact _not_ duplicates.
You can see the branch the workflow runs from in this view. It is listed in the table between the workflow name and the time/duration of the run. When blank, the workflow is running in the context of the `main` branch.
### How to view what workflow file is being referenced from a run
- When viewing the output of a specific workflow run, click the 3 dots at the top right of the display. There will be an option to `View workflow file`.
### How to manually run a workflow
- If a workflow has the `on: workflow_dispatch` trigger, it can be manually triggered
- From the [**Actions**](https://github.com/dbt-labs/dbt-core/actions) tab, find the workflow you want to run, select it and fill in any inputs requied. That's it!
### How to re-run jobs
- Some actions cannot be rerun in the GitHub UI. Namely the snyk checks and the cla check. Snyk checks are rerun by closing and reopening the PR. You can retrigger the cla check by commenting on the PR with `@cla-bot check`
___
## General Standards
### Permissions
- By default, workflows have read permissions in the repository for the contents scope only when no permissions are explicitly set.
- It is best practice to always define the permissions explicitly. This will allow actions to continue to work when the default permissions on the repository are changed. It also allows explicit grants of the least permissions possible.
- There are a lot of permissions available. [Read up on them](https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs) if you're unsure what to use.
```yaml
permissions:
contents: read
pull-requests: write
```
### Secrets
- When to use a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) vs the [GITHUB_TOKEN](https://docs.github.com/en/actions/security-guides/automatic-token-authentication) generated for the action?
The `GITHUB_TOKEN` is used by default. In most cases it is sufficient for what you need.
If you expect the workflow to result in a commit to that should retrigger workflows, you will need to use a Personal Access Token for the bot to commit the file. When using the GITHUB_TOKEN, the resulting commit will not trigger another GitHub Actions Workflow run. This is due to limitations set by GitHub. See [the docs](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#using-the-github_token-in-a-workflow) for a more detailed explanation.
For example, we must use a PAT in our workflow to commit a new changelog yaml file for bot PRs. Once the file has been committed to the branch, it should retrigger the check to validate that a changelog exists on the PR. Otherwise, it would stay in a failed state since the check would never retrigger.
### Triggers
You can configure your workflows to run when specific activity on GitHub happens, at a scheduled time, or when an event outside of GitHub occurs. Read more details in the [GitHub docs](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows).
These triggers are under the `on` key of the workflow and more than one can be listed.
```yaml
on:
push:
branches:
- "main"
- "*.latest"
- "releases/*"
pull_request:
# catch when the PR is opened with the label or when the label is added
types: [opened, labeled]
workflow_dispatch:
```
Some triggers of note that we use:
- `push` - Runs your workflow when you push a commit or tag.
- `pull_request` - Runs your workflow when activity on a pull request in the workflow's repository occurs. Takes in a list of activity types (opened, labeled, etc) if appropriate.
- `pull_request_target` - Same as `pull_request` but runs in the context of the PR target branch.
- `workflow_call` - used with reusable workflows. Triggered by another workflow calling it.
- `workflow_dispatch` - Gives the ability to manually trigger a workflow from the GitHub API, GitHub CLI, or GitHub browser interface.
### Basic Formatting
- Add a description of what your workflow does at the top in this format
```
# **what?**
# Describe what the action does.
# **why?**
# Why does this action exist?
# **when?**
# How/when will it be triggered?
```
- Leave blank lines between steps and jobs
```yaml
jobs:
dependency_changelog:
runs-on: ubuntu-latest
steps:
- name: Get File Name Timestamp
id: filename_time
uses: nanzm/get-time-action@v1.1
with:
format: 'YYYYMMDD-HHmmss'
- name: Get File Content Timestamp
id: file_content_time
uses: nanzm/get-time-action@v1.1
with:
format: 'YYYY-MM-DDTHH:mm:ss.000000-05:00'
- name: Generate Filepath
id: fp
run: |
FILEPATH=.changes/unreleased/Dependencies-${{ steps.filename_time.outputs.time }}.yaml
echo "FILEPATH=$FILEPATH" >> $GITHUB_OUTPUT
```
- Print out all variables you will reference as the first step of a job. This allows for easier debugging. The first job should log all inputs. Subsequent jobs should reference outputs of other jobs, if present.
When possible, generate variables at the top of your workflow in a single place to reference later. This is not always strictly possible since you may generate a value to be used later mid-workflow.
Be sure to use quotes around these logs so special characters are not interpreted.
```yaml
job1:
- name: "[DEBUG] Print Variables"
run: |
echo "all variables defined as inputs"
echo "The last commit sha in the release: ${{ inputs.sha }}"
echo "The release version number: ${{ inputs.version_number }}"
echo "The changelog_path: ${{ inputs.changelog_path }}"
echo "The build_script_path: ${{ inputs.build_script_path }}"
echo "The s3_bucket_name: ${{ inputs.s3_bucket_name }}"
echo "The package_test_command: ${{ inputs.package_test_command }}"
# collect all the variables that need to be used in subsequent jobs
- name: Set Variables
id: variables
run: |
echo "important_path='performance/runner/Cargo.toml'" >> $GITHUB_OUTPUT
echo "release_id=${{github.event.inputs.release_id}}" >> $GITHUB_OUTPUT
echo "open_prs=${{github.event.inputs.open_prs}}" >> $GITHUB_OUTPUT
job2:
needs: [job1]
- name: "[DEBUG] Print Variables"
run: |
echo "all variables defined in job1 > Set Variables > outputs"
echo "important_path: ${{ needs.job1.outputs.important_path }}"
echo "release_id: ${{ needs.job1.outputs.release_id }}"
echo "open_prs: ${{ needs.job1.outputs.open_prs }}"
```
- When it's not obvious what something does, add a comment!
___
## Tips
### Context
- The [GitHub CLI](https://cli.github.com/) is available in the default runners
- Actions run in your context. ie, using an action from the marketplace that uses the GITHUB_TOKEN uses the GITHUB_TOKEN generated by your workflow run.
### Actions from the Marketplace
- Dont use external actions for things that can easily be accomplished manually.
- Always read through what an external action does before using it! Often an action in the GitHub Actions Marketplace can be replaced with a few lines in bash. This is much more maintainable (and wont change under us) and clear as to whats actually happening. It also prevents any
- Pin actions _we don't control_ to tags.
### Connecting to AWS
- Authenticate with the aws managed workflow
```yaml
- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
```
- Then access with the aws command that comes installed on the action runner machines
```yaml
- name: Copy Artifacts from S3 via CLI
run: aws s3 cp ${{ env.s3_bucket }} . --recursive
```
### Testing
- Depending on what your action does, you may be able to use [`act`](https://github.com/nektos/act) to test the action locally. Some features of GitHub Actions do not work with `act`, among those are reusable workflows. If you can't use `act`, you'll have to push your changes up before being able to test. This can be slow.

View File

@@ -28,12 +28,11 @@ if __name__ == "__main__":
if package_request.status_code == 404: if package_request.status_code == 404:
if halt_on_missing: if halt_on_missing:
sys.exit(1) sys.exit(1)
# everything is the latest if the package doesn't exist else:
github_output = os.environ.get("GITHUB_OUTPUT") # everything is the latest if the package doesn't exist
with open(github_output, "at", encoding="utf-8") as gh_output: print(f"::set-output name=latest::{True}")
gh_output.write("latest=True") print(f"::set-output name=minor_latest::{True}")
gh_output.write("minor_latest=True") sys.exit(0)
sys.exit(0)
# TODO: verify package meta is "correct" # TODO: verify package meta is "correct"
# https://github.com/dbt-labs/dbt-core/issues/4640 # https://github.com/dbt-labs/dbt-core/issues/4640
@@ -92,7 +91,5 @@ if __name__ == "__main__":
latest = is_latest(pre_rel, new_version, current_latest) latest = is_latest(pre_rel, new_version, current_latest)
minor_latest = is_latest(pre_rel, new_version, current_minor_latest) minor_latest = is_latest(pre_rel, new_version, current_minor_latest)
github_output = os.environ.get("GITHUB_OUTPUT") print(f"::set-output name=latest::{latest}")
with open(github_output, "at", encoding="utf-8") as gh_output: print(f"::set-output name=minor_latest::{minor_latest}")
gh_output.write(f"latest={latest}")
gh_output.write(f"minor_latest={minor_latest}")

View File

@@ -20,4 +20,4 @@ resolves #
- [ ] I have run this code in development and it appears to resolve the stated issue - [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR - [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] I have [opened an issue to add/update docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose), or docs changes are not required/relevant for this PR - [ ] I have [opened an issue to add/update docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose), or docs changes are not required/relevant for this PR
- [ ] I have run `changie new` to [create a changelog entry](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-a-changelog-entry) - [ ] I have run `changie new` to [create a changelog entry](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#Adding-CHANGELOG-Entry)

View File

@@ -1,61 +0,0 @@
# **what?**
# When bots create a PR, this action will add a corresponding changie yaml file to that
# PR when a specific label is added.
#
# The file is created off a template:
#
# kind: <per action matrix>
# body: <PR title>
# time: <current timestamp>
# custom:
# Author: <PR User Login (generally the bot)>
# Issue: 4904
# PR: <PR number>
#
# **why?**
# Automate changelog generation for more visability with automated bot PRs.
#
# **when?**
# Once a PR is created, label should be added to PR before or after creation. You can also
# manually trigger this by adding the appropriate label at any time.
#
# **how to add another bot?**
# Add the label and changie kind to the include matrix. That's it!
#
name: Bot Changelog
on:
pull_request:
# catch when the PR is opened with the label or when the label is added
types: [labeled]
permissions:
contents: write
pull-requests: read
jobs:
generate_changelog:
strategy:
matrix:
include:
- label: "dependencies"
changie_kind: "Dependencies"
- label: "snyk"
changie_kind: "Security"
runs-on: ubuntu-latest
steps:
- name: Create and commit changelog on bot PR
if: ${{ contains(github.event.pull_request.labels.*.name, matrix.label) }}
id: bot_changelog
uses: emmyoop/changie_bot@v1.0.1
with:
GITHUB_TOKEN: ${{ secrets.FISHTOWN_BOT_PAT }}
commit_author_name: "Github Build Bot"
commit_author_email: "<buildbot@fishtownanalytics.com>"
commit_message: "Add automated changelog yaml from template for bot PR"
changie_kind: ${{ matrix.changie_kind }}
label: ${{ matrix.label }}
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n PR: ${{ github.event.pull_request.number }}"

78
.github/workflows/changelog-check.yml vendored Normal file
View File

@@ -0,0 +1,78 @@
# **what?**
# Checks that a file has been committed under the /.changes directory
# as a new CHANGELOG entry. Cannot check for a specific filename as
# it is dynamically generated by change type and timestamp.
# This workflow should not require any secrets since it runs for PRs
# from forked repos.
# By default, secrets are not passed to workflows running from
# a forked repo.
# **why?**
# Ensure code change gets reflected in the CHANGELOG.
# **when?**
# This will run for all PRs going into main and *.latest. It will
# run when they are opened, reopened, when any label is added or removed
# and when new code is pushed to the branch. The action will then get
# skipped if the 'Skip Changelog' label is present is any of the labels.
name: Check Changelog Entry
on:
pull_request:
types: [opened, reopened, labeled, unlabeled, synchronize]
workflow_dispatch:
defaults:
run:
shell: bash
permissions:
contents: read
pull-requests: write
env:
changelog_comment: 'Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry).'
jobs:
changelog:
name: changelog
if: "!contains(github.event.pull_request.labels.*.name, 'Skip Changelog')"
runs-on: ubuntu-latest
steps:
- name: Check if changelog file was added
# https://github.com/marketplace/actions/paths-changes-filter
# For each filter, it sets output variable named by the filter to the text:
# 'true' - if any of changed files matches any of filter rules
# 'false' - if none of changed files matches any of filter rules
# also, returns:
# `changes` - JSON array with names of all filters matching any of the changed files
uses: dorny/paths-filter@v2
id: filter
with:
token: ${{ secrets.GITHUB_TOKEN }}
filters: |
changelog:
- added: '.changes/unreleased/**.yaml'
- name: Check if comment already exists
uses: peter-evans/find-comment@v1
id: changelog_comment
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: 'github-actions[bot]'
body-includes: ${{ env.changelog_comment }}
- name: Create PR comment if changelog entry is missing, required, and does not exist
if: |
steps.filter.outputs.changelog == 'false' &&
steps.changelog_comment.outputs.comment-body == ''
uses: peter-evans/create-or-update-comment@v1
with:
issue-number: ${{ github.event.pull_request.number }}
body: ${{ env.changelog_comment }}
- name: Fail job if changelog entry is missing and required
if: steps.filter.outputs.changelog == 'false'
uses: actions/github-script@v6
with:
script: core.setFailed('Changelog entry required to merge.')

View File

@@ -1,40 +0,0 @@
# **what?**
# Checks that a file has been committed under the /.changes directory
# as a new CHANGELOG entry. Cannot check for a specific filename as
# it is dynamically generated by change type and timestamp.
# This workflow should not require any secrets since it runs for PRs
# from forked repos.
# By default, secrets are not passed to workflows running from
# a forked repo.
# **why?**
# Ensure code change gets reflected in the CHANGELOG.
# **when?**
# This will run for all PRs going into main and *.latest. It will
# run when they are opened, reopened, when any label is added or removed
# and when new code is pushed to the branch. The action will then get
# skipped if the 'Skip Changelog' label is present is any of the labels.
name: Check Changelog Entry
on:
pull_request:
types: [opened, reopened, labeled, unlabeled, synchronize]
workflow_dispatch:
defaults:
run:
shell: bash
permissions:
contents: read
pull-requests: write
jobs:
changelog:
uses: dbt-labs/actions/.github/workflows/changelog-existence.yml@main
with:
changelog_comment: 'Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry).'
skip_label: 'Skip Changelog'
secrets: inherit

View File

@@ -0,0 +1,114 @@
# **what?**
# When dependabot create a PR, it always adds the `dependencies` label. This
# action will add a corresponding changie yaml file to that PR when that label is added.
# The file is created off a template:
#
# kind: Dependencies
# body: <PR title>
# time: <current timestamp>
# custom:
# Author: dependabot
# Issue: 4904
# PR: <PR number>
#
# **why?**
# Automate changelog generation for more visability with automated dependency updates via dependabot.
# **when?**
# Once a PR is created and it has been correctly labeled with `dependencies`. The intended use
# is for the PRs created by dependabot. You can also manually trigger this by adding the
# `dependencies` label at any time.
name: Dependency Changelog
on:
pull_request:
# catch when the PR is opened with the label or when the label is added
types: [opened, labeled]
permissions:
contents: write
pull-requests: read
jobs:
dependency_changelog:
if: "contains(github.event.pull_request.labels.*.name, 'dependencies')"
runs-on: ubuntu-latest
steps:
# timestamp changes the order the changelog entries are listed in the final Changelog.md file. Precision is not
# important here.
# The timestamp on the filename and the timestamp in the contents of the file have different expected formats.
- name: Get File Name Timestamp
id: filename_time
uses: nanzm/get-time-action@v1.1
with:
format: 'YYYYMMDD-HHmmss'
- name: Get File Content Timestamp
id: file_content_time
uses: nanzm/get-time-action@v1.1
with:
format: 'YYYY-MM-DDTHH:mm:ss.000000-05:00'
# changie expects files to be named in a specific pattern.
- name: Generate Filepath
id: fp
run: |
FILEPATH=.changes/unreleased/Dependencies-${{ steps.filename_time.outputs.time }}.yaml
echo "::set-output name=FILEPATH::$FILEPATH"
- name: Check if changelog file exists already
# if there's already a changelog entry, don't add another one!
# https://github.com/marketplace/actions/paths-changes-filter
# For each filter, it sets output variable named by the filter to the text:
# 'true' - if any of changed files matches any of filter rules
# 'false' - if none of changed files matches any of filter rules
# also, returns:
# `changes` - JSON array with names of all filters matching any of the changed files
uses: dorny/paths-filter@v2
id: changelog_check
with:
token: ${{ secrets.GITHUB_TOKEN }}
filters: |
exists:
- added: '.changes/unreleased/**.yaml'
- name: Checkout Branch
if: steps.changelog_check.outputs.exists == 'false'
uses: actions/checkout@v2
with:
# specifying the ref avoids checking out the repository in a detached state
ref: ${{ github.event.pull_request.head.ref }}
# If this is not set to false, Git push is performed with github.token and not the token
# configured using the env: GITHUB_TOKEN in commit step
persist-credentials: false
- name: Create file from template
if: steps.changelog_check.outputs.exists == 'false'
run: |
echo kind: Dependencies > "${{ steps.fp.outputs.FILEPATH }}"
echo 'body: "${{ github.event.pull_request.title }}"' >> "${{ steps.fp.outputs.FILEPATH }}"
echo time: "${{ steps.file_content_time.outputs.time }}" >> "${{ steps.fp.outputs.FILEPATH }}"
echo custom: >> "${{ steps.fp.outputs.FILEPATH }}"
echo ' Author: ${{ github.event.pull_request.user.login }}' >> "${{ steps.fp.outputs.FILEPATH }}"
echo ' Issue: "4904"' >> "${{ steps.fp.outputs.FILEPATH }}" # github.event.pull_request.issue for auto id?
echo ' PR: "${{ github.event.pull_request.number }}"' >> "${{ steps.fp.outputs.FILEPATH }}"
- name: Commit Changelog File
if: steps.changelog_check.outputs.exists == 'false'
uses: gr2m/create-or-update-pull-request-action@v1
env:
# When using the GITHUB_TOKEN, the resulting commit will not trigger another GitHub Actions
# Workflow run. This is due to limitations set by GitHub.
# See: https://docs.github.com/en/actions/security-guides/automatic-token-authentication#using-the-github_token-in-a-workflow
# When you use the repository's GITHUB_TOKEN to perform tasks on behalf of the GitHub Actions
# app, events triggered by the GITHUB_TOKEN will not create a new workflow run. This prevents
# you from accidentally creating recursive workflow runs. To get around this, use a Personal
# Access Token to commit changes.
GITHUB_TOKEN: ${{ secrets.FISHTOWN_BOT_PAT }}
with:
branch: ${{ github.event.pull_request.head.ref }}
# author expected in the format "Lorem J. Ipsum <lorem@example.com>"
author: "Github Build Bot <buildbot@fishtownanalytics.com>"
commit-message: "Add automated changelog yaml from template"

View File

@@ -1,165 +0,0 @@
# **what?**
# On push, if anything in core/dbt/docs or core/dbt/cli has been
# created or modified, regenerate the CLI API docs using sphinx.
# **why?**
# We watch for changes in core/dbt/cli because the CLI API docs rely on click
# and all supporting flags/params to be generated. We watch for changes in
# core/dbt/docs since any changes to sphinx configuration or any of the
# .rst files there could result in a differently build final index.html file.
# **when?**
# Whenever a change has been pushed to a branch, and only if there is a diff
# between the PR branch and main's core/dbt/cli and or core/dbt/docs dirs.
# TODO: add bot comment to PR informing contributor that the docs have been committed
# TODO: figure out why github action triggered pushes cause github to fail to report
# the status of jobs
name: Generate CLI API docs
on:
pull_request:
permissions:
contents: write
pull-requests: write
env:
CLI_DIR: ${{ github.workspace }}/core/dbt/cli
DOCS_DIR: ${{ github.workspace }}/core/dbt/docs
DOCS_BUILD_DIR: ${{ github.workspace }}/core/dbt/docs/build
jobs:
check_gen:
name: check if generation needed
runs-on: ubuntu-latest
if: ${{ github.event.pull_request.head.repo.fork == false }}
outputs:
cli_dir_changed: ${{ steps.check_cli.outputs.cli_dir_changed }}
docs_dir_changed: ${{ steps.check_docs.outputs.docs_dir_changed }}
steps:
- name: "[DEBUG] print variables"
run: |
echo "env.CLI_DIR: ${{ env.CLI_DIR }}"
echo "env.DOCS_BUILD_DIR: ${{ env.DOCS_BUILD_DIR }}"
echo "env.DOCS_DIR: ${{ env.DOCS_DIR }}"
- name: git checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
ref: ${{ github.head_ref }}
- name: set shas
id: set_shas
run: |
THIS_SHA=$(git rev-parse @)
LAST_SHA=$(git rev-parse @~1)
echo "this sha: $THIS_SHA"
echo "last sha: $LAST_SHA"
echo "this_sha=$THIS_SHA" >> $GITHUB_OUTPUT
echo "last_sha=$LAST_SHA" >> $GITHUB_OUTPUT
- name: check for changes in core/dbt/cli
id: check_cli
run: |
CLI_DIR_CHANGES=$(git diff \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.CLI_DIR }})
if [ -n "$CLI_DIR_CHANGES" ]; then
echo "changes found"
echo $CLI_DIR_CHANGES
echo "cli_dir_changed=true" >> $GITHUB_OUTPUT
exit 0
fi
echo "cli_dir_changed=false" >> $GITHUB_OUTPUT
echo "no changes found"
- name: check for changes in core/dbt/docs
id: check_docs
if: steps.check_cli.outputs.cli_dir_changed == 'false'
run: |
DOCS_DIR_CHANGES=$(git diff --name-only \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.DOCS_DIR }} ':!${{ env.DOCS_BUILD_DIR }}')
DOCS_BUILD_DIR_CHANGES=$(git diff --name-only \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.DOCS_BUILD_DIR }})
if [ -n "$DOCS_DIR_CHANGES" ] && [ -z "$DOCS_BUILD_DIR_CHANGES" ]; then
echo "changes found"
echo $DOCS_DIR_CHANGES
echo "docs_dir_changed=true" >> $GITHUB_OUTPUT
exit 0
fi
echo "docs_dir_changed=false" >> $GITHUB_OUTPUT
echo "no changes found"
gen_docs:
name: generate docs
runs-on: ubuntu-latest
needs: [check_gen]
if: |
needs.check_gen.outputs.cli_dir_changed == 'true'
|| needs.check_gen.outputs.docs_dir_changed == 'true'
steps:
- name: "[DEBUG] print variables"
run: |
echo "env.DOCS_DIR: ${{ env.DOCS_DIR }}"
echo "github head_ref: ${{ github.head_ref }}"
- name: git checkout
uses: actions/checkout@v3
with:
ref: ${{ github.head_ref }}
- name: install python
uses: actions/setup-python@v4.3.0
with:
python-version: 3.8
- name: install dev requirements
run: |
python3 -m venv env
source env/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt -r dev-requirements.txt
- name: generate docs
run: |
source env/bin/activate
cd ${{ env.DOCS_DIR }}
echo "cleaning existing docs"
make clean
echo "creating docs"
make html
- name: debug
run: |
echo ">>>>> status"
git status
echo ">>>>> remotes"
git remote -v
echo ">>>>> branch"
git branch -v
echo ">>>>> log"
git log --pretty=oneline | head -5
- name: commit docs
run: |
git config user.name 'Github Build Bot'
git config user.email 'buildbot@fishtownanalytics.com'
git commit -am "Add generated CLI API docs"
git push -u origin ${{ github.head_ref }}

View File

@@ -15,9 +15,6 @@ on:
issues: issues:
types: [closed, deleted, reopened] types: [closed, deleted, reopened]
# no special access is needed
permissions: read-all
jobs: jobs:
call-label-action: call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-transition.yml@main uses: dbt-labs/jira-actions/.github/workflows/jira-transition.yml@main

View File

@@ -38,7 +38,6 @@ jobs:
name: code-quality name: code-quality
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 10
steps: steps:
- name: Check out the repository - name: Check out the repository
@@ -68,12 +67,11 @@ jobs:
name: unit test / python ${{ matrix.python-version }} name: unit test / python ${{ matrix.python-version }}
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 10
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"] python-version: ["3.7", "3.8", "3.9", "3.10"]
env: env:
TOXENV: "unit" TOXENV: "unit"
@@ -101,9 +99,7 @@ jobs:
- name: Get current date - name: Get current date
if: always() if: always()
id: date id: date
run: | run: echo "::set-output name=date::$(date +'%Y-%m-%dT%H_%M_%S')" #no colons allowed for artifacts
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v2 - uses: actions/upload-artifact@v2
if: always() if: always()
@@ -115,13 +111,12 @@ jobs:
name: integration test / python ${{ matrix.python-version }} / ${{ matrix.os }} name: integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
timeout-minutes: 45
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"] python-version: ["3.7", "3.8", "3.9", "3.10"]
os: [ubuntu-20.04] os: [ubuntu-latest]
include: include:
- python-version: 3.8 - python-version: 3.8
os: windows-latest os: windows-latest
@@ -170,9 +165,7 @@ jobs:
- name: Get current date - name: Get current date
if: always() if: always()
id: date id: date
run: | run: echo "::set-output name=date::$(date +'%Y_%m_%dT%H_%M_%S')" #no colons allowed for artifacts
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v2 - uses: actions/upload-artifact@v2
if: always() if: always()

View File

@@ -39,7 +39,7 @@ jobs:
max-parallel: 1 max-parallel: 1
fail-fast: false fail-fast: false
matrix: matrix:
branch: [1.0.latest, 1.1.latest, 1.2.latest, 1.3.latest, main] branch: [1.0.latest, 1.1.latest, main]
steps: steps:
- name: Call CI workflow for ${{ matrix.branch }} branch - name: Call CI workflow for ${{ matrix.branch }} branch

View File

@@ -41,9 +41,9 @@ jobs:
id: version id: version
run: | run: |
IFS="." read -r MAJOR MINOR PATCH <<< ${{ github.event.inputs.version_number }} IFS="." read -r MAJOR MINOR PATCH <<< ${{ github.event.inputs.version_number }}
echo "major=$MAJOR" >> $GITHUB_OUTPUT echo "::set-output name=major::$MAJOR"
echo "minor=$MINOR" >> $GITHUB_OUTPUT echo "::set-output name=minor::$MINOR"
echo "patch=$PATCH" >> $GITHUB_OUTPUT echo "::set-output name=patch::$PATCH"
- name: Is pkg 'latest' - name: Is pkg 'latest'
id: latest id: latest
@@ -70,10 +70,8 @@ jobs:
- name: Get docker build arg - name: Get docker build arg
id: build_arg id: build_arg
run: | run: |
BUILD_ARG_NAME=$(echo ${{ github.event.inputs.package }} | sed 's/\-/_/g') echo "::set-output name=build_arg_name::"$(echo ${{ github.event.inputs.package }} | sed 's/\-/_/g')
BUILD_ARG_VALUE=$(echo ${{ github.event.inputs.package }} | sed 's/postgres/core/g') echo "::set-output name=build_arg_value::"$(echo ${{ github.event.inputs.package }} | sed 's/postgres/core/g')
echo "build_arg_name=$BUILD_ARG_NAME" >> $GITHUB_OUTPUT
echo "build_arg_value=$BUILD_ARG_VALUE" >> $GITHUB_OUTPUT
- name: Log in to the GHCR - name: Log in to the GHCR
uses: docker/login-action@v1 uses: docker/login-action@v1

View File

@@ -20,9 +20,6 @@ on:
description: 'The release version number (i.e. 1.0.0b1)' description: 'The release version number (i.e. 1.0.0b1)'
required: true required: true
permissions:
contents: write # this is the permission that allows creating a new release
defaults: defaults:
run: run:
shell: bash shell: bash
@@ -165,7 +162,7 @@ jobs:
env: env:
IS_PRERELEASE: ${{ contains(github.event.inputs.version_number, 'rc') || contains(github.event.inputs.version_number, 'b') }} IS_PRERELEASE: ${{ contains(github.event.inputs.version_number, 'rc') || contains(github.event.inputs.version_number, 'b') }}
run: | run: |
echo "isPrerelease=$IS_PRERELEASE" >> $GITHUB_OUTPUT echo ::set-output name=isPrerelease::$IS_PRERELEASE
- name: Creating GitHub Release - name: Creating GitHub Release
uses: softprops/action-gh-release@v1 uses: softprops/action-gh-release@v1

View File

@@ -21,9 +21,6 @@ on:
- "*.latest" - "*.latest"
- "releases/*" - "releases/*"
# no special access is needed
permissions: read-all
env: env:
LATEST_SCHEMA_PATH: ${{ github.workspace }}/new_schemas LATEST_SCHEMA_PATH: ${{ github.workspace }}/new_schemas
SCHEMA_DIFF_ARTIFACT: ${{ github.workspace }}//schema_schanges.txt SCHEMA_DIFF_ARTIFACT: ${{ github.workspace }}//schema_schanges.txt

View File

@@ -3,10 +3,15 @@ on:
schedule: schedule:
- cron: "30 1 * * *" - cron: "30 1 * * *"
permissions:
issues: write
pull-requests: write
jobs: jobs:
stale: stale:
uses: dbt-labs/actions/.github/workflows/stale-bot-matrix.yml@main runs-on: ubuntu-latest
steps:
# pinned at v4 (https://github.com/actions/stale/releases/tag/v4.0.0)
- uses: actions/stale@cdf15f641adb27a71842045a94023bef6945e3aa
with:
stale-issue-message: "This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days."
stale-pr-message: "This PR has been marked as Stale because it has been open for 180 days with no activity. If you would like the PR to remain open, please remove the stale label or comment on the PR, or it will be closed in 7 days."
close-issue-message: "Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers."
# mark issues/PRs stale when they haven't seen activity in 180 days
days-before-stale: 180

View File

@@ -22,7 +22,7 @@ jobs:
# run the performance measurements on the current or default branch # run the performance measurements on the current or default branch
test-schema: test-schema:
name: Test Log Schema name: Test Log Schema
runs-on: ubuntu-20.04 runs-on: ubuntu-latest
env: env:
# turns warnings into errors # turns warnings into errors
RUSTFLAGS: "-D warnings" RUSTFLAGS: "-D warnings"
@@ -46,6 +46,12 @@ jobs:
with: with:
python-version: "3.8" python-version: "3.8"
- uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: stable
override: true
- name: Install python dependencies - name: Install python dependencies
run: | run: |
pip install --user --upgrade pip pip install --user --upgrade pip
@@ -63,3 +69,10 @@ jobs:
# we actually care if these pass, because the normal test run doesn't usually include many json log outputs # we actually care if these pass, because the normal test run doesn't usually include many json log outputs
- name: Run integration tests - name: Run integration tests
run: tox -e integration -- -nauto run: tox -e integration -- -nauto
# apply our schema tests to every log event from the previous step
# skips any output that isn't valid json
- uses: actions-rs/cargo@v1
with:
command: run
args: --manifest-path test/interop/log_parsing/Cargo.toml

View File

@@ -20,10 +20,6 @@ on:
description: 'The version number to bump to (ex. 1.2.0, 1.3.0b1)' description: 'The version number to bump to (ex. 1.2.0, 1.3.0b1)'
required: true required: true
permissions:
contents: write
pull-requests: write
jobs: jobs:
bump: bump:
runs-on: ubuntu-latest runs-on: ubuntu-latest
@@ -65,7 +61,7 @@ jobs:
- name: Set branch value - name: Set branch value
id: variables id: variables
run: | run: |
echo "BRANCH_NAME=prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID" >> $GITHUB_OUTPUT echo "::set-output name=BRANCH_NAME::prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID"
- name: Create PR branch - name: Create PR branch
run: | run: |

8
.gitignore vendored
View File

@@ -11,7 +11,6 @@ __pycache__/
env*/ env*/
dbt_env/ dbt_env/
build/ build/
!core/dbt/docs/build
develop-eggs/ develop-eggs/
dist/ dist/
downloads/ downloads/
@@ -25,8 +24,7 @@ var/
*.egg-info/ *.egg-info/
.installed.cfg .installed.cfg
*.egg *.egg
.mypy_cache/ *.mypy_cache/
.dmypy.json
logs/ logs/
# PyInstaller # PyInstaller
@@ -97,7 +95,3 @@ venv/
# vscode # vscode
.vscode/ .vscode/
*.code-workspace
# poetry
poetry.lock

View File

@@ -2,11 +2,11 @@
# Eventually the hooks described here will be run as tests before merging each PR. # Eventually the hooks described here will be run as tests before merging each PR.
# TODO: remove global exclusion of tests when testing overhaul is complete # TODO: remove global exclusion of tests when testing overhaul is complete
exclude: ^(test/|core/dbt/docs/build/) exclude: ^test/
# Force all unspecified python hooks to run python 3.8 # Force all unspecified python hooks to run python 3.8
default_language_version: default_language_version:
python: python3 python: python3.8
repos: repos:
- repo: https://github.com/pre-commit/pre-commit-hooks - repo: https://github.com/pre-commit/pre-commit-hooks
@@ -24,10 +24,15 @@ repos:
rev: 22.3.0 rev: 22.3.0
hooks: hooks:
- id: black - id: black
args:
- "--line-length=99"
- "--target-version=py38"
- id: black - id: black
alias: black-check alias: black-check
stages: [manual] stages: [manual]
args: args:
- "--line-length=99"
- "--target-version=py38"
- "--check" - "--check"
- "--diff" - "--diff"
- repo: https://github.com/pycqa/flake8 - repo: https://github.com/pycqa/flake8

View File

@@ -5,14 +5,146 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version. - "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry) - Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.2.3 - November 16, 2022
### Features
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201), [#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
### Contributors
- [@racheldaniel](https://github.com/racheldaniel) ([#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
## dbt-core 1.2.2 - October 03, 2022
### Features
- This conditionally no-ops warehouse connection at compile depending on an env var, disabling introspection/queries during compilation only. This is a temporary solution to more complex permissions requirements for the semantic layer. ([#5936](https://github.com/dbt-labs/dbt-core/issues/5936), [#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
### Fixes
- Fix race condition when invoking dbt via lib.py concurrently ([#5919](https://github.com/dbt-labs/dbt-core/issues/5919), [#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
### Contributors
- [@drewbanin](https://github.com/drewbanin) ([#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
- [@racheldaniel](https://github.com/racheldaniel) ([#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
## dbt-core 1.2.1 - August 25, 2022
### Fixes
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
### Contributors
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
## dbt-core 1.2.0 - July 26, 2022
### Features
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
- Adds itertools to modules Jinja namespace ([#5130](https://github.com/dbt-labs/dbt-core/issues/5130), [#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- allow target as an option in profile_template.yml ([#5179](https://github.com/dbt-labs/dbt-core/issues/5179), [#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- seed: Add new macro get_csv_sql ([#5206](https://github.com/dbt-labs/dbt-core/issues/5206), [#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- Grants as Node Configs ([#5189](https://github.com/dbt-labs/dbt-core/issues/5189), [#5230](https://github.com/dbt-labs/dbt-core/pull/5230))
- Adds file selectors and support for file selectors in the default method selector ([#5240](https://github.com/dbt-labs/dbt-core/issues/5240), [#5241](https://github.com/dbt-labs/dbt-core/pull/5241))
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- Allow customizing `target-path` and `log-path` through environment variables and CLI flags. ([#5399](https://github.com/dbt-labs/dbt-core/issues/5399), [#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- Move type_* macros from dbt-utils into dbt-core, with tests ([#5317](https://github.com/dbt-labs/dbt-core/issues/5317), [#5428](https://github.com/dbt-labs/dbt-core/pull/5428))
- Add support for ratio metrics ([#4884](https://github.com/dbt-labs/dbt-core/issues/4884), [#5027](https://github.com/dbt-labs/dbt-core/pull/5027))
- Allow users to define grants as a reasonable default in the dbt_project.yml or within each model sql or yml file combined. ([#5263](https://github.com/dbt-labs/dbt-core/issues/5263), [#5369](https://github.com/dbt-labs/dbt-core/pull/5369))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
### Fixes
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- Use yaml renderer (with target context) for rendering selectors ([#5131](https://github.com/dbt-labs/dbt-core/issues/5131), [#5136](https://github.com/dbt-labs/dbt-core/pull/5136))
- Fix retry logic to return values after initial try ([#5023](https://github.com/dbt-labs/dbt-core/issues/5023), [#5137](https://github.com/dbt-labs/dbt-core/pull/5137))
- Scrub secret env vars from CommandError in exception stacktrace ([#5151](https://github.com/dbt-labs/dbt-core/issues/5151), [#5152](https://github.com/dbt-labs/dbt-core/pull/5152))
- Ensure the metric name does not contain spaces ([#4572](https://github.com/dbt-labs/dbt-core/issues/4572), [#5173](https://github.com/dbt-labs/dbt-core/pull/5173))
- When parsing 'all_sources' should be a list of unique dirs ([#5120](https://github.com/dbt-labs/dbt-core/issues/5120), [#5176](https://github.com/dbt-labs/dbt-core/pull/5176))
- Add warning if yaml contains duplicate keys ([#5114](https://github.com/dbt-labs/dbt-core/issues/5114), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146))
- Modifying the drop_test_schema to work better with Redshift issues around locked tables and current transactions ([#5200](https://github.com/dbt-labs/dbt-core/issues/5200), [#5198](https://github.com/dbt-labs/dbt-core/pull/5198))
- Fix column comparison in snapshot_check_all_get_existing_columns for check-strategy snapshots with explicit check_cols defined ([#5222](https://github.com/dbt-labs/dbt-core/issues/5222), [#5223](https://github.com/dbt-labs/dbt-core/pull/5223))
- Changed how `--select state:modified` detects changes for macros nodes depend on ([#5202](https://github.com/dbt-labs/dbt-core/issues/5202), [#5224](https://github.com/dbt-labs/dbt-core/pull/5224))
- Fix column comparison in snapshot_check_all_get_existing_columns to use adapter.get_columns_in_relation ([#5222](https://github.com/dbt-labs/dbt-core/issues/5222), [#5232](https://github.com/dbt-labs/dbt-core/pull/5232))
- Remove docs file from manifest when removing doc node ([#4146](https://github.com/dbt-labs/dbt-core/issues/4146), [#5270](https://github.com/dbt-labs/dbt-core/pull/5270))
- Remove duplicate dbt script entry ([#5314](https://github.com/dbt-labs/dbt-core/issues/5314), [#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
- Add inheritance to materialization macro resolution ([#4646](https://github.com/dbt-labs/dbt-core/issues/4646), [#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
- Improve pluralizations for Documentation and SqlOperation NodeTypes ([#5352](https://github.com/dbt-labs/dbt-core/issues/5352), [#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- Properly use quotes for Snowflake snapshots when checking all columns ([#2975](https://github.com/dbt-labs/dbt-core/issues/2975), [#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- fixes handling of RESET color code with USE_COLORS=False ([#5288](https://github.com/dbt-labs/dbt-core/issues/5288), [#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
### Docs
- Fixed sample SQL Code for sources when no database is defined ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Add support for `file:` selector in DAG viz ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [Snyk] Upgrade prismjs from 1.27.0 to 1.28.0 ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Run build and tests in CI checks ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Improve metrics DAG viz and documentation page ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Upgrade cytoscape.js fork ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
### Under the Hood
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
- Move package deprecation check outside of package cache ([#5068](https://github.com/dbt-labs/dbt-core/issues/5068), [#5069](https://github.com/dbt-labs/dbt-core/pull/5069))
- removal of scaffold first attempt and create_adapter_plugin.py as they are deprecated new scaffold can be found https://github.com/dbt-labs/dbt-database-adapter-scaffold ([#4980](https://github.com/dbt-labs/dbt-core/issues/4980), [#5117](https://github.com/dbt-labs/dbt-core/pull/5117))
- Mypy -> 0.942 + fixed import logic to allow for full mypy coverage ([#4805](https://github.com/dbt-labs/dbt-core/issues/4805), [#5171](https://github.com/dbt-labs/dbt-core/pull/5171))
- Converted dbt list tests to pytest ([#5049](https://github.com/dbt-labs/dbt-core/issues/5049), [#5178](https://github.com/dbt-labs/dbt-core/pull/5178))
- Fix: Call str and repr for UnsetProfileConfig without a RuntimeException ([#5081](https://github.com/dbt-labs/dbt-core/issues/5081), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- Improve tracking error logging message ([#5197](https://github.com/dbt-labs/dbt-core/issues/5197), [#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- Clean up materialization logic: more consistent relation names, loading from cache ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- Use the default Python version for local dev and test instead of requiring Python 3.8 ([#5257](https://github.com/dbt-labs/dbt-core/issues/5257), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- Fix test for context set function ([#5266](https://github.com/dbt-labs/dbt-core/issues/5266), [#5272](https://github.com/dbt-labs/dbt-core/pull/5272))
- Fix pip upgrade step in CI for Windows ([#5321](https://github.com/dbt-labs/dbt-core/issues/5321), [#5320](https://github.com/dbt-labs/dbt-core/pull/5320))
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
- Added the suggested RegEx to check the SemVer string within a package dependency and improved invalid version error handling. ([#5201](https://github.com/dbt-labs/dbt-core/issues/5201), [#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
- Bump manifest version to v6 ([#5417](https://github.com/dbt-labs/dbt-core/issues/5417), [#5430](https://github.com/dbt-labs/dbt-core/pull/5430))
- Add tests for SQL grants ([#5437](https://github.com/dbt-labs/dbt-core/issues/5437), [#5447](https://github.com/dbt-labs/dbt-core/pull/5447))
### Dependencies
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Bump mypy from 0.942 to 0.961 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5337](https://github.com/dbt-labs/dbt-core/pull/5337))
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5388](https://github.com/dbt-labs/dbt-core/pull/5388))
- Bump black from 22.3.0 to 22.6.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5420](https://github.com/dbt-labs/dbt-core/pull/5420))
### Security
- Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Contributors
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- [@b-per](https://github.com/b-per) ([#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- [@darin-reify](https://github.com/darin-reify) ([#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@drewbanin](https://github.com/drewbanin) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) ([#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- [@isidentical](https://github.com/isidentical) ([#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- [@pdebelak](https://github.com/pdebelak) ([#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- [@pquadri](https://github.com/pquadri) ([#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- [@volkangurel](https://github.com/volkangurel) ([#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
## Previous Releases ## Previous Releases
For information on prior major and minor releases, see their changelogs: For information on prior major and minor releases, see their changelogs:
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md) * [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)
* [1.0](https://github.com/dbt-labs/dbt-core/blob/1.0.latest/CHANGELOG.md) * [1.0](https://github.com/dbt-labs/dbt-core/blob/1.0.latest/CHANGELOG.md)
* [0.21](https://github.com/dbt-labs/dbt-core/blob/0.21.latest/CHANGELOG.md) * [0.21](https://github.com/dbt-labs/dbt-core/blob/0.21.latest/CHANGELOG.md)

View File

@@ -7,9 +7,7 @@
3. [Setting up an environment](#setting-up-an-environment) 3. [Setting up an environment](#setting-up-an-environment)
4. [Running `dbt` in development](#running-dbt-core-in-development) 4. [Running `dbt` in development](#running-dbt-core-in-development)
5. [Testing dbt-core](#testing) 5. [Testing dbt-core](#testing)
6. [Debugging](#debugging) 6. [Submitting a Pull Request](#submitting-a-pull-request)
7. [Adding a changelog entry](#adding-a-changelog-entry)
8. [Submitting a Pull Request](#submitting-a-pull-request)
## About this document ## About this document
@@ -23,8 +21,7 @@ If you get stuck, we're happy to help! Drop us a line in the `#dbt-core-developm
- **Adapters:** Is your issue or proposed code change related to a specific [database adapter](https://docs.getdbt.com/docs/available-adapters)? If so, please open issues, PRs, and discussions in that adapter's repository instead. The sole exception is Postgres; the `dbt-postgres` plugin lives in this repository (`dbt-core`). - **Adapters:** Is your issue or proposed code change related to a specific [database adapter](https://docs.getdbt.com/docs/available-adapters)? If so, please open issues, PRs, and discussions in that adapter's repository instead. The sole exception is Postgres; the `dbt-postgres` plugin lives in this repository (`dbt-core`).
- **CLA:** Please note that anyone contributing code to `dbt-core` must sign the [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements). If you are unable to sign the CLA, the `dbt-core` maintainers will unfortunately be unable to merge any of your Pull Requests. We welcome you to participate in discussions, open issues, and comment on existing ones. - **CLA:** Please note that anyone contributing code to `dbt-core` must sign the [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements). If you are unable to sign the CLA, the `dbt-core` maintainers will unfortunately be unable to merge any of your Pull Requests. We welcome you to participate in discussions, open issues, and comment on existing ones.
- **Branches:** All pull requests from community contributors should target the `main` branch (default). If the change is needed as a patch for a minor version of dbt that has already been released (or is already a release candidate), a maintainer will backport the changes in your PR to the relevant "latest" release branch (`1.0.latest`, `1.1.latest`, ...). If an issue fix applies to a release branch, that fix should be first committed to the development branch and then to the release branch (rarely release-branch fixes may not apply to `main`). - **Branches:** All pull requests from community contributors should target the `main` branch (default). If the change is needed as a patch for a minor version of dbt that has already been released (or is already a release candidate), a maintainer will backport the changes in your PR to the relevant "latest" release branch (`1.0.latest`, `1.1.latest`, ...)
- **Releases**: Before releasing a new minor version of Core, we prepare a series of alphas and release candidates to allow users (especially employees of dbt Labs!) to test the new version in live environments. This is an important quality assurance step, as it exposes the new code to a wide variety of complicated deployments and can surface bugs before official release. Releases are accessible via pip, homebrew, and dbt Cloud.
## Getting the code ## Getting the code
@@ -44,9 +41,7 @@ If you are not a member of the `dbt-labs` GitHub organization, you can contribut
### dbt Labs contributors ### dbt Labs contributors
If you are a member of the `dbt-labs` GitHub organization, you will have push access to the `dbt-core` repo. Rather than forking `dbt-core` to make your changes, just clone the repository, check out a new branch, and push directly to that branch. Branch names should be fixed by `CT-XXX/` where: If you are a member of the `dbt-labs` GitHub organization, you will have push access to the `dbt-core` repo. Rather than forking `dbt-core` to make your changes, just clone the repository, check out a new branch, and push directly to that branch.
* CT stands for 'core team'
* XXX stands for a JIRA ticket number
## Setting up an environment ## Setting up an environment
@@ -56,7 +51,7 @@ There are some tools that will be helpful to you in developing locally. While th
These are the tools used in `dbt-core` development and testing: These are the tools used in `dbt-core` development and testing:
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.7, 3.8, 3.9, 3.10 and 3.11 - [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.7, 3.8, 3.9, and 3.10
- [`pytest`](https://docs.pytest.org/en/latest/) to define, discover, and run tests - [`pytest`](https://docs.pytest.org/en/latest/) to define, discover, and run tests
- [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting - [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting
- [`black`](https://github.com/psf/black) for code formatting - [`black`](https://github.com/psf/black) for code formatting
@@ -96,15 +91,12 @@ brew install postgresql
### Installation ### Installation
First make sure that you set up your `virtualenv` as described in [Setting up an environment](#setting-up-an-environment). Also ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-core` (and its dependencies): First make sure that you set up your `virtualenv` as described in [Setting up an environment](#setting-up-an-environment). Also ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-core` (and its dependencies) with:
```sh ```sh
make dev make dev
``` # or
or, alternatively:
```sh
pip install -r dev-requirements.txt -r editable-requirements.txt pip install -r dev-requirements.txt -r editable-requirements.txt
pre-commit install
``` ```
When installed in this way, any changes you make to your local copy of the source code will be reflected immediately in your next `dbt` run. When installed in this way, any changes you make to your local copy of the source code will be reflected immediately in your next `dbt` run.
@@ -159,11 +151,11 @@ Check out the other targets in the Makefile to see other commonly used test
suites. suites.
#### `pre-commit` #### `pre-commit`
[`pre-commit`](https://pre-commit.com) takes care of running all code-checks for formatting and linting. Run `make dev` to install `pre-commit` in your local environment (we recommend running this command with a python virtual environment active). This command installs several pip executables including black, mypy, and flake8. Once this is done you can use any of the linter-based make targets as well as a git pre-commit hook that will ensure proper formatting and linting. [`pre-commit`](https://pre-commit.com) takes care of running all code-checks for formatting and linting. Run `make dev` to install `pre-commit` in your local environment. Once this is done you can use any of the linter-based make targets as well as a git pre-commit hook that will ensure proper formatting and linting.
#### `tox` #### `tox`
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.7, Python 3.8, Python 3.9, Python 3.10 and Python 3.11 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration for these tests in located in `tox.ini`. [`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.7, Python 3.8, Python 3.9, and Python 3.10 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration for these tests in located in `tox.ini`.
#### `pytest` #### `pytest`
@@ -182,52 +174,20 @@ python3 -m pytest tests/functional/sources
> See [pytest usage docs](https://docs.pytest.org/en/6.2.x/usage.html) for an overview of useful command-line options. > See [pytest usage docs](https://docs.pytest.org/en/6.2.x/usage.html) for an overview of useful command-line options.
### Unit, Integration, Functional? ## Adding CHANGELOG Entry
Here are some general rules for adding tests:
* unit tests (`test/unit` & `tests/unit`) dont need to access a database; "pure Python" tests should be written as unit tests
* functional tests (`test/integration` & `tests/functional`) cover anything that interacts with a database, namely adapter
* *everything in* `test/*` *is being steadily migrated to* `tests/*`
## Debugging
1. The logs for a `dbt run` have stack traces and other information for debugging errors (in `logs/dbt.log` in your project directory).
2. Try using a debugger, like `ipdb`. For pytest: `--pdb --pdbcls=IPython.terminal.debugger:pdb`
3. Sometimes, its easier to debug on a single thread: `dbt --single-threaded run`
4. To make print statements from Jinja macros: `{{ log(msg, info=true) }}`
5. You can also add `{{ debug() }}` statements, which will drop you into some auto-generated code that the macro wrote.
6. The dbt “artifacts” are written out to the target directory of your dbt project. They are in unformatted json, which can be hard to read. Format them with:
> python -m json.tool target/run_results.json > run_results.json
### Assorted development tips
* Append `# type: ignore` to the end of a line if you need to disable `mypy` on that line.
* Sometimes flake8 complains about lines that are actually fine, in which case you can put a comment on the line such as: # noqa or # noqa: ANNN, where ANNN is the error code that flake8 issues.
* To collect output for `CProfile`, run dbt with the `-r` option and the name of an output file, i.e. `dbt -r dbt.cprof run`. If you just want to profile parsing, you can do: `dbt -r dbt.cprof parse`. `pip` install `snakeviz` to view the output. Run `snakeviz dbt.cprof` and output will be rendered in a browser window.
## Adding or modifying a CHANGELOG Entry
We use [changie](https://changie.dev) to generate `CHANGELOG` entries. **Note:** Do not edit the `CHANGELOG.md` directly. Your modifications will be lost. We use [changie](https://changie.dev) to generate `CHANGELOG` entries. **Note:** Do not edit the `CHANGELOG.md` directly. Your modifications will be lost.
Follow the steps to [install `changie`](https://changie.dev/guide/installation/) for your system. Follow the steps to [install `changie`](https://changie.dev/guide/installation/) for your system.
Once changie is installed and your PR is created for a new feature, simply run the following command and changie will walk you through the process of creating a changelog entry: Once changie is installed and your PR is created, simply run `changie new` and changie will walk you through the process of creating a changelog entry. Commit the file that's created and your changelog entry is complete!
```shell
changie new
```
Commit the file that's created and your changelog entry is complete!
If you are contributing to a feature already in progress, you will modify the changie yaml file in dbt/.changes/unreleased/ related to your change. If you need help finding this file, please ask within the discussion for the pull request!
You don't need to worry about which `dbt-core` version your change will go into. Just create the changelog entry with `changie`, and open your PR against the `main` branch. All merged changes will be included in the next minor version of `dbt-core`. The Core maintainers _may_ choose to "backport" specific changes in order to patch older minor versions. In that case, a maintainer will take care of that backport after merging your PR, before releasing the new version of `dbt-core`. You don't need to worry about which `dbt-core` version your change will go into. Just create the changelog entry with `changie`, and open your PR against the `main` branch. All merged changes will be included in the next minor version of `dbt-core`. The Core maintainers _may_ choose to "backport" specific changes in order to patch older minor versions. In that case, a maintainer will take care of that backport after merging your PR, before releasing the new version of `dbt-core`.
## Submitting a Pull Request ## Submitting a Pull Request
Code can be merged into the current development branch `main` by opening a pull request. A `dbt-core` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. A `dbt-core` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code.
Automated tests run via GitHub Actions. If you're a first-time contributor, all tests (including code checks and unit tests) will require a maintainer to approve. Changes in the `dbt-core` repository trigger integration tests against Postgres. dbt Labs also provides CI environments in which to test changes to other adapters, triggered by PRs in those adapters' repositories, as well as periodic maintenance checks of each adapter in concert with the latest `dbt-core` code changes. Automated tests run via GitHub Actions. If you're a first-time contributor, all tests (including code checks and unit tests) will require a maintainer to approve. Changes in the `dbt-core` repository trigger integration tests against Postgres. dbt Labs also provides CI environments in which to test changes to other adapters, triggered by PRs in those adapters' repositories, as well as periodic maintenance checks of each adapter in concert with the latest `dbt-core` code changes.
Once all tests are passing and your PR has been approved, a `dbt-core` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: Once all tests are passing and your PR has been approved, a `dbt-core` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada:
Sometimes, the content license agreement auto-check bot doesn't find a user's entry in its roster. If you need to force a rerun, add `@cla-bot check` in a comment on the pull request.

View File

@@ -49,9 +49,6 @@ RUN apt-get update \
python3.10 \ python3.10 \
python3.10-dev \ python3.10-dev \
python3.10-venv \ python3.10-venv \
python3.11 \
python3.11-dev \
python3.11-venv \
&& apt-get clean \ && apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

View File

@@ -6,29 +6,11 @@ ifeq ($(USE_DOCKER),true)
DOCKER_CMD := docker-compose run --rm test DOCKER_CMD := docker-compose run --rm test
endif endif
LOGS_DIR := ./logs .PHONY: dev
dev: ## Installs dbt-* packages in develop mode along with development dependencies.
# Optional flag to invoke tests using our CI env.
# But we always want these active for structured
# log testing.
CI_FLAGS =\
DBT_TEST_USER_1=dbt_test_user_1\
DBT_TEST_USER_2=dbt_test_user_2\
DBT_TEST_USER_3=dbt_test_user_3\
RUSTFLAGS="-D warnings"\
LOG_DIR=./logs\
DBT_LOG_FORMAT=json
.PHONY: dev_req
dev_req: ## Installs dbt-* packages in develop mode along with only development dependencies.
@\ @\
pip install -r dev-requirements.txt -r editable-requirements.txt pip install -r dev-requirements.txt -r editable-requirements.txt
.PHONY: dev
dev: dev_req ## Installs dbt-* packages in develop mode along with development dependencies and pre-commit.
@\
pre-commit install
.PHONY: mypy .PHONY: mypy
mypy: .env ## Runs mypy against staged changes for static type checking. mypy: .env ## Runs mypy against staged changes for static type checking.
@\ @\
@@ -66,20 +48,13 @@ test: .env ## Runs unit tests with py and code checks against staged changes.
.PHONY: integration .PHONY: integration
integration: .env ## Runs postgres integration tests with py-integration integration: .env ## Runs postgres integration tests with py-integration
@\ @\
$(if $(USE_CI_FLAGS), $(CI_FLAGS)) $(DOCKER_CMD) tox -e py-integration -- -nauto $(DOCKER_CMD) tox -e py-integration -- -nauto
.PHONY: integration-fail-fast .PHONY: integration-fail-fast
integration-fail-fast: .env ## Runs postgres integration tests with py-integration in "fail fast" mode. integration-fail-fast: .env ## Runs postgres integration tests with py-integration in "fail fast" mode.
@\ @\
$(DOCKER_CMD) tox -e py-integration -- -x -nauto $(DOCKER_CMD) tox -e py-integration -- -x -nauto
.PHONY: interop
interop: clean
@\
mkdir $(LOGS_DIR) && \
$(CI_FLAGS) $(DOCKER_CMD) tox -e py-integration -- -nauto && \
LOG_DIR=$(LOGS_DIR) cargo run --manifest-path test/interop/log_parsing/Cargo.toml
.PHONY: setup-db .PHONY: setup-db
setup-db: ## Setup Postgres database with docker-compose for system testing. setup-db: ## Setup Postgres database with docker-compose for system testing.
@\ @\
@@ -101,7 +76,6 @@ endif
clean: ## Resets development environment. clean: ## Resets development environment.
@echo 'cleaning repo...' @echo 'cleaning repo...'
@rm -f .coverage @rm -f .coverage
@rm -f .coverage.*
@rm -rf .eggs/ @rm -rf .eggs/
@rm -f .env @rm -f .env
@rm -rf .tox/ @rm -rf .tox/

View File

@@ -1,2 +1 @@
recursive-include dbt/include *.py *.sql *.yml *.html *.md .gitkeep .gitignore recursive-include dbt/include *.py *.sql *.yml *.html *.md .gitkeep .gitignore
include dbt/py.typed

View File

@@ -2,59 +2,50 @@
## The following are individual files in this directory. ## The following are individual files in this directory.
### compilation.py
### constants.py
### dataclass_schema.py
### deprecations.py ### deprecations.py
### exceptions.py
### flags.py ### flags.py
### helper_types.py
### hooks.py
### lib.py
### links.py
### logger.py
### main.py ### main.py
### node_types.py
### profiler.py
### selected_resources.py
### semver.py
### tracking.py ### tracking.py
### version.py
### lib.py
### node_types.py
### helper_types.py
### links.py
### semver.py
### ui.py ### ui.py
### compilation.py
### dataclass_schema.py
### exceptions.py
### hooks.py
### logger.py
### profiler.py
### utils.py ### utils.py
### version.py
## The subdirectories will be documented in a README in the subdirectory ## The subdirectories will be documented in a README in the subdirectory
* adapters
* cli
* clients
* config * config
* context
* contracts
* deps
* docs
* events
* graph
* include * include
* parser * adapters
* context
* deps
* graph
* task * task
* tests * clients
* events

View File

@@ -1,10 +0,0 @@
## Base adapters
### impl.py
The class `SQLAdapter` in [base/imply.py](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/adapters/base/impl.py) is a (mostly) abstract object that adapter objects inherit from. The base class scaffolds out methods that every adapter project usually should implement for smooth communication between dbt and database.
Some target databases require more or fewer methods--it all depends on what the warehouse's featureset is.
Look into the class for function-level comments.

View File

@@ -10,5 +10,5 @@ from dbt.adapters.base.relation import ( # noqa
SchemaSearchMap, SchemaSearchMap,
) )
from dbt.adapters.base.column import Column # noqa from dbt.adapters.base.column import Column # noqa
from dbt.adapters.base.impl import AdapterConfig, BaseAdapter, PythonJobHelper # noqa from dbt.adapters.base.impl import AdapterConfig, BaseAdapter # noqa
from dbt.adapters.base.plugin import AdapterPlugin # noqa from dbt.adapters.base.plugin import AdapterPlugin # noqa

View File

@@ -2,7 +2,7 @@ from dataclasses import dataclass
import re import re
from typing import Dict, ClassVar, Any, Optional from typing import Dict, ClassVar, Any, Optional
from dbt.exceptions import DbtRuntimeError from dbt.exceptions import RuntimeException
@dataclass @dataclass
@@ -12,7 +12,6 @@ class Column:
"TIMESTAMP": "TIMESTAMP", "TIMESTAMP": "TIMESTAMP",
"FLOAT": "FLOAT", "FLOAT": "FLOAT",
"INTEGER": "INT", "INTEGER": "INT",
"BOOLEAN": "BOOLEAN",
} }
column: str column: str
dtype: str dtype: str
@@ -85,7 +84,7 @@ class Column:
def string_size(self) -> int: def string_size(self) -> int:
if not self.is_string(): if not self.is_string():
raise DbtRuntimeError("Called string_size() on non-string field!") raise RuntimeException("Called string_size() on non-string field!")
if self.dtype == "text" or self.char_size is None: if self.dtype == "text" or self.char_size is None:
# char_size should never be None. Handle it reasonably just in case # char_size should never be None. Handle it reasonably just in case
@@ -124,7 +123,7 @@ class Column:
def from_description(cls, name: str, raw_data_type: str) -> "Column": def from_description(cls, name: str, raw_data_type: str) -> "Column":
match = re.match(r"([^(]+)(\([^)]+\))?", raw_data_type) match = re.match(r"([^(]+)(\([^)]+\))?", raw_data_type)
if match is None: if match is None:
raise DbtRuntimeError(f'Could not interpret data type "{raw_data_type}"') raise RuntimeException(f'Could not interpret data type "{raw_data_type}"')
data_type, size_info = match.groups() data_type, size_info = match.groups()
char_size = None char_size = None
numeric_precision = None numeric_precision = None
@@ -137,7 +136,7 @@ class Column:
try: try:
char_size = int(parts[0]) char_size = int(parts[0])
except ValueError: except ValueError:
raise DbtRuntimeError( raise RuntimeException(
f'Could not interpret data_type "{raw_data_type}": ' f'Could not interpret data_type "{raw_data_type}": '
f'could not convert "{parts[0]}" to an integer' f'could not convert "{parts[0]}" to an integer'
) )
@@ -145,14 +144,14 @@ class Column:
try: try:
numeric_precision = int(parts[0]) numeric_precision = int(parts[0])
except ValueError: except ValueError:
raise DbtRuntimeError( raise RuntimeException(
f'Could not interpret data_type "{raw_data_type}": ' f'Could not interpret data_type "{raw_data_type}": '
f'could not convert "{parts[0]}" to an integer' f'could not convert "{parts[0]}" to an integer'
) )
try: try:
numeric_scale = int(parts[1]) numeric_scale = int(parts[1])
except ValueError: except ValueError:
raise DbtRuntimeError( raise RuntimeException(
f'Could not interpret data_type "{raw_data_type}": ' f'Could not interpret data_type "{raw_data_type}": '
f'could not convert "{parts[1]}" to an integer' f'could not convert "{parts[1]}" to an integer'
) )

View File

@@ -2,7 +2,6 @@ import abc
import os import os
from time import sleep from time import sleep
import sys import sys
import traceback
# multiprocessing.RLock is a function returning this type # multiprocessing.RLock is a function returning this type
from multiprocessing.synchronize import RLock from multiprocessing.synchronize import RLock
@@ -41,16 +40,14 @@ from dbt.events.functions import fire_event
from dbt.events.types import ( from dbt.events.types import (
NewConnection, NewConnection,
ConnectionReused, ConnectionReused,
ConnectionLeftOpenInCleanup,
ConnectionLeftOpen, ConnectionLeftOpen,
ConnectionClosedInCleanup, ConnectionLeftOpen2,
ConnectionClosed, ConnectionClosed,
ConnectionClosed2,
Rollback, Rollback,
RollbackFailed, RollbackFailed,
) )
from dbt.events.contextvars import get_node_info
from dbt import flags from dbt import flags
from dbt.utils import cast_to_str
SleepTime = Union[int, float] # As taken by time.sleep. SleepTime = Union[int, float] # As taken by time.sleep.
AdapterHandle = Any # Adapter connection handle objects can be any class. AdapterHandle = Any # Adapter connection handle objects can be any class.
@@ -91,13 +88,13 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
key = self.get_thread_identifier() key = self.get_thread_identifier()
with self.lock: with self.lock:
if key not in self.thread_connections: if key not in self.thread_connections:
raise dbt.exceptions.InvalidConnectionError(key, list(self.thread_connections)) raise dbt.exceptions.InvalidConnectionException(key, list(self.thread_connections))
return self.thread_connections[key] return self.thread_connections[key]
def set_thread_connection(self, conn: Connection) -> None: def set_thread_connection(self, conn: Connection) -> None:
key = self.get_thread_identifier() key = self.get_thread_identifier()
if key in self.thread_connections: if key in self.thread_connections:
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
"In set_thread_connection, existing connection exists for {}" "In set_thread_connection, existing connection exists for {}"
) )
self.thread_connections[key] = conn self.thread_connections[key] = conn
@@ -137,49 +134,47 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:return: A context manager that handles exceptions raised by the :return: A context manager that handles exceptions raised by the
underlying database. underlying database.
""" """
raise dbt.exceptions.NotImplementedError( raise dbt.exceptions.NotImplementedException(
"`exception_handler` is not implemented for this adapter!" "`exception_handler` is not implemented for this adapter!"
) )
def set_connection_name(self, name: Optional[str] = None) -> Connection: def set_connection_name(self, name: Optional[str] = None) -> Connection:
"""Called by 'acquire_connection' in BaseAdapter, which is called by conn_name: str
'connection_named', called by 'connection_for(node)'. if name is None:
Creates a connection for this thread if one doesn't already # if a name isn't specified, we'll re-use a single handle
exist, and will rename an existing connection.""" # named 'master'
conn_name = "master"
else:
if not isinstance(name, str):
raise dbt.exceptions.CompilerException(
f"For connection name, got {name} - not a string!"
)
assert isinstance(name, str)
conn_name = name
conn_name: str = "master" if name is None else name
# Get a connection for this thread
conn = self.get_if_exists() conn = self.get_if_exists()
if conn and conn.name == conn_name and conn.state == "open":
# Found a connection and nothing to do, so just return it
return conn
if conn is None: if conn is None:
# Create a new connection
conn = Connection( conn = Connection(
type=Identifier(self.TYPE), type=Identifier(self.TYPE),
name=conn_name, name=None,
state=ConnectionState.INIT, state=ConnectionState.INIT,
transaction_open=False, transaction_open=False,
handle=None, handle=None,
credentials=self.profile.credentials, credentials=self.profile.credentials,
) )
conn.handle = LazyHandle(self.open)
# Add the connection to thread_connections for this thread
self.set_thread_connection(conn) self.set_thread_connection(conn)
fire_event(
NewConnection(conn_name=conn_name, conn_type=self.TYPE, node_info=get_node_info())
)
else: # existing connection either wasn't open or didn't have the right name
if conn.state != "open":
conn.handle = LazyHandle(self.open)
if conn.name != conn_name:
orig_conn_name: str = conn.name or ""
conn.name = conn_name
fire_event(ConnectionReused(orig_conn_name=orig_conn_name, conn_name=conn_name))
if conn.name == conn_name and conn.state == "open":
return conn
fire_event(NewConnection(conn_name=conn_name, conn_type=self.TYPE))
if conn.state == "open":
fire_event(ConnectionReused(conn_name=conn_name))
else:
conn.handle = LazyHandle(self.open)
conn.name = conn_name
return conn return conn
@classmethod @classmethod
@@ -211,7 +206,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
connect should trigger a retry. connect should trigger a retry.
:type retryable_exceptions: Iterable[Type[Exception]] :type retryable_exceptions: Iterable[Type[Exception]]
:param int retry_limit: How many times to retry the call to connect. If this limit :param int retry_limit: How many times to retry the call to connect. If this limit
is exceeded before a successful call, a FailedToConnectError will be raised. is exceeded before a successful call, a FailedToConnectException will be raised.
Must be non-negative. Must be non-negative.
:param retry_timeout: Time to wait between attempts to connect. Can also take a :param retry_timeout: Time to wait between attempts to connect. Can also take a
Callable that takes the number of attempts so far, beginning at 0, and returns an int Callable that takes the number of attempts so far, beginning at 0, and returns an int
@@ -220,14 +215,14 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:param int _attempts: Parameter used to keep track of the number of attempts in calling the :param int _attempts: Parameter used to keep track of the number of attempts in calling the
connect function across recursive calls. Passed as an argument to retry_timeout if it connect function across recursive calls. Passed as an argument to retry_timeout if it
is a Callable. This parameter should not be set by the initial caller. is a Callable. This parameter should not be set by the initial caller.
:raises dbt.exceptions.FailedToConnectError: Upon exhausting all retry attempts without :raises dbt.exceptions.FailedToConnectException: Upon exhausting all retry attempts without
successfully acquiring a handle. successfully acquiring a handle.
:return: The given connection with its appropriate state and handle attributes set :return: The given connection with its appropriate state and handle attributes set
depending on whether we successfully acquired a handle or not. depending on whether we successfully acquired a handle or not.
""" """
timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout
if timeout < 0: if timeout < 0:
raise dbt.exceptions.FailedToConnectError( raise dbt.exceptions.FailedToConnectException(
"retry_timeout cannot be negative or return a negative time." "retry_timeout cannot be negative or return a negative time."
) )
@@ -235,7 +230,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
# This guard is not perfect others may add to the recursion limit (e.g. built-ins). # This guard is not perfect others may add to the recursion limit (e.g. built-ins).
connection.handle = None connection.handle = None
connection.state = ConnectionState.FAIL connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError("retry_limit cannot be negative") raise dbt.exceptions.FailedToConnectException("retry_limit cannot be negative")
try: try:
connection.handle = connect() connection.handle = connect()
@@ -246,7 +241,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
if retry_limit <= 0: if retry_limit <= 0:
connection.handle = None connection.handle = None
connection.state = ConnectionState.FAIL connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError(str(e)) raise dbt.exceptions.FailedToConnectException(str(e))
logger.debug( logger.debug(
f"Got a retryable error when attempting to open a {cls.TYPE} connection.\n" f"Got a retryable error when attempting to open a {cls.TYPE} connection.\n"
@@ -268,12 +263,12 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
except Exception as e: except Exception as e:
connection.handle = None connection.handle = None
connection.state = ConnectionState.FAIL connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError(str(e)) raise dbt.exceptions.FailedToConnectException(str(e))
@abc.abstractmethod @abc.abstractmethod
def cancel_open(self) -> Optional[List[str]]: def cancel_open(self) -> Optional[List[str]]:
"""Cancel all open connections on the adapter. (passable)""" """Cancel all open connections on the adapter. (passable)"""
raise dbt.exceptions.NotImplementedError( raise dbt.exceptions.NotImplementedException(
"`cancel_open` is not implemented for this adapter!" "`cancel_open` is not implemented for this adapter!"
) )
@@ -288,7 +283,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
This should be thread-safe, or hold the lock if necessary. The given This should be thread-safe, or hold the lock if necessary. The given
connection should not be in either in_use or available. connection should not be in either in_use or available.
""" """
raise dbt.exceptions.NotImplementedError("`open` is not implemented for this adapter!") raise dbt.exceptions.NotImplementedException("`open` is not implemented for this adapter!")
def release(self) -> None: def release(self) -> None:
with self.lock: with self.lock:
@@ -309,9 +304,9 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
with self.lock: with self.lock:
for connection in self.thread_connections.values(): for connection in self.thread_connections.values():
if connection.state not in {"closed", "init"}: if connection.state not in {"closed", "init"}:
fire_event(ConnectionLeftOpenInCleanup(conn_name=cast_to_str(connection.name))) fire_event(ConnectionLeftOpen(conn_name=connection.name))
else: else:
fire_event(ConnectionClosedInCleanup(conn_name=cast_to_str(connection.name))) fire_event(ConnectionClosed(conn_name=connection.name))
self.close(connection) self.close(connection)
# garbage collect these connections # garbage collect these connections
@@ -320,12 +315,16 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
@abc.abstractmethod @abc.abstractmethod
def begin(self) -> None: def begin(self) -> None:
"""Begin a transaction. (passable)""" """Begin a transaction. (passable)"""
raise dbt.exceptions.NotImplementedError("`begin` is not implemented for this adapter!") raise dbt.exceptions.NotImplementedException(
"`begin` is not implemented for this adapter!"
)
@abc.abstractmethod @abc.abstractmethod
def commit(self) -> None: def commit(self) -> None:
"""Commit a transaction. (passable)""" """Commit a transaction. (passable)"""
raise dbt.exceptions.NotImplementedError("`commit` is not implemented for this adapter!") raise dbt.exceptions.NotImplementedException(
"`commit` is not implemented for this adapter!"
)
@classmethod @classmethod
def _rollback_handle(cls, connection: Connection) -> None: def _rollback_handle(cls, connection: Connection) -> None:
@@ -333,40 +332,28 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
try: try:
connection.handle.rollback() connection.handle.rollback()
except Exception: except Exception:
fire_event( fire_event(RollbackFailed(conn_name=connection.name))
RollbackFailed(
conn_name=cast_to_str(connection.name),
exc_info=traceback.format_exc(),
node_info=get_node_info(),
)
)
@classmethod @classmethod
def _close_handle(cls, connection: Connection) -> None: def _close_handle(cls, connection: Connection) -> None:
"""Perform the actual close operation.""" """Perform the actual close operation."""
# On windows, sometimes connection handles don't have a close() attr. # On windows, sometimes connection handles don't have a close() attr.
if hasattr(connection.handle, "close"): if hasattr(connection.handle, "close"):
fire_event( fire_event(ConnectionClosed2(conn_name=connection.name))
ConnectionClosed(conn_name=cast_to_str(connection.name), node_info=get_node_info())
)
connection.handle.close() connection.handle.close()
else: else:
fire_event( fire_event(ConnectionLeftOpen2(conn_name=connection.name))
ConnectionLeftOpen(
conn_name=cast_to_str(connection.name), node_info=get_node_info()
)
)
@classmethod @classmethod
def _rollback(cls, connection: Connection) -> None: def _rollback(cls, connection: Connection) -> None:
"""Roll back the given connection.""" """Roll back the given connection."""
if connection.transaction_open is False: if connection.transaction_open is False:
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
f"Tried to rollback transaction on connection " f"Tried to rollback transaction on connection "
f'"{connection.name}", but it does not have one open!' f'"{connection.name}", but it does not have one open!'
) )
fire_event(Rollback(conn_name=cast_to_str(connection.name), node_info=get_node_info())) fire_event(Rollback(conn_name=connection.name))
cls._rollback_handle(connection) cls._rollback_handle(connection)
connection.transaction_open = False connection.transaction_open = False
@@ -378,7 +365,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
return connection return connection
if connection.transaction_open and connection.handle: if connection.transaction_open and connection.handle:
fire_event(Rollback(conn_name=cast_to_str(connection.name), node_info=get_node_info())) fire_event(Rollback(conn_name=connection.name))
cls._rollback_handle(connection) cls._rollback_handle(connection)
connection.transaction_open = False connection.transaction_open = False
@@ -411,4 +398,6 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:return: A tuple of the query status and results (empty if fetch=False). :return: A tuple of the query status and results (empty if fetch=False).
:rtype: Tuple[AdapterResponse, agate.Table] :rtype: Tuple[AdapterResponse, agate.Table]
""" """
raise dbt.exceptions.NotImplementedError("`execute` is not implemented for this adapter!") raise dbt.exceptions.NotImplementedException(
"`execute` is not implemented for this adapter!"
)

View File

@@ -2,7 +2,6 @@ import abc
from concurrent.futures import as_completed, Future from concurrent.futures import as_completed, Future
from contextlib import contextmanager from contextlib import contextmanager
from datetime import datetime from datetime import datetime
import time
from itertools import chain from itertools import chain
from typing import ( from typing import (
Optional, Optional,
@@ -15,6 +14,7 @@ from typing import (
List, List,
Mapping, Mapping,
Iterator, Iterator,
Union,
Set, Set,
) )
@@ -22,20 +22,13 @@ import agate
import pytz import pytz
from dbt.exceptions import ( from dbt.exceptions import (
DbtInternalError, raise_database_error,
MacroArgTypeError, raise_compiler_error,
MacroResultError, invalid_type_error,
QuoteConfigTypeError, get_relation_returned_multiple_results,
NotImplementedError, InternalException,
NullRelationCacheAttemptedError, NotImplementedException,
NullRelationDropAttemptedError, RuntimeException,
RelationReturnedMultipleResultsError,
RenameToNoneAttemptedError,
DbtRuntimeError,
SnapshotTargetIncompleteError,
SnapshotTargetNotSnapshotTableError,
UnexpectedNullError,
UnexpectedNonTimestampError,
) )
from dbt.adapters.protocol import ( from dbt.adapters.protocol import (
@@ -44,17 +37,13 @@ from dbt.adapters.protocol import (
) )
from dbt.clients.agate_helper import empty_table, merge_tables, table_from_rows from dbt.clients.agate_helper import empty_table, merge_tables, table_from_rows
from dbt.clients.jinja import MacroGenerator from dbt.clients.jinja import MacroGenerator
from dbt.contracts.graph.compiled import CompileResultNode, CompiledSeedNode
from dbt.contracts.graph.manifest import Manifest, MacroManifest from dbt.contracts.graph.manifest import Manifest, MacroManifest
from dbt.contracts.graph.nodes import ResultNode from dbt.contracts.graph.parsed import ParsedSeedNode
from dbt.events.functions import fire_event, warn_or_error from dbt.exceptions import warn_or_error
from dbt.events.types import ( from dbt.events.functions import fire_event
CacheMiss, from dbt.events.types import CacheMiss, ListRelations
ListRelations, from dbt.utils import filter_null_values, executor
CodeExecution,
CodeExecutionStatus,
CatalogGenerationError,
)
from dbt.utils import filter_null_values, executor, cast_to_str
from dbt.adapters.base.connections import Connection, AdapterResponse from dbt.adapters.base.connections import Connection, AdapterResponse
from dbt.adapters.base.meta import AdapterMeta, available from dbt.adapters.base.meta import AdapterMeta, available
@@ -65,8 +54,10 @@ from dbt.adapters.base.relation import (
SchemaSearchMap, SchemaSearchMap,
) )
from dbt.adapters.base import Column as BaseColumn from dbt.adapters.base import Column as BaseColumn
from dbt.adapters.base import Credentials from dbt.adapters.cache import RelationsCache, _make_key
from dbt.adapters.cache import RelationsCache, _make_ref_key_msg
SeedModel = Union[ParsedSeedNode, CompiledSeedNode]
GET_CATALOG_MACRO_NAME = "get_catalog" GET_CATALOG_MACRO_NAME = "get_catalog"
@@ -75,7 +66,7 @@ FRESHNESS_MACRO_NAME = "collect_freshness"
def _expect_row_value(key: str, row: agate.Row): def _expect_row_value(key: str, row: agate.Row):
if key not in row.keys(): if key not in row.keys():
raise DbtInternalError( raise InternalException(
'Got a row without "{}" column, columns: {}'.format(key, row.keys()) 'Got a row without "{}" column, columns: {}'.format(key, row.keys())
) )
return row[key] return row[key]
@@ -104,10 +95,18 @@ def _utc(dt: Optional[datetime], source: BaseRelation, field_name: str) -> datet
assume the datetime is already for UTC and add the timezone. assume the datetime is already for UTC and add the timezone.
""" """
if dt is None: if dt is None:
raise UnexpectedNullError(field_name, source) raise raise_database_error(
"Expected a non-null value when querying field '{}' of table "
" {} but received value 'null' instead".format(field_name, source)
)
elif not hasattr(dt, "tzinfo"): elif not hasattr(dt, "tzinfo"):
raise UnexpectedNonTimestampError(field_name, source, dt) raise raise_database_error(
"Expected a timestamp value when querying field '{}' of table "
"{} but received value of type '{}' instead".format(
field_name, source, type(dt).__name__
)
)
elif dt.tzinfo: elif dt.tzinfo:
return dt.astimezone(pytz.UTC) return dt.astimezone(pytz.UTC)
@@ -122,35 +121,6 @@ def _relation_name(rel: Optional[BaseRelation]) -> str:
return str(rel) return str(rel)
def log_code_execution(code_execution_function):
# decorator to log code and execution time
if code_execution_function.__name__ != "submit_python_job":
raise ValueError("this should be only used to log submit_python_job now")
def execution_with_log(*args):
self = args[0]
connection_name = self.connections.get_thread_connection().name
fire_event(CodeExecution(conn_name=connection_name, code_content=args[2]))
start_time = time.time()
response = code_execution_function(*args)
fire_event(
CodeExecutionStatus(
status=response._message, elapsed=round((time.time() - start_time), 2)
)
)
return response
return execution_with_log
class PythonJobHelper:
def __init__(self, parsed_model: Dict, credential: Credentials) -> None:
raise NotImplementedError("PythonJobHelper is not implemented yet")
def submit(self, compiled_code: str) -> Any:
raise NotImplementedError("PythonJobHelper submit function is not implemented yet")
class BaseAdapter(metaclass=AdapterMeta): class BaseAdapter(metaclass=AdapterMeta):
"""The BaseAdapter provides an abstract base class for adapters. """The BaseAdapter provides an abstract base class for adapters.
@@ -237,7 +207,9 @@ class BaseAdapter(metaclass=AdapterMeta):
return conn.name return conn.name
@contextmanager @contextmanager
def connection_named(self, name: str, node: Optional[ResultNode] = None) -> Iterator[None]: def connection_named(
self, name: str, node: Optional[CompileResultNode] = None
) -> Iterator[None]:
try: try:
if self.connections.query_header is not None: if self.connections.query_header is not None:
self.connections.query_header.set(name, node) self.connections.query_header.set(name, node)
@@ -249,7 +221,7 @@ class BaseAdapter(metaclass=AdapterMeta):
self.connections.query_header.reset() self.connections.query_header.reset()
@contextmanager @contextmanager
def connection_for(self, node: ResultNode) -> Iterator[None]: def connection_for(self, node: CompileResultNode) -> Iterator[None]:
with self.connection_named(node.unique_id, node): with self.connection_named(node.unique_id, node):
yield yield
@@ -312,9 +284,7 @@ class BaseAdapter(metaclass=AdapterMeta):
from dbt.parser.manifest import ManifestLoader from dbt.parser.manifest import ManifestLoader
manifest = ManifestLoader.load_macros( manifest = ManifestLoader.load_macros(
self.config, self.config, self.connections.set_query_header, base_macros_only=base_macros_only
self.connections.set_query_header,
base_macros_only=base_macros_only,
) )
# TODO CT-211 # TODO CT-211
self._macro_manifest_lazy = manifest # type: ignore[assignment] self._macro_manifest_lazy = manifest # type: ignore[assignment]
@@ -333,11 +303,7 @@ class BaseAdapter(metaclass=AdapterMeta):
if (database, schema) not in self.cache: if (database, schema) not in self.cache:
fire_event( fire_event(
CacheMiss( CacheMiss(conn_name=self.nice_connection_name(), database=database, schema=schema)
conn_name=self.nice_connection_name(),
database=cast_to_str(database),
schema=schema,
)
) )
return False return False
else: else:
@@ -364,7 +330,7 @@ class BaseAdapter(metaclass=AdapterMeta):
lowercase strings. lowercase strings.
""" """
info_schema_name_map = SchemaSearchMap() info_schema_name_map = SchemaSearchMap()
nodes: Iterator[ResultNode] = chain( nodes: Iterator[CompileResultNode] = chain(
[ [
node node
for node in manifest.nodes.values() for node in manifest.nodes.values()
@@ -415,10 +381,7 @@ class BaseAdapter(metaclass=AdapterMeta):
self.cache.update_schemas(cache_update) self.cache.update_schemas(cache_update)
def set_relations_cache( def set_relations_cache(
self, self, manifest: Manifest, clear: bool = False, required_schemas: Set[BaseRelation] = None
manifest: Manifest,
clear: bool = False,
required_schemas: Set[BaseRelation] = None,
) -> None: ) -> None:
"""Run a query that gets a populated cache of the relations in the """Run a query that gets a populated cache of the relations in the
database and set the cache on this adapter. database and set the cache on this adapter.
@@ -433,7 +396,7 @@ class BaseAdapter(metaclass=AdapterMeta):
"""Cache a new relation in dbt. It will show up in `list relations`.""" """Cache a new relation in dbt. It will show up in `list relations`."""
if relation is None: if relation is None:
name = self.nice_connection_name() name = self.nice_connection_name()
raise NullRelationCacheAttemptedError(name) raise_compiler_error("Attempted to cache a null relation for {}".format(name))
self.cache.add(relation) self.cache.add(relation)
# so jinja doesn't render things # so jinja doesn't render things
return "" return ""
@@ -445,7 +408,7 @@ class BaseAdapter(metaclass=AdapterMeta):
""" """
if relation is None: if relation is None:
name = self.nice_connection_name() name = self.nice_connection_name()
raise NullRelationDropAttemptedError(name) raise_compiler_error("Attempted to drop a null relation for {}".format(name))
self.cache.drop(relation) self.cache.drop(relation)
return "" return ""
@@ -462,7 +425,9 @@ class BaseAdapter(metaclass=AdapterMeta):
name = self.nice_connection_name() name = self.nice_connection_name()
src_name = _relation_name(from_relation) src_name = _relation_name(from_relation)
dst_name = _relation_name(to_relation) dst_name = _relation_name(to_relation)
raise RenameToNoneAttemptedError(src_name, dst_name, name) raise_compiler_error(
"Attempted to rename {} to {} for {}".format(src_name, dst_name, name)
)
self.cache.rename(from_relation, to_relation) self.cache.rename(from_relation, to_relation)
return "" return ""
@@ -474,12 +439,12 @@ class BaseAdapter(metaclass=AdapterMeta):
@abc.abstractmethod @abc.abstractmethod
def date_function(cls) -> str: def date_function(cls) -> str:
"""Get the date function used by this adapter's database.""" """Get the date function used by this adapter's database."""
raise NotImplementedError("`date_function` is not implemented for this adapter!") raise NotImplementedException("`date_function` is not implemented for this adapter!")
@classmethod @classmethod
@abc.abstractmethod @abc.abstractmethod
def is_cancelable(cls) -> bool: def is_cancelable(cls) -> bool:
raise NotImplementedError("`is_cancelable` is not implemented for this adapter!") raise NotImplementedException("`is_cancelable` is not implemented for this adapter!")
### ###
# Abstract methods about schemas # Abstract methods about schemas
@@ -487,7 +452,7 @@ class BaseAdapter(metaclass=AdapterMeta):
@abc.abstractmethod @abc.abstractmethod
def list_schemas(self, database: str) -> List[str]: def list_schemas(self, database: str) -> List[str]:
"""Get a list of existing schemas in database""" """Get a list of existing schemas in database"""
raise NotImplementedError("`list_schemas` is not implemented for this adapter!") raise NotImplementedException("`list_schemas` is not implemented for this adapter!")
@available.parse(lambda *a, **k: False) @available.parse(lambda *a, **k: False)
def check_schema_exists(self, database: str, schema: str) -> bool: def check_schema_exists(self, database: str, schema: str) -> bool:
@@ -510,13 +475,13 @@ class BaseAdapter(metaclass=AdapterMeta):
*Implementors must call self.cache.drop() to preserve cache state!* *Implementors must call self.cache.drop() to preserve cache state!*
""" """
raise NotImplementedError("`drop_relation` is not implemented for this adapter!") raise NotImplementedException("`drop_relation` is not implemented for this adapter!")
@abc.abstractmethod @abc.abstractmethod
@available.parse_none @available.parse_none
def truncate_relation(self, relation: BaseRelation) -> None: def truncate_relation(self, relation: BaseRelation) -> None:
"""Truncate the given relation.""" """Truncate the given relation."""
raise NotImplementedError("`truncate_relation` is not implemented for this adapter!") raise NotImplementedException("`truncate_relation` is not implemented for this adapter!")
@abc.abstractmethod @abc.abstractmethod
@available.parse_none @available.parse_none
@@ -525,13 +490,15 @@ class BaseAdapter(metaclass=AdapterMeta):
Implementors must call self.cache.rename() to preserve cache state. Implementors must call self.cache.rename() to preserve cache state.
""" """
raise NotImplementedError("`rename_relation` is not implemented for this adapter!") raise NotImplementedException("`rename_relation` is not implemented for this adapter!")
@abc.abstractmethod @abc.abstractmethod
@available.parse_list @available.parse_list
def get_columns_in_relation(self, relation: BaseRelation) -> List[BaseColumn]: def get_columns_in_relation(self, relation: BaseRelation) -> List[BaseColumn]:
"""Get a list of the columns in the given Relation.""" """Get a list of the columns in the given Relation."""
raise NotImplementedError("`get_columns_in_relation` is not implemented for this adapter!") raise NotImplementedException(
"`get_columns_in_relation` is not implemented for this adapter!"
)
@available.deprecated("get_columns_in_relation", lambda *a, **k: []) @available.deprecated("get_columns_in_relation", lambda *a, **k: [])
def get_columns_in_table(self, schema: str, identifier: str) -> List[BaseColumn]: def get_columns_in_table(self, schema: str, identifier: str) -> List[BaseColumn]:
@@ -553,7 +520,7 @@ class BaseAdapter(metaclass=AdapterMeta):
:param self.Relation current: A relation that currently exists in the :param self.Relation current: A relation that currently exists in the
database with columns of unspecified types. database with columns of unspecified types.
""" """
raise NotImplementedError( raise NotImplementedException(
"`expand_target_column_types` is not implemented for this adapter!" "`expand_target_column_types` is not implemented for this adapter!"
) )
@@ -568,8 +535,8 @@ class BaseAdapter(metaclass=AdapterMeta):
:return: The relations in schema :return: The relations in schema
:rtype: List[self.Relation] :rtype: List[self.Relation]
""" """
raise NotImplementedError( raise NotImplementedException(
"`list_relations_without_caching` is not implemented for this adapter!" "`list_relations_without_caching` is not implemented for this " "adapter!"
) )
### ###
@@ -610,7 +577,7 @@ class BaseAdapter(metaclass=AdapterMeta):
to_relation. to_relation.
""" """
if not isinstance(from_relation, self.Relation): if not isinstance(from_relation, self.Relation):
raise MacroArgTypeError( invalid_type_error(
method_name="get_missing_columns", method_name="get_missing_columns",
arg_name="from_relation", arg_name="from_relation",
got_value=from_relation, got_value=from_relation,
@@ -618,7 +585,7 @@ class BaseAdapter(metaclass=AdapterMeta):
) )
if not isinstance(to_relation, self.Relation): if not isinstance(to_relation, self.Relation):
raise MacroArgTypeError( invalid_type_error(
method_name="get_missing_columns", method_name="get_missing_columns",
arg_name="to_relation", arg_name="to_relation",
got_value=to_relation, got_value=to_relation,
@@ -639,11 +606,11 @@ class BaseAdapter(metaclass=AdapterMeta):
expected columns. expected columns.
:param Relation relation: The relation to check :param Relation relation: The relation to check
:raises InvalidMacroArgType: If the columns are :raises CompilationException: If the columns are
incorrect. incorrect.
""" """
if not isinstance(relation, self.Relation): if not isinstance(relation, self.Relation):
raise MacroArgTypeError( invalid_type_error(
method_name="valid_snapshot_target", method_name="valid_snapshot_target",
arg_name="relation", arg_name="relation",
got_value=relation, got_value=relation,
@@ -664,16 +631,24 @@ class BaseAdapter(metaclass=AdapterMeta):
if missing: if missing:
if extra: if extra:
raise SnapshotTargetIncompleteError(extra, missing) msg = (
'Snapshot target has ("{}") but not ("{}") - is it an '
"unmigrated previous version archive?".format(
'", "'.join(extra), '", "'.join(missing)
)
)
else: else:
raise SnapshotTargetNotSnapshotTableError(missing) msg = 'Snapshot target is not a snapshot table (missing "{}")'.format(
'", "'.join(missing)
)
raise_compiler_error(msg)
@available.parse_none @available.parse_none
def expand_target_column_types( def expand_target_column_types(
self, from_relation: BaseRelation, to_relation: BaseRelation self, from_relation: BaseRelation, to_relation: BaseRelation
) -> None: ) -> None:
if not isinstance(from_relation, self.Relation): if not isinstance(from_relation, self.Relation):
raise MacroArgTypeError( invalid_type_error(
method_name="expand_target_column_types", method_name="expand_target_column_types",
arg_name="from_relation", arg_name="from_relation",
got_value=from_relation, got_value=from_relation,
@@ -681,7 +656,7 @@ class BaseAdapter(metaclass=AdapterMeta):
) )
if not isinstance(to_relation, self.Relation): if not isinstance(to_relation, self.Relation):
raise MacroArgTypeError( invalid_type_error(
method_name="expand_target_column_types", method_name="expand_target_column_types",
arg_name="to_relation", arg_name="to_relation",
got_value=to_relation, got_value=to_relation,
@@ -695,10 +670,7 @@ class BaseAdapter(metaclass=AdapterMeta):
return self.cache.get_relations(database, schema) return self.cache.get_relations(database, schema)
schema_relation = self.Relation.create( schema_relation = self.Relation.create(
database=database, database=database, schema=schema, identifier="", quote_policy=self.config.quoting
schema=schema,
identifier="",
quote_policy=self.config.quoting,
).without_identifier() ).without_identifier()
# we can't build the relations cache because we don't have a # we can't build the relations cache because we don't have a
@@ -706,9 +678,7 @@ class BaseAdapter(metaclass=AdapterMeta):
relations = self.list_relations_without_caching(schema_relation) relations = self.list_relations_without_caching(schema_relation)
fire_event( fire_event(
ListRelations( ListRelations(
database=cast_to_str(database), database=database, schema=schema, relations=[_make_key(x) for x in relations]
schema=schema,
relations=[_make_ref_key_msg(x) for x in relations],
) )
) )
@@ -763,7 +733,7 @@ class BaseAdapter(metaclass=AdapterMeta):
"schema": schema, "schema": schema,
"database": database, "database": database,
} }
raise RelationReturnedMultipleResultsError(kwargs, matches) get_relation_returned_multiple_results(kwargs, matches)
elif matches: elif matches:
return matches[0] return matches[0]
@@ -785,20 +755,20 @@ class BaseAdapter(metaclass=AdapterMeta):
@available.parse_none @available.parse_none
def create_schema(self, relation: BaseRelation): def create_schema(self, relation: BaseRelation):
"""Create the given schema if it does not exist.""" """Create the given schema if it does not exist."""
raise NotImplementedError("`create_schema` is not implemented for this adapter!") raise NotImplementedException("`create_schema` is not implemented for this adapter!")
@abc.abstractmethod @abc.abstractmethod
@available.parse_none @available.parse_none
def drop_schema(self, relation: BaseRelation): def drop_schema(self, relation: BaseRelation):
"""Drop the given schema (and everything in it) if it exists.""" """Drop the given schema (and everything in it) if it exists."""
raise NotImplementedError("`drop_schema` is not implemented for this adapter!") raise NotImplementedException("`drop_schema` is not implemented for this adapter!")
@available @available
@classmethod @classmethod
@abc.abstractmethod @abc.abstractmethod
def quote(cls, identifier: str) -> str: def quote(cls, identifier: str) -> str:
"""Quote the given identifier, as appropriate for the database.""" """Quote the given identifier, as appropriate for the database."""
raise NotImplementedError("`quote` is not implemented for this adapter!") raise NotImplementedException("`quote` is not implemented for this adapter!")
@available @available
def quote_as_configured(self, identifier: str, quote_key: str) -> str: def quote_as_configured(self, identifier: str, quote_key: str) -> str:
@@ -827,7 +797,10 @@ class BaseAdapter(metaclass=AdapterMeta):
elif quote_config is None: elif quote_config is None:
pass pass
else: else:
raise QuoteConfigTypeError(quote_config) raise_compiler_error(
f'The seed configuration value of "quote_columns" has an '
f"invalid type {type(quote_config)}"
)
if quote_columns: if quote_columns:
return self.quote(column) return self.quote(column)
@@ -848,7 +821,7 @@ class BaseAdapter(metaclass=AdapterMeta):
:param col_idx: The index into the agate table for the column. :param col_idx: The index into the agate table for the column.
:return: The name of the type in the database :return: The name of the type in the database
""" """
raise NotImplementedError("`convert_text_type` is not implemented for this adapter!") raise NotImplementedException("`convert_text_type` is not implemented for this adapter!")
@classmethod @classmethod
@abc.abstractmethod @abc.abstractmethod
@@ -860,7 +833,7 @@ class BaseAdapter(metaclass=AdapterMeta):
:param col_idx: The index into the agate table for the column. :param col_idx: The index into the agate table for the column.
:return: The name of the type in the database :return: The name of the type in the database
""" """
raise NotImplementedError("`convert_number_type` is not implemented for this adapter!") raise NotImplementedException("`convert_number_type` is not implemented for this adapter!")
@classmethod @classmethod
@abc.abstractmethod @abc.abstractmethod
@@ -872,7 +845,9 @@ class BaseAdapter(metaclass=AdapterMeta):
:param col_idx: The index into the agate table for the column. :param col_idx: The index into the agate table for the column.
:return: The name of the type in the database :return: The name of the type in the database
""" """
raise NotImplementedError("`convert_boolean_type` is not implemented for this adapter!") raise NotImplementedException(
"`convert_boolean_type` is not implemented for this adapter!"
)
@classmethod @classmethod
@abc.abstractmethod @abc.abstractmethod
@@ -884,7 +859,9 @@ class BaseAdapter(metaclass=AdapterMeta):
:param col_idx: The index into the agate table for the column. :param col_idx: The index into the agate table for the column.
:return: The name of the type in the database :return: The name of the type in the database
""" """
raise NotImplementedError("`convert_datetime_type` is not implemented for this adapter!") raise NotImplementedException(
"`convert_datetime_type` is not implemented for this adapter!"
)
@classmethod @classmethod
@abc.abstractmethod @abc.abstractmethod
@@ -896,7 +873,7 @@ class BaseAdapter(metaclass=AdapterMeta):
:param col_idx: The index into the agate table for the column. :param col_idx: The index into the agate table for the column.
:return: The name of the type in the database :return: The name of the type in the database
""" """
raise NotImplementedError("`convert_date_type` is not implemented for this adapter!") raise NotImplementedException("`convert_date_type` is not implemented for this adapter!")
@classmethod @classmethod
@abc.abstractmethod @abc.abstractmethod
@@ -908,7 +885,7 @@ class BaseAdapter(metaclass=AdapterMeta):
:param col_idx: The index into the agate table for the column. :param col_idx: The index into the agate table for the column.
:return: The name of the type in the database :return: The name of the type in the database
""" """
raise NotImplementedError("`convert_time_type` is not implemented for this adapter!") raise NotImplementedException("`convert_time_type` is not implemented for this adapter!")
@available @available
@classmethod @classmethod
@@ -975,7 +952,7 @@ class BaseAdapter(metaclass=AdapterMeta):
else: else:
package_name = 'the "{}" package'.format(project) package_name = 'the "{}" package'.format(project)
raise DbtRuntimeError( raise RuntimeException(
'dbt could not find a macro with the name "{}" in {}'.format( 'dbt could not find a macro with the name "{}" in {}'.format(
macro_name, package_name macro_name, package_name
) )
@@ -1073,7 +1050,11 @@ class BaseAdapter(metaclass=AdapterMeta):
# now we have a 1-row table of the maximum `loaded_at_field` value and # now we have a 1-row table of the maximum `loaded_at_field` value and
# the current time according to the db. # the current time according to the db.
if len(table) != 1 or len(table[0]) != 2: if len(table) != 1 or len(table[0]) != 2:
raise MacroResultError(FRESHNESS_MACRO_NAME, table) raise_compiler_error(
'Got an invalid result from "{}" macro: {}'.format(
FRESHNESS_MACRO_NAME, [tuple(r) for r in table]
)
)
if table[0][0] is None: if table[0][0] is None:
# no records in the table, so really the max_loaded_at was # no records in the table, so really the max_loaded_at was
# infinitely long ago. Just call it 0:00 January 1 year UTC # infinitely long ago. Just call it 0:00 January 1 year UTC
@@ -1150,7 +1131,7 @@ class BaseAdapter(metaclass=AdapterMeta):
elif location == "prepend": elif location == "prepend":
return f"'{value}' || {add_to}" return f"'{value}' || {add_to}"
else: else:
raise DbtRuntimeError(f'Got an unexpected location value of "{location}"') raise RuntimeException(f'Got an unexpected location value of "{location}"')
def get_rows_different_sql( def get_rows_different_sql(
self, self,
@@ -1181,74 +1162,6 @@ class BaseAdapter(metaclass=AdapterMeta):
return sql return sql
@property
def python_submission_helpers(self) -> Dict[str, Type[PythonJobHelper]]:
raise NotImplementedError("python_submission_helpers is not specified")
@property
def default_python_submission_method(self) -> str:
raise NotImplementedError("default_python_submission_method is not specified")
@log_code_execution
def submit_python_job(self, parsed_model: dict, compiled_code: str) -> AdapterResponse:
submission_method = parsed_model["config"].get(
"submission_method", self.default_python_submission_method
)
if submission_method not in self.python_submission_helpers:
raise NotImplementedError(
"Submission method {} is not supported for current adapter".format(
submission_method
)
)
job_helper = self.python_submission_helpers[submission_method](
parsed_model, self.connections.profile.credentials
)
submission_result = job_helper.submit(compiled_code)
# process submission result to generate adapter response
return self.generate_python_submission_response(submission_result)
def generate_python_submission_response(self, submission_result: Any) -> AdapterResponse:
raise NotImplementedError(
"Your adapter need to implement generate_python_submission_response"
)
def valid_incremental_strategies(self):
"""The set of standard builtin strategies which this adapter supports out-of-the-box.
Not used to validate custom strategies defined by end users.
"""
return ["append"]
def builtin_incremental_strategies(self):
return ["append", "delete+insert", "merge", "insert_overwrite"]
@available.parse_none
def get_incremental_strategy_macro(self, model_context, strategy: str):
# Construct macro_name from strategy name
if strategy is None:
strategy = "default"
# validate strategies for this adapter
valid_strategies = self.valid_incremental_strategies()
valid_strategies.append("default")
builtin_strategies = self.builtin_incremental_strategies()
if strategy in builtin_strategies and strategy not in valid_strategies:
raise DbtRuntimeError(
f"The incremental strategy '{strategy}' is not valid for this adapter"
)
strategy = strategy.replace("+", "_")
macro_name = f"get_incremental_{strategy}_sql"
# The model_context should have MacroGenerator callable objects for all macros
if macro_name not in model_context:
raise DbtRuntimeError(
'dbt could not find an incremental strategy macro with the name "{}" in {}'.format(
macro_name, self.config.project_name
)
)
# This returns a callable macro
return model_context[macro_name]
COLUMNS_EQUAL_SQL = """ COLUMNS_EQUAL_SQL = """
with diff_count as ( with diff_count as (
@@ -1296,7 +1209,7 @@ def catch_as_completed(
elif isinstance(exc, KeyboardInterrupt) or not isinstance(exc, Exception): elif isinstance(exc, KeyboardInterrupt) or not isinstance(exc, Exception):
raise exc raise exc
else: else:
warn_or_error(CatalogGenerationError(exc=str(exc))) warn_or_error(f"Encountered an error while generating catalog: {str(exc)}")
# exc is not None, derives from Exception, and isn't ctrl+c # exc is not None, derives from Exception, and isn't ctrl+c
exceptions.append(exc) exceptions.append(exc)
return merge_tables(tables), exceptions return merge_tables(tables), exceptions

View File

@@ -1,7 +1,7 @@
from typing import List, Optional, Type from typing import List, Optional, Type
from dbt.adapters.base import Credentials from dbt.adapters.base import Credentials
from dbt.exceptions import CompilationError from dbt.exceptions import CompilationException
from dbt.adapters.protocol import AdapterProtocol from dbt.adapters.protocol import AdapterProtocol
@@ -11,7 +11,7 @@ def project_name_from_path(include_path: str) -> str:
partial = Project.partial_load(include_path) partial = Project.partial_load(include_path)
if partial.project_name is None: if partial.project_name is None:
raise CompilationError(f"Invalid project at {include_path}: name not set!") raise CompilationException(f"Invalid project at {include_path}: name not set!")
return partial.project_name return partial.project_name

View File

@@ -5,9 +5,9 @@ from dbt.clients.jinja import QueryStringGenerator
from dbt.context.manifest import generate_query_header_context from dbt.context.manifest import generate_query_header_context
from dbt.contracts.connection import AdapterRequiredConfig, QueryComment from dbt.contracts.connection import AdapterRequiredConfig, QueryComment
from dbt.contracts.graph.nodes import ResultNode from dbt.contracts.graph.compiled import CompileResultNode
from dbt.contracts.graph.manifest import Manifest from dbt.contracts.graph.manifest import Manifest
from dbt.exceptions import DbtRuntimeError from dbt.exceptions import RuntimeException
class NodeWrapper: class NodeWrapper:
@@ -48,7 +48,7 @@ class _QueryComment(local):
if isinstance(comment, str) and "*/" in comment: if isinstance(comment, str) and "*/" in comment:
# tell the user "no" so they don't hurt themselves by writing # tell the user "no" so they don't hurt themselves by writing
# garbage # garbage
raise DbtRuntimeError(f'query comment contains illegal value "*/": {comment}') raise RuntimeException(f'query comment contains illegal value "*/": {comment}')
self.query_comment = comment self.query_comment = comment
self.append = append self.append = append
@@ -90,7 +90,7 @@ class MacroQueryStringSetter:
def reset(self): def reset(self):
self.set("master", None) self.set("master", None)
def set(self, name: str, node: Optional[ResultNode]): def set(self, name: str, node: Optional[CompileResultNode]):
wrapped: Optional[NodeWrapper] = None wrapped: Optional[NodeWrapper] = None
if node is not None: if node is not None:
wrapped = NodeWrapper(node) wrapped = NodeWrapper(node)

View File

@@ -1,8 +1,9 @@
from collections.abc import Hashable from collections.abc import Hashable
from dataclasses import dataclass, field from dataclasses import dataclass
from typing import Optional, TypeVar, Any, Type, Dict, Iterator, Tuple, Set from typing import Optional, TypeVar, Any, Type, Dict, Union, Iterator, Tuple, Set
from dbt.contracts.graph.nodes import SourceDefinition, ManifestNode, ResultNode, ParsedNode from dbt.contracts.graph.compiled import CompiledNode
from dbt.contracts.graph.parsed import ParsedSourceDefinition, ParsedNode
from dbt.contracts.relation import ( from dbt.contracts.relation import (
RelationType, RelationType,
ComponentName, ComponentName,
@@ -11,11 +12,7 @@ from dbt.contracts.relation import (
Policy, Policy,
Path, Path,
) )
from dbt.exceptions import ( from dbt.exceptions import InternalException
ApproximateMatchError,
DbtInternalError,
MultipleDatabasesNotAllowedError,
)
from dbt.node_types import NodeType from dbt.node_types import NodeType
from dbt.utils import filter_null_values, deep_merge, classproperty from dbt.utils import filter_null_values, deep_merge, classproperty
@@ -30,10 +27,8 @@ class BaseRelation(FakeAPIObject, Hashable):
path: Path path: Path
type: Optional[RelationType] = None type: Optional[RelationType] = None
quote_character: str = '"' quote_character: str = '"'
# Python 3.11 requires that these use default_factory instead of simple default include_policy: Policy = Policy()
# ValueError: mutable default <class 'dbt.contracts.relation.Policy'> for field include_policy is not allowed: use default_factory quote_policy: Policy = Policy()
include_policy: Policy = field(default_factory=lambda: Policy())
quote_policy: Policy = field(default_factory=lambda: Policy())
dbt_created: bool = False dbt_created: bool = False
def _is_exactish_match(self, field: ComponentName, value: str) -> bool: def _is_exactish_match(self, field: ComponentName, value: str) -> bool:
@@ -44,9 +39,9 @@ class BaseRelation(FakeAPIObject, Hashable):
@classmethod @classmethod
def _get_field_named(cls, field_name): def _get_field_named(cls, field_name):
for f, _ in cls._get_fields(): for field, _ in cls._get_fields():
if f.name == field_name: if field.name == field_name:
return f return field
# this should be unreachable # this should be unreachable
raise ValueError(f"BaseRelation has no {field_name} field!") raise ValueError(f"BaseRelation has no {field_name} field!")
@@ -57,11 +52,11 @@ class BaseRelation(FakeAPIObject, Hashable):
@classmethod @classmethod
def get_default_quote_policy(cls) -> Policy: def get_default_quote_policy(cls) -> Policy:
return cls._get_field_named("quote_policy").default_factory() return cls._get_field_named("quote_policy").default
@classmethod @classmethod
def get_default_include_policy(cls) -> Policy: def get_default_include_policy(cls) -> Policy:
return cls._get_field_named("include_policy").default_factory() return cls._get_field_named("include_policy").default
def get(self, key, default=None): def get(self, key, default=None):
"""Override `.get` to return a metadata object so we don't break """Override `.get` to return a metadata object so we don't break
@@ -87,7 +82,7 @@ class BaseRelation(FakeAPIObject, Hashable):
if not search: if not search:
# nothing was passed in # nothing was passed in
raise dbt.exceptions.DbtRuntimeError( raise dbt.exceptions.RuntimeException(
"Tried to match relation, but no search path was passed!" "Tried to match relation, but no search path was passed!"
) )
@@ -104,7 +99,7 @@ class BaseRelation(FakeAPIObject, Hashable):
if approximate_match and not exact_match: if approximate_match and not exact_match:
target = self.create(database=database, schema=schema, identifier=identifier) target = self.create(database=database, schema=schema, identifier=identifier)
raise ApproximateMatchError(target, self) dbt.exceptions.approximate_relation_match(target, self)
return exact_match return exact_match
@@ -189,7 +184,7 @@ class BaseRelation(FakeAPIObject, Hashable):
) )
@classmethod @classmethod
def create_from_source(cls: Type[Self], source: SourceDefinition, **kwargs: Any) -> Self: def create_from_source(cls: Type[Self], source: ParsedSourceDefinition, **kwargs: Any) -> Self:
source_quoting = source.quoting.to_dict(omit_none=True) source_quoting = source.quoting.to_dict(omit_none=True)
source_quoting.pop("column", None) source_quoting.pop("column", None)
quote_policy = deep_merge( quote_policy = deep_merge(
@@ -214,7 +209,7 @@ class BaseRelation(FakeAPIObject, Hashable):
def create_ephemeral_from_node( def create_ephemeral_from_node(
cls: Type[Self], cls: Type[Self],
config: HasQuoting, config: HasQuoting,
node: ManifestNode, node: Union[ParsedNode, CompiledNode],
) -> Self: ) -> Self:
# Note that ephemeral models are based on the name. # Note that ephemeral models are based on the name.
identifier = cls.add_ephemeral_prefix(node.name) identifier = cls.add_ephemeral_prefix(node.name)
@@ -227,7 +222,7 @@ class BaseRelation(FakeAPIObject, Hashable):
def create_from_node( def create_from_node(
cls: Type[Self], cls: Type[Self],
config: HasQuoting, config: HasQuoting,
node: ManifestNode, node: Union[ParsedNode, CompiledNode],
quote_policy: Optional[Dict[str, bool]] = None, quote_policy: Optional[Dict[str, bool]] = None,
**kwargs: Any, **kwargs: Any,
) -> Self: ) -> Self:
@@ -248,20 +243,20 @@ class BaseRelation(FakeAPIObject, Hashable):
def create_from( def create_from(
cls: Type[Self], cls: Type[Self],
config: HasQuoting, config: HasQuoting,
node: ResultNode, node: Union[CompiledNode, ParsedNode, ParsedSourceDefinition],
**kwargs: Any, **kwargs: Any,
) -> Self: ) -> Self:
if node.resource_type == NodeType.Source: if node.resource_type == NodeType.Source:
if not isinstance(node, SourceDefinition): if not isinstance(node, ParsedSourceDefinition):
raise DbtInternalError( raise InternalException(
"type mismatch, expected SourceDefinition but got {}".format(type(node)) "type mismatch, expected ParsedSourceDefinition but got {}".format(type(node))
) )
return cls.create_from_source(node, **kwargs) return cls.create_from_source(node, **kwargs)
else: else:
# Can't use ManifestNode here because of parameterized generics if not isinstance(node, (ParsedNode, CompiledNode)):
if not isinstance(node, (ParsedNode)): raise InternalException(
raise DbtInternalError( "type mismatch, expected ParsedNode or CompiledNode but "
f"type mismatch, expected ManifestNode but got {type(node)}" "got {}".format(type(node))
) )
return cls.create_from_node(config, node, **kwargs) return cls.create_from_node(config, node, **kwargs)
@@ -358,7 +353,7 @@ class InformationSchema(BaseRelation):
def __post_init__(self): def __post_init__(self):
if not isinstance(self.information_schema_view, (type(None), str)): if not isinstance(self.information_schema_view, (type(None), str)):
raise dbt.exceptions.CompilationError( raise dbt.exceptions.CompilationException(
"Got an invalid name: {}".format(self.information_schema_view) "Got an invalid name: {}".format(self.information_schema_view)
) )
@@ -442,7 +437,7 @@ class SchemaSearchMap(Dict[InformationSchema, Set[Optional[str]]]):
if not allow_multiple_databases: if not allow_multiple_databases:
seen = {r.database.lower() for r in self if r.database} seen = {r.database.lower() for r in self if r.database}
if len(seen) > 1: if len(seen) > 1:
raise MultipleDatabasesNotAllowedError(seen) dbt.exceptions.raise_compiler_error(str(seen))
for information_schema_name, schema in self.search(): for information_schema_name, schema in self.search():
path = {"database": information_schema_name.database, "schema": schema} path = {"database": information_schema_name.database, "schema": schema}

View File

@@ -1,24 +1,28 @@
import re
import threading import threading
from copy import deepcopy from copy import deepcopy
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
from dbt.adapters.reference_keys import ( from dbt.adapters.reference_keys import _make_key, _ReferenceKey
_make_ref_key, import dbt.exceptions
_make_ref_key_msg, from dbt.events.functions import fire_event
_make_msg_from_ref_key, from dbt.events.types import (
_ReferenceKey, AddLink,
AddRelation,
DropCascade,
DropMissingRelation,
DropRelation,
DumpAfterAddGraph,
DumpAfterRenameSchema,
DumpBeforeAddGraph,
DumpBeforeRenameSchema,
RenameSchema,
TemporaryRelation,
UncachedRelation,
UpdateReference,
) )
from dbt.exceptions import (
DependentLinkNotCachedError,
NewNameAlreadyInCacheError,
NoneRelationFoundError,
ReferencedLinkNotCachedError,
TruncatedModelNameCausedCollisionError,
)
from dbt.events.functions import fire_event, fire_event_if
from dbt.events.types import CacheAction, CacheDumpGraph
import dbt.flags as flags
from dbt.utils import lowercase from dbt.utils import lowercase
from dbt.helper_types import Lazy
def dot_separated(key: _ReferenceKey) -> str: def dot_separated(key: _ReferenceKey) -> str:
@@ -78,7 +82,7 @@ class _CachedRelation:
:return _ReferenceKey: A key for this relation. :return _ReferenceKey: A key for this relation.
""" """
return _make_ref_key(self) return _make_key(self)
def add_reference(self, referrer: "_CachedRelation"): def add_reference(self, referrer: "_CachedRelation"):
"""Add a reference from referrer to self, indicating that if this node """Add a reference from referrer to self, indicating that if this node
@@ -141,7 +145,11 @@ class _CachedRelation:
:raises InternalError: If the new key already exists. :raises InternalError: If the new key already exists.
""" """
if new_key in self.referenced_by: if new_key in self.referenced_by:
raise NewNameAlreadyInCacheError(old_key, new_key) dbt.exceptions.raise_cache_inconsistent(
'in rename of "{}" -> "{}", new name is in the cache already'.format(
old_key, new_key
)
)
if old_key not in self.referenced_by: if old_key not in self.referenced_by:
return return
@@ -257,17 +265,21 @@ class RelationsCache:
if referenced is None: if referenced is None:
return return
if referenced is None: if referenced is None:
raise ReferencedLinkNotCachedError(referenced_key) dbt.exceptions.raise_cache_inconsistent(
"in add_link, referenced link key {} not in cache!".format(referenced_key)
)
dependent = self.relations.get(dependent_key) dependent = self.relations.get(dependent_key)
if dependent is None: if dependent is None:
raise DependentLinkNotCachedError(dependent_key) dbt.exceptions.raise_cache_inconsistent(
"in add_link, dependent link key {} not in cache!".format(dependent_key)
)
assert dependent is not None # we just raised! assert dependent is not None # we just raised!
referenced.add_reference(dependent) referenced.add_reference(dependent)
# This is called in plugins/postgres/dbt/adapters/postgres/impl.py # TODO: Is this dead code? I can't seem to find it grepping the codebase.
def add_link(self, referenced, dependent): def add_link(self, referenced, dependent):
"""Add a link between two relations to the database. If either relation """Add a link between two relations to the database. If either relation
does not exist, it will be added as an "external" relation. does not exist, it will be added as an "external" relation.
@@ -282,18 +294,13 @@ class RelationsCache:
:param BaseRelation dependent: The dependent model. :param BaseRelation dependent: The dependent model.
:raises InternalError: If either entry does not exist. :raises InternalError: If either entry does not exist.
""" """
ref_key = _make_ref_key(referenced) ref_key = _make_key(referenced)
dep_key = _make_ref_key(dependent) dep_key = _make_key(dependent)
if (ref_key.database, ref_key.schema) not in self: if (ref_key.database, ref_key.schema) not in self:
# if we have not cached the referenced schema at all, we must be # if we have not cached the referenced schema at all, we must be
# referring to a table outside our control. There's no need to make # referring to a table outside our control. There's no need to make
# a link - we will never drop the referenced relation during a run. # a link - we will never drop the referenced relation during a run.
fire_event( fire_event(UncachedRelation(dep_key=dep_key, ref_key=ref_key))
CacheAction(
ref_key=_make_msg_from_ref_key(ref_key),
ref_key_2=_make_msg_from_ref_key(dep_key),
)
)
return return
if ref_key not in self.relations: if ref_key not in self.relations:
# Insert a dummy "external" relation. # Insert a dummy "external" relation.
@@ -303,13 +310,7 @@ class RelationsCache:
# Insert a dummy "external" relation. # Insert a dummy "external" relation.
dependent = dependent.replace(type=referenced.External) dependent = dependent.replace(type=referenced.External)
self.add(dependent) self.add(dependent)
fire_event( fire_event(AddLink(dep_key=dep_key, ref_key=ref_key))
CacheAction(
action="add_link",
ref_key=_make_msg_from_ref_key(dep_key),
ref_key_2=_make_msg_from_ref_key(ref_key),
)
)
with self.lock: with self.lock:
self._add_link(ref_key, dep_key) self._add_link(ref_key, dep_key)
@@ -320,18 +321,12 @@ class RelationsCache:
:param BaseRelation relation: The underlying relation. :param BaseRelation relation: The underlying relation.
""" """
cached = _CachedRelation(relation) cached = _CachedRelation(relation)
fire_event_if( fire_event(AddRelation(relation=_make_key(cached)))
flags.LOG_CACHE_EVENTS, fire_event(DumpBeforeAddGraph(dump=Lazy.defer(lambda: self.dump_graph())))
lambda: CacheDumpGraph(before_after="before", action="adding", dump=self.dump_graph()),
)
fire_event(CacheAction(action="add_relation", ref_key=_make_ref_key_msg(cached)))
with self.lock: with self.lock:
self._setdefault(cached) self._setdefault(cached)
fire_event_if( fire_event(DumpAfterAddGraph(dump=Lazy.defer(lambda: self.dump_graph())))
flags.LOG_CACHE_EVENTS,
lambda: CacheDumpGraph(before_after="after", action="adding", dump=self.dump_graph()),
)
def _remove_refs(self, keys): def _remove_refs(self, keys):
"""Removes all references to all entries in keys. This does not """Removes all references to all entries in keys. This does not
@@ -346,6 +341,19 @@ class RelationsCache:
for cached in self.relations.values(): for cached in self.relations.values():
cached.release_references(keys) cached.release_references(keys)
def _drop_cascade_relation(self, dropped_key):
"""Drop the given relation and cascade it appropriately to all
dependent relations.
:param _CachedRelation dropped: An existing _CachedRelation to drop.
"""
if dropped_key not in self.relations:
fire_event(DropMissingRelation(relation=dropped_key))
return
consequences = self.relations[dropped_key].collect_consequences()
fire_event(DropCascade(dropped=dropped_key, consequences=consequences))
self._remove_refs(consequences)
def drop(self, relation): def drop(self, relation):
"""Drop the named relation and cascade it appropriately to all """Drop the named relation and cascade it appropriately to all
dependent relations. dependent relations.
@@ -357,22 +365,10 @@ class RelationsCache:
:param str schema: The schema of the relation to drop. :param str schema: The schema of the relation to drop.
:param str identifier: The identifier of the relation to drop. :param str identifier: The identifier of the relation to drop.
""" """
dropped_key = _make_ref_key(relation) dropped_key = _make_key(relation)
dropped_key_msg = _make_ref_key_msg(relation) fire_event(DropRelation(dropped=dropped_key))
fire_event(CacheAction(action="drop_relation", ref_key=dropped_key_msg))
with self.lock: with self.lock:
if dropped_key not in self.relations: self._drop_cascade_relation(dropped_key)
fire_event(CacheAction(action="drop_missing_relation", ref_key=dropped_key_msg))
return
consequences = self.relations[dropped_key].collect_consequences()
# convert from a list of _ReferenceKeys to a list of ReferenceKeyMsgs
consequence_msgs = [_make_msg_from_ref_key(key) for key in consequences]
fire_event(
CacheAction(
action="drop_cascade", ref_key=dropped_key_msg, ref_list=consequence_msgs
)
)
self._remove_refs(consequences)
def _rename_relation(self, old_key, new_relation): def _rename_relation(self, old_key, new_relation):
"""Rename a relation named old_key to new_key, updating references. """Rename a relation named old_key to new_key, updating references.
@@ -388,20 +384,14 @@ class RelationsCache:
relation = self.relations.pop(old_key) relation = self.relations.pop(old_key)
new_key = new_relation.key() new_key = new_relation.key()
# relation has to rename its innards, so it needs the _CachedRelation. # relaton has to rename its innards, so it needs the _CachedRelation.
relation.rename(new_relation) relation.rename(new_relation)
# update all the relations that refer to it # update all the relations that refer to it
for cached in self.relations.values(): for cached in self.relations.values():
if cached.is_referenced_by(old_key): if cached.is_referenced_by(old_key):
fire_event( fire_event(
CacheAction( UpdateReference(old_key=old_key, new_key=new_key, cached_key=cached.key())
action="update_reference",
ref_key=_make_ref_key_msg(old_key),
ref_key_2=_make_ref_key_msg(new_key),
ref_key_3=_make_ref_key_msg(cached.key()),
)
) )
cached.rename_key(old_key, new_key) cached.rename_key(old_key, new_key)
self.relations[new_key] = relation self.relations[new_key] = relation
@@ -426,12 +416,27 @@ class RelationsCache:
if new_key in self.relations: if new_key in self.relations:
# Tell user when collision caused by model names truncated during # Tell user when collision caused by model names truncated during
# materialization. # materialization.
raise TruncatedModelNameCausedCollisionError(new_key, self.relations) match = re.search("__dbt_backup|__dbt_tmp$", new_key.identifier)
if match:
truncated_model_name_prefix = new_key.identifier[: match.start()]
message_addendum = (
"\n\nName collisions can occur when the length of two "
"models' names approach your database's builtin limit. "
"Try restructuring your project such that no two models "
"share the prefix '{}'.".format(truncated_model_name_prefix)
+ " Then, clean your warehouse of any removed models."
)
else:
message_addendum = ""
dbt.exceptions.raise_cache_inconsistent(
"in rename, new key {} already in cache: {}{}".format(
new_key, list(self.relations.keys()), message_addendum
)
)
if old_key not in self.relations: if old_key not in self.relations:
fire_event( fire_event(TemporaryRelation(key=old_key))
CacheAction(action="temporary_relation", ref_key=_make_msg_from_ref_key(old_key))
)
return False return False
return True return True
@@ -447,20 +452,11 @@ class RelationsCache:
:param BaseRelation new: The new relation name information. :param BaseRelation new: The new relation name information.
:raises InternalError: If the new key is already present. :raises InternalError: If the new key is already present.
""" """
old_key = _make_ref_key(old) old_key = _make_key(old)
new_key = _make_ref_key(new) new_key = _make_key(new)
fire_event( fire_event(RenameSchema(old_key=old_key, new_key=new_key))
CacheAction(
action="rename_relation",
ref_key=_make_msg_from_ref_key(old_key),
ref_key_2=_make_msg_from_ref_key(new),
)
)
fire_event_if( fire_event(DumpBeforeRenameSchema(dump=Lazy.defer(lambda: self.dump_graph())))
flags.LOG_CACHE_EVENTS,
lambda: CacheDumpGraph(before_after="before", action="rename", dump=self.dump_graph()),
)
with self.lock: with self.lock:
if self._check_rename_constraints(old_key, new_key): if self._check_rename_constraints(old_key, new_key):
@@ -468,10 +464,7 @@ class RelationsCache:
else: else:
self._setdefault(_CachedRelation(new)) self._setdefault(_CachedRelation(new))
fire_event_if( fire_event(DumpAfterRenameSchema(dump=Lazy.defer(lambda: self.dump_graph())))
flags.LOG_CACHE_EVENTS,
lambda: CacheDumpGraph(before_after="after", action="rename", dump=self.dump_graph()),
)
def get_relations(self, database: Optional[str], schema: Optional[str]) -> List[Any]: def get_relations(self, database: Optional[str], schema: Optional[str]) -> List[Any]:
"""Case-insensitively yield all relations matching the given schema. """Case-insensitively yield all relations matching the given schema.
@@ -490,7 +483,9 @@ class RelationsCache:
] ]
if None in results: if None in results:
raise NoneRelationFoundError() dbt.exceptions.raise_cache_inconsistent(
"in get_relations, a None relation was found in the cache!"
)
return results return results
def clear(self): def clear(self):
@@ -517,6 +512,6 @@ class RelationsCache:
""" """
for relation in to_remove: for relation in to_remove:
# it may have been cascaded out already # it may have been cascaded out already
drop_key = _make_ref_key(relation) drop_key = _make_key(relation)
if drop_key in self.relations: if drop_key in self.relations:
self.drop(drop_key) self.drop(drop_key)

View File

@@ -1,18 +1,23 @@
import threading import threading
import traceback
from contextlib import contextmanager
from importlib import import_module
from pathlib import Path from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Type from importlib import import_module
from typing import Type, Dict, Any, List, Optional, Set
from dbt.adapters.base.plugin import AdapterPlugin from dbt.exceptions import RuntimeException, InternalException
from dbt.adapters.protocol import AdapterConfig, AdapterProtocol, RelationProtocol from dbt.include.global_project import (
from dbt.contracts.connection import AdapterRequiredConfig, Credentials PACKAGE_PATH as GLOBAL_PROJECT_PATH,
PROJECT_NAME as GLOBAL_PROJECT_NAME,
)
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import AdapterImportError, PluginLoadError from dbt.events.types import AdapterImportError, PluginLoadError
from dbt.exceptions import DbtInternalError, DbtRuntimeError from dbt.contracts.connection import Credentials, AdapterRequiredConfig
from dbt.include.global_project import PACKAGE_PATH as GLOBAL_PROJECT_PATH from dbt.adapters.protocol import (
from dbt.include.global_project import PROJECT_NAME as GLOBAL_PROJECT_NAME AdapterProtocol,
AdapterConfig,
RelationProtocol,
)
from dbt.adapters.base.plugin import AdapterPlugin
Adapter = AdapterProtocol Adapter = AdapterProtocol
@@ -34,7 +39,7 @@ class AdapterContainer:
names = ", ".join(self.plugins.keys()) names = ", ".join(self.plugins.keys())
message = f"Invalid adapter type {name}! Must be one of {names}" message = f"Invalid adapter type {name}! Must be one of {names}"
raise DbtRuntimeError(message) raise RuntimeException(message)
def get_adapter_class_by_name(self, name: str) -> Type[Adapter]: def get_adapter_class_by_name(self, name: str) -> Type[Adapter]:
plugin = self.get_plugin_by_name(name) plugin = self.get_plugin_by_name(name)
@@ -59,18 +64,18 @@ class AdapterContainer:
# if we failed to import the target module in particular, inform # if we failed to import the target module in particular, inform
# the user about it via a runtime error # the user about it via a runtime error
if exc.name == "dbt.adapters." + name: if exc.name == "dbt.adapters." + name:
fire_event(AdapterImportError(exc=str(exc))) fire_event(AdapterImportError(exc=exc))
raise DbtRuntimeError(f"Could not find adapter type {name}!") raise RuntimeException(f"Could not find adapter type {name}!")
# otherwise, the error had to have come from some underlying # otherwise, the error had to have come from some underlying
# library. Log the stack trace. # library. Log the stack trace.
fire_event(PluginLoadError(exc_info=traceback.format_exc())) fire_event(PluginLoadError())
raise raise
plugin: AdapterPlugin = mod.Plugin plugin: AdapterPlugin = mod.Plugin
plugin_type = plugin.adapter.type() plugin_type = plugin.adapter.type()
if plugin_type != name: if plugin_type != name:
raise DbtRuntimeError( raise RuntimeException(
f"Expected to find adapter with type named {name}, got " f"Expected to find adapter with type named {name}, got "
f"adapter with type {plugin_type}" f"adapter with type {plugin_type}"
) )
@@ -132,7 +137,7 @@ class AdapterContainer:
try: try:
plugin = self.plugins[plugin_name] plugin = self.plugins[plugin_name]
except KeyError: except KeyError:
raise DbtInternalError(f"No plugin found for {plugin_name}") from None raise InternalException(f"No plugin found for {plugin_name}") from None
plugins.append(plugin) plugins.append(plugin)
seen.add(plugin_name) seen.add(plugin_name)
for dep in plugin.dependencies: for dep in plugin.dependencies:
@@ -151,7 +156,7 @@ class AdapterContainer:
try: try:
path = self.packages[package_name] path = self.packages[package_name]
except KeyError: except KeyError:
raise DbtInternalError(f"No internal package listing found for {package_name}") raise InternalException(f"No internal package listing found for {package_name}")
paths.append(path) paths.append(path)
return paths return paths
@@ -212,12 +217,3 @@ def get_adapter_package_names(name: Optional[str]) -> List[str]:
def get_adapter_type_names(name: Optional[str]) -> List[str]: def get_adapter_type_names(name: Optional[str]) -> List[str]:
return FACTORY.get_adapter_type_names(name) return FACTORY.get_adapter_type_names(name)
@contextmanager
def adapter_management():
reset_adapters()
try:
yield
finally:
cleanup_connections()

View File

@@ -8,6 +8,7 @@ from typing import (
Generic, Generic,
TypeVar, TypeVar,
Tuple, Tuple,
Union,
Dict, Dict,
Any, Any,
) )
@@ -16,7 +17,8 @@ from typing_extensions import Protocol
import agate import agate
from dbt.contracts.connection import Connection, AdapterRequiredConfig, AdapterResponse from dbt.contracts.connection import Connection, AdapterRequiredConfig, AdapterResponse
from dbt.contracts.graph.nodes import ResultNode, ManifestNode from dbt.contracts.graph.compiled import CompiledNode, ManifestNode, NonSourceCompiledNode
from dbt.contracts.graph.parsed import ParsedNode, ParsedSourceDefinition
from dbt.contracts.graph.model_config import BaseConfig from dbt.contracts.graph.model_config import BaseConfig
from dbt.contracts.graph.manifest import Manifest from dbt.contracts.graph.manifest import Manifest
from dbt.contracts.relation import Policy, HasQuoting from dbt.contracts.relation import Policy, HasQuoting
@@ -46,7 +48,11 @@ class RelationProtocol(Protocol):
... ...
@classmethod @classmethod
def create_from(cls: Type[Self], config: HasQuoting, node: ResultNode) -> Self: def create_from(
cls: Type[Self],
config: HasQuoting,
node: Union[CompiledNode, ParsedNode, ParsedSourceDefinition],
) -> Self:
... ...
@@ -59,7 +65,7 @@ class CompilerProtocol(Protocol):
node: ManifestNode, node: ManifestNode,
manifest: Manifest, manifest: Manifest,
extra_context: Optional[Dict[str, Any]] = None, extra_context: Optional[Dict[str, Any]] = None,
) -> ManifestNode: ) -> NonSourceCompiledNode:
... ...
@@ -82,7 +88,7 @@ class AdapterProtocol( # type: ignore[misc]
], ],
): ):
# N.B. Technically these are ClassVars, but mypy doesn't support putting type vars in a # N.B. Technically these are ClassVars, but mypy doesn't support putting type vars in a
# ClassVar due to the restrictiveness of PEP-526 # ClassVar due to the restirctiveness of PEP-526
# See: https://github.com/python/mypy/issues/5144 # See: https://github.com/python/mypy/issues/5144
AdapterSpecificConfigs: Type[AdapterConfig_T] AdapterSpecificConfigs: Type[AdapterConfig_T]
Column: Type[Column_T] Column: Type[Column_T]

View File

@@ -2,7 +2,6 @@
from collections import namedtuple from collections import namedtuple
from typing import Any, Optional from typing import Any, Optional
from dbt.events.proto_types import ReferenceKeyMsg
_ReferenceKey = namedtuple("_ReferenceKey", "database schema identifier") _ReferenceKey = namedtuple("_ReferenceKey", "database schema identifier")
@@ -15,12 +14,7 @@ def lowercase(value: Optional[str]) -> Optional[str]:
return value.lower() return value.lower()
# For backwards compatibility. New code should use _make_ref_key
def _make_key(relation: Any) -> _ReferenceKey: def _make_key(relation: Any) -> _ReferenceKey:
return _make_ref_key(relation)
def _make_ref_key(relation: Any) -> _ReferenceKey:
"""Make _ReferenceKeys with lowercase values for the cache so we don't have """Make _ReferenceKeys with lowercase values for the cache so we don't have
to keep track of quoting to keep track of quoting
""" """
@@ -28,13 +22,3 @@ def _make_ref_key(relation: Any) -> _ReferenceKey:
return _ReferenceKey( return _ReferenceKey(
lowercase(relation.database), lowercase(relation.schema), lowercase(relation.identifier) lowercase(relation.database), lowercase(relation.schema), lowercase(relation.identifier)
) )
def _make_ref_key_msg(relation: Any):
return _make_msg_from_ref_key(_make_ref_key(relation))
def _make_msg_from_ref_key(ref_key: _ReferenceKey) -> ReferenceKeyMsg:
return ReferenceKeyMsg(
database=ref_key.database, schema=ref_key.schema, identifier=ref_key.identifier
)

View File

@@ -10,8 +10,6 @@ from dbt.adapters.base import BaseConnectionManager
from dbt.contracts.connection import Connection, ConnectionState, AdapterResponse from dbt.contracts.connection import Connection, ConnectionState, AdapterResponse
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import ConnectionUsed, SQLQuery, SQLCommit, SQLQueryStatus from dbt.events.types import ConnectionUsed, SQLQuery, SQLCommit, SQLQueryStatus
from dbt.events.contextvars import get_node_info
from dbt.utils import cast_to_str
class SQLConnectionManager(BaseConnectionManager): class SQLConnectionManager(BaseConnectionManager):
@@ -27,7 +25,9 @@ class SQLConnectionManager(BaseConnectionManager):
@abc.abstractmethod @abc.abstractmethod
def cancel(self, connection: Connection): def cancel(self, connection: Connection):
"""Cancel the given connection.""" """Cancel the given connection."""
raise dbt.exceptions.NotImplementedError("`cancel` is not implemented for this adapter!") raise dbt.exceptions.NotImplementedException(
"`cancel` is not implemented for this adapter!"
)
def cancel_open(self) -> List[str]: def cancel_open(self) -> List[str]:
names = [] names = []
@@ -55,13 +55,7 @@ class SQLConnectionManager(BaseConnectionManager):
connection = self.get_thread_connection() connection = self.get_thread_connection()
if auto_begin and connection.transaction_open is False: if auto_begin and connection.transaction_open is False:
self.begin() self.begin()
fire_event( fire_event(ConnectionUsed(conn_type=self.TYPE, conn_name=connection.name))
ConnectionUsed(
conn_type=self.TYPE,
conn_name=cast_to_str(connection.name),
node_info=get_node_info(),
)
)
with self.exception_handler(sql): with self.exception_handler(sql):
if abridge_sql_log: if abridge_sql_log:
@@ -69,11 +63,7 @@ class SQLConnectionManager(BaseConnectionManager):
else: else:
log_sql = sql log_sql = sql
fire_event( fire_event(SQLQuery(conn_name=connection.name, sql=log_sql))
SQLQuery(
conn_name=cast_to_str(connection.name), sql=log_sql, node_info=get_node_info()
)
)
pre = time.time() pre = time.time()
cursor = connection.handle.cursor() cursor = connection.handle.cursor()
@@ -81,9 +71,7 @@ class SQLConnectionManager(BaseConnectionManager):
fire_event( fire_event(
SQLQueryStatus( SQLQueryStatus(
status=str(self.get_response(cursor)), status=str(self.get_response(cursor)), elapsed=round((time.time() - pre), 2)
elapsed=round((time.time() - pre)),
node_info=get_node_info(),
) )
) )
@@ -93,7 +81,7 @@ class SQLConnectionManager(BaseConnectionManager):
@abc.abstractmethod @abc.abstractmethod
def get_response(cls, cursor: Any) -> AdapterResponse: def get_response(cls, cursor: Any) -> AdapterResponse:
"""Get the status of the cursor.""" """Get the status of the cursor."""
raise dbt.exceptions.NotImplementedError( raise dbt.exceptions.NotImplementedException(
"`get_response` is not implemented for this adapter!" "`get_response` is not implemented for this adapter!"
) )
@@ -149,7 +137,7 @@ class SQLConnectionManager(BaseConnectionManager):
def begin(self): def begin(self):
connection = self.get_thread_connection() connection = self.get_thread_connection()
if connection.transaction_open is True: if connection.transaction_open is True:
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
'Tried to begin a new transaction on connection "{}", but ' 'Tried to begin a new transaction on connection "{}", but '
"it already had one open!".format(connection.name) "it already had one open!".format(connection.name)
) )
@@ -162,12 +150,12 @@ class SQLConnectionManager(BaseConnectionManager):
def commit(self): def commit(self):
connection = self.get_thread_connection() connection = self.get_thread_connection()
if connection.transaction_open is False: if connection.transaction_open is False:
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
'Tried to commit transaction on connection "{}", but ' 'Tried to commit transaction on connection "{}", but '
"it does not have one open!".format(connection.name) "it does not have one open!".format(connection.name)
) )
fire_event(SQLCommit(conn_name=connection.name, node_info=get_node_info())) fire_event(SQLCommit(conn_name=connection.name))
self.add_commit_query() self.add_commit_query()
connection.transaction_open = False connection.transaction_open = False

View File

@@ -1,10 +1,11 @@
import agate import agate
from typing import Any, Optional, Tuple, Type, List from typing import Any, Optional, Tuple, Type, List
import dbt.clients.agate_helper
from dbt.contracts.connection import Connection from dbt.contracts.connection import Connection
from dbt.exceptions import RelationTypeNullError import dbt.exceptions
from dbt.adapters.base import BaseAdapter, available from dbt.adapters.base import BaseAdapter, available
from dbt.adapters.cache import _make_ref_key_msg from dbt.adapters.cache import _make_key
from dbt.adapters.sql import SQLConnectionManager from dbt.adapters.sql import SQLConnectionManager
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import ColTypeChange, SchemaCreation, SchemaDrop from dbt.events.types import ColTypeChange, SchemaCreation, SchemaDrop
@@ -109,7 +110,7 @@ class SQLAdapter(BaseAdapter):
ColTypeChange( ColTypeChange(
orig_type=target_column.data_type, orig_type=target_column.data_type,
new_type=new_type, new_type=new_type,
table=_make_ref_key_msg(current), table=_make_key(current),
) )
) )
@@ -131,7 +132,9 @@ class SQLAdapter(BaseAdapter):
def drop_relation(self, relation): def drop_relation(self, relation):
if relation.type is None: if relation.type is None:
raise RelationTypeNullError(relation) dbt.exceptions.raise_compiler_error(
"Tried to drop relation {}, but its type is null.".format(relation)
)
self.cache_dropped(relation) self.cache_dropped(relation)
self.execute_macro(DROP_RELATION_MACRO_NAME, kwargs={"relation": relation}) self.execute_macro(DROP_RELATION_MACRO_NAME, kwargs={"relation": relation})
@@ -152,7 +155,7 @@ class SQLAdapter(BaseAdapter):
def create_schema(self, relation: BaseRelation) -> None: def create_schema(self, relation: BaseRelation) -> None:
relation = relation.without_identifier() relation = relation.without_identifier()
fire_event(SchemaCreation(relation=_make_ref_key_msg(relation))) fire_event(SchemaCreation(relation=_make_key(relation)))
kwargs = { kwargs = {
"relation": relation, "relation": relation,
} }
@@ -163,7 +166,7 @@ class SQLAdapter(BaseAdapter):
def drop_schema(self, relation: BaseRelation) -> None: def drop_schema(self, relation: BaseRelation) -> None:
relation = relation.without_identifier() relation = relation.without_identifier()
fire_event(SchemaDrop(relation=_make_ref_key_msg(relation))) fire_event(SchemaDrop(relation=_make_key(relation)))
kwargs = { kwargs = {
"relation": relation, "relation": relation,
} }

View File

@@ -1 +0,0 @@
TODO

View File

@@ -1,44 +0,0 @@
# TODO Move this to /core/dbt/flags.py when we're ready to break things
import os
from dataclasses import dataclass
from multiprocessing import get_context
from pprint import pformat as pf
from click import get_current_context
if os.name != "nt":
# https://bugs.python.org/issue41567
import multiprocessing.popen_spawn_posix # type: ignore # noqa: F401
@dataclass(frozen=True)
class Flags:
def __init__(self, ctx=None) -> None:
if ctx is None:
ctx = get_current_context()
def assign_params(ctx):
"""Recursively adds all click params to flag object"""
for param_name, param_value in ctx.params.items():
# N.B. You have to use the base MRO method (object.__setattr__) to set attributes
# when using frozen dataclasses.
# https://docs.python.org/3/library/dataclasses.html#frozen-instances
if hasattr(self, param_name):
raise Exception(f"Duplicate flag names found in click command: {param_name}")
object.__setattr__(self, param_name.upper(), param_value)
if ctx.parent:
assign_params(ctx.parent)
assign_params(ctx)
# Hard coded flags
object.__setattr__(self, "WHICH", ctx.info_name)
object.__setattr__(self, "MP_CONTEXT", get_context("spawn"))
# Support console DO NOT TRACK initiave
if os.getenv("DO_NOT_TRACK", "").lower() in (1, "t", "true", "y", "yes"):
object.__setattr__(self, "ANONYMOUS_USAGE_STATS", False)
def __str__(self) -> str:
return str(pf(self.__dict__))

View File

@@ -1,412 +0,0 @@
import inspect # This is temporary for RAT-ing
from copy import copy
from pprint import pformat as pf # This is temporary for RAT-ing
import click
from dbt.adapters.factory import adapter_management
from dbt.cli import params as p
from dbt.cli.flags import Flags
from dbt.profiler import profiler
def cli_runner():
# Alias "list" to "ls"
ls = copy(cli.commands["list"])
ls.hidden = True
cli.add_command(ls, "ls")
# Run the cli
cli()
# dbt
@click.group(
context_settings={"help_option_names": ["-h", "--help"]},
invoke_without_command=True,
no_args_is_help=True,
epilog="Specify one of these sub-commands and you can find more help from there.",
)
@click.pass_context
@p.anonymous_usage_stats
@p.cache_selected_only
@p.debug
@p.enable_legacy_logger
@p.fail_fast
@p.log_cache_events
@p.log_format
@p.macro_debugging
@p.partial_parse
@p.print
@p.printer_width
@p.quiet
@p.record_timing_info
@p.static_parser
@p.use_colors
@p.use_experimental_parser
@p.version
@p.version_check
@p.warn_error
@p.warn_error_options
@p.write_json
def cli(ctx, **kwargs):
"""An ELT tool for managing your SQL transformations and data models.
For more documentation on these commands, visit: docs.getdbt.com
"""
incomplete_flags = Flags()
# Profiling
if incomplete_flags.RECORD_TIMING_INFO:
ctx.with_resource(profiler(enable=True, outfile=incomplete_flags.RECORD_TIMING_INFO))
# Adapter management
ctx.with_resource(adapter_management())
# Version info
if incomplete_flags.VERSION:
click.echo(f"`version` called\n ctx.params: {pf(ctx.params)}")
return
else:
del ctx.params["version"]
# dbt build
@cli.command("build")
@click.pass_context
@p.defer
@p.exclude
@p.fail_fast
@p.full_refresh
@p.indirect_selection
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.show
@p.state
@p.store_failures
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def build(ctx, **kwargs):
"""Run all Seeds, Models, Snapshots, and tests in DAG order"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt clean
@cli.command("clean")
@click.pass_context
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
def clean(ctx, **kwargs):
"""Delete all folders in the clean-targets list (usually the dbt_packages and target directories.)"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt docs
@cli.group()
@click.pass_context
def docs(ctx, **kwargs):
"""Generate or serve the documentation website for your project"""
# dbt docs generate
@docs.command("generate")
@click.pass_context
@p.compile_docs
@p.defer
@p.exclude
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def docs_generate(ctx, **kwargs):
"""Generate the documentation website for your project"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt docs serve
@docs.command("serve")
@click.pass_context
@p.browser
@p.port
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
def docs_serve(ctx, **kwargs):
"""Serve the documentation website for your project"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt compile
@cli.command("compile")
@click.pass_context
@p.defer
@p.exclude
@p.full_refresh
@p.log_path
@p.models
@p.parse_only
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def compile(ctx, **kwargs):
"""Generates executable SQL from source, model, test, and analysis files. Compiled SQL files are written to the target/ directory."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt debug
@cli.command("debug")
@click.pass_context
@p.config_dir
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.version_check
def debug(ctx, **kwargs):
"""Show some helpful information about dbt for debugging. Not to be confused with the --debug option which increases verbosity."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt deps
@cli.command("deps")
@click.pass_context
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
def deps(ctx, **kwargs):
"""Pull the most recent version of the dependencies listed in packages.yml"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt init
@cli.command("init")
@click.pass_context
@p.profile
@p.profiles_dir
@p.project_dir
@p.skip_profile_setup
@p.target
@p.vars
def init(ctx, **kwargs):
"""Initialize a new DBT project."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt list
@cli.command("list")
@click.pass_context
@p.exclude
@p.indirect_selection
@p.models
@p.output
@p.output_keys
@p.profile
@p.profiles_dir
@p.project_dir
@p.resource_type
@p.selector
@p.state
@p.target
@p.vars
def list(ctx, **kwargs):
"""List the resources in your project"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt parse
@cli.command("parse")
@click.pass_context
@p.compile_parse
@p.log_path
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
@p.write_manifest
def parse(ctx, **kwargs):
"""Parses the project and provides information on performance"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt run
@cli.command("run")
@click.pass_context
@p.defer
@p.exclude
@p.fail_fast
@p.full_refresh
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def run(ctx, **kwargs):
"""Compile SQL and execute against the current target database."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt run operation
@cli.command("run-operation")
@click.pass_context
@p.args
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
def run_operation(ctx, **kwargs):
"""Run the named macro with any supplied arguments."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt seed
@cli.command("seed")
@click.pass_context
@p.exclude
@p.full_refresh
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.show
@p.state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def seed(ctx, **kwargs):
"""Load data from csv files into your data warehouse."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt snapshot
@cli.command("snapshot")
@click.pass_context
@p.defer
@p.exclude
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.state
@p.target
@p.threads
@p.vars
def snapshot(ctx, **kwargs):
"""Execute snapshots defined in your project"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt source
@cli.group()
@click.pass_context
def source(ctx, **kwargs):
"""Manage your project's sources"""
# dbt source freshness
@source.command("freshness")
@click.pass_context
@p.exclude
@p.models
@p.output_path # TODO: Is this ok to re-use? We have three different output params, how much can we consolidate?
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.state
@p.target
@p.threads
@p.vars
def freshness(ctx, **kwargs):
"""Snapshots the current freshness of the project's sources"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt test
@cli.command("test")
@click.pass_context
@p.defer
@p.exclude
@p.fail_fast
@p.indirect_selection
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.state
@p.store_failures
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def test(ctx, **kwargs):
"""Runs tests on data in deployed models. Run this after `dbt run`"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# Support running as a module
if __name__ == "__main__":
cli_runner()

View File

@@ -1,48 +0,0 @@
from click import ParamType
import yaml
from dbt.helper_types import WarnErrorOptions
class YAML(ParamType):
"""The Click YAML type. Converts YAML strings into objects."""
name = "YAML"
def convert(self, value, param, ctx):
# assume non-string values are a problem
if not isinstance(value, str):
self.fail(f"Cannot load YAML from type {type(value)}", param, ctx)
try:
return yaml.load(value, Loader=yaml.Loader)
except yaml.parser.ParserError:
self.fail(f"String '{value}' is not valid YAML", param, ctx)
class WarnErrorOptionsType(YAML):
"""The Click WarnErrorOptions type. Converts YAML strings into objects."""
name = "WarnErrorOptionsType"
def convert(self, value, param, ctx):
include_exclude = super().convert(value, param, ctx)
return WarnErrorOptions(
include=include_exclude.get("include", []), exclude=include_exclude.get("exclude", [])
)
class Truthy(ParamType):
"""The Click Truthy type. Converts strings into a "truthy" type"""
name = "TRUTHY"
def convert(self, value, param, ctx):
# assume non-string / non-None values are a problem
if not isinstance(value, (str, None)):
self.fail(f"Cannot load TRUTHY from type {type(value)}", param, ctx)
if value is None or value.lower() in ("0", "false", "f"):
return None
else:
return value

View File

@@ -1,389 +0,0 @@
from pathlib import Path, PurePath
import click
from dbt.cli.option_types import YAML, WarnErrorOptionsType
from dbt.cli.resolvers import default_project_dir, default_profiles_dir
# TODO: The name (reflected in flags) is a correction!
# The original name was `SEND_ANONYMOUS_USAGE_STATS` and used an env var called "DBT_SEND_ANONYMOUS_USAGE_STATS"
# Both of which break existing naming conventions (doesn't match param flag).
# This will need to be fixed before use in the main codebase and communicated as a change to the community!
anonymous_usage_stats = click.option(
"--anonymous-usage-stats/--no-anonymous-usage-stats",
envvar="DBT_ANONYMOUS_USAGE_STATS",
help="Send anonymous usage stats to dbt Labs.",
default=True,
)
args = click.option(
"--args",
envvar=None,
help="Supply arguments to the macro. This dictionary will be mapped to the keyword arguments defined in the selected macro. This argument should be a YAML string, eg. '{my_variable: my_value}'",
type=YAML(),
)
browser = click.option(
"--browser/--no-browser",
envvar=None,
help="Wether or not to open a local web browser after starting the server",
default=True,
)
cache_selected_only = click.option(
"--cache-selected-only/--no-cache-selected-only",
envvar="DBT_CACHE_SELECTED_ONLY",
help="Pre cache database objects relevant to selected resource only.",
)
compile_docs = click.option(
"--compile/--no-compile",
envvar=None,
help="Wether or not to run 'dbt compile' as part of docs generation",
default=True,
)
compile_parse = click.option(
"--compile/--no-compile",
envvar=None,
help="TODO: No help text currently available",
default=True,
)
config_dir = click.option(
"--config-dir",
envvar=None,
help="If specified, DBT will show path information for this project",
type=click.STRING,
)
debug = click.option(
"--debug/--no-debug",
"-d/ ",
envvar="DBT_DEBUG",
help="Display debug logging during dbt execution. Useful for debugging and making bug reports.",
)
# TODO: The env var and name (reflected in flags) are corrections!
# The original name was `DEFER_MODE` and used an env var called "DBT_DEFER_TO_STATE"
# Both of which break existing naming conventions.
# This will need to be fixed before use in the main codebase and communicated as a change to the community!
defer = click.option(
"--defer/--no-defer",
envvar="DBT_DEFER",
help="If set, defer to the state variable for resolving unselected nodes.",
)
enable_legacy_logger = click.option(
"--enable-legacy-logger/--no-enable-legacy-logger",
envvar="DBT_ENABLE_LEGACY_LOGGER",
hidden=True,
)
exclude = click.option("--exclude", envvar=None, help="Specify the nodes to exclude.")
fail_fast = click.option(
"--fail-fast/--no-fail-fast",
"-x/ ",
envvar="DBT_FAIL_FAST",
help="Stop execution on first failure.",
)
full_refresh = click.option(
"--full-refresh",
"-f",
envvar="DBT_FULL_REFRESH",
help="If specified, dbt will drop incremental models and fully-recalculate the incremental table from the model definition.",
is_flag=True,
)
indirect_selection = click.option(
"--indirect-selection",
envvar="DBT_INDIRECT_SELECTION",
help="Select all tests that are adjacent to selected resources, even if they those resources have been explicitly selected.",
type=click.Choice(["eager", "cautious"], case_sensitive=False),
default="eager",
)
log_cache_events = click.option(
"--log-cache-events/--no-log-cache-events",
help="Enable verbose adapter cache logging.",
envvar="DBT_LOG_CACHE_EVENTS",
)
log_format = click.option(
"--log-format",
envvar="DBT_LOG_FORMAT",
help="Specify the log format, overriding the command's default.",
type=click.Choice(["text", "json", "default"], case_sensitive=False),
default="default",
)
log_path = click.option(
"--log-path",
envvar="DBT_LOG_PATH",
help="Configure the 'log-path'. Only applies this setting for the current run. Overrides the 'DBT_LOG_PATH' if it is set.",
type=click.Path(),
)
macro_debugging = click.option(
"--macro-debugging/--no-macro-debugging",
envvar="DBT_MACRO_DEBUGGING",
hidden=True,
)
models = click.option(
"-m",
"-s",
"models",
envvar=None,
help="Specify the nodes to include.",
multiple=True,
)
output = click.option(
"--output",
envvar=None,
help="TODO: No current help text",
type=click.Choice(["json", "name", "path", "selector"], case_sensitive=False),
default="name",
)
output_keys = click.option(
"--output-keys", envvar=None, help="TODO: No current help text", type=click.STRING
)
output_path = click.option(
"--output",
"-o",
envvar=None,
help="Specify the output path for the json report. By default, outputs to 'target/sources.json'",
type=click.Path(file_okay=True, dir_okay=False, writable=True),
default=PurePath.joinpath(Path.cwd(), "target/sources.json"),
)
parse_only = click.option(
"--parse-only",
envvar=None,
help="TODO: No help text currently available",
is_flag=True,
)
partial_parse = click.option(
"--partial-parse/--no-partial-parse",
envvar="DBT_PARTIAL_PARSE",
help="Allow for partial parsing by looking for and writing to a pickle file in the target directory. This overrides the user configuration file.",
default=True,
)
port = click.option(
"--port",
envvar=None,
help="Specify the port number for the docs server",
default=8080,
type=click.INT,
)
# TODO: The env var and name (reflected in flags) are corrections!
# The original name was `NO_PRINT` and used the env var `DBT_NO_PRINT`.
# Both of which break existing naming conventions.
# This will need to be fixed before use in the main codebase and communicated as a change to the community!
print = click.option(
"--print/--no-print",
envvar="DBT_PRINT",
help="Output all {{ print() }} macro calls.",
default=True,
)
printer_width = click.option(
"--printer-width",
envvar="DBT_PRINTER_WIDTH",
help="Sets the width of terminal output",
type=click.INT,
default=80,
)
profile = click.option(
"--profile",
envvar=None,
help="Which profile to load. Overrides setting in dbt_project.yml.",
)
profiles_dir = click.option(
"--profiles-dir",
envvar="DBT_PROFILES_DIR",
help="Which directory to look in for the profiles.yml file. If not set, dbt will look in the current working directory first, then HOME/.dbt/",
default=default_profiles_dir(),
type=click.Path(exists=True),
)
project_dir = click.option(
"--project-dir",
envvar=None,
help="Which directory to look in for the dbt_project.yml file. Default is the current working directory and its parents.",
default=default_project_dir(),
type=click.Path(exists=True),
)
quiet = click.option(
"--quiet/--no-quiet",
envvar="DBT_QUIET",
help="Suppress all non-error logging to stdout. Does not affect {{ print() }} macro calls.",
)
record_timing_info = click.option(
"--record-timing-info",
"-r",
envvar=None,
help="When this option is passed, dbt will output low-level timing stats to the specified file. Example: `--record-timing-info output.profile`",
type=click.Path(exists=False),
)
resource_type = click.option(
"--resource-type",
envvar=None,
help="TODO: No current help text",
type=click.Choice(
[
"metric",
"source",
"analysis",
"model",
"test",
"exposure",
"snapshot",
"seed",
"default",
"all",
],
case_sensitive=False,
),
default="default",
)
selector = click.option(
"--selector", envvar=None, help="The selector name to use, as defined in selectors.yml"
)
show = click.option(
"--show", envvar=None, help="Show a sample of the loaded data in the terminal", is_flag=True
)
skip_profile_setup = click.option(
"--skip-profile-setup", "-s", envvar=None, help="Skip interactive profile setup.", is_flag=True
)
# TODO: The env var and name (reflected in flags) are corrections!
# The original name was `ARTIFACT_STATE_PATH` and used the env var `DBT_ARTIFACT_STATE_PATH`.
# Both of which break existing naming conventions.
# This will need to be fixed before use in the main codebase and communicated as a change to the community!
state = click.option(
"--state",
envvar="DBT_STATE",
help="If set, use the given directory as the source for json files to compare with this project.",
type=click.Path(
dir_okay=True,
exists=True,
file_okay=False,
readable=True,
resolve_path=True,
),
)
static_parser = click.option(
"--static-parser/--no-static-parser",
envvar="DBT_STATIC_PARSER",
help="Use the static parser.",
default=True,
)
store_failures = click.option(
"--store-failures",
envvar="DBT_STORE_FAILURES",
help="Store test results (failing rows) in the database",
is_flag=True,
)
target = click.option(
"--target", "-t", envvar=None, help="Which target to load for the given profile"
)
target_path = click.option(
"--target-path",
envvar="DBT_TARGET_PATH",
help="Configure the 'target-path'. Only applies this setting for the current run. Overrides the 'DBT_TARGET_PATH' if it is set.",
type=click.Path(),
)
threads = click.option(
"--threads",
envvar=None,
help="Specify number of threads to use while executing models. Overrides settings in profiles.yml.",
default=1,
type=click.INT,
)
use_colors = click.option(
"--use-colors/--no-use-colors",
envvar="DBT_USE_COLORS",
help="Output is colorized by default and may also be set in a profile or at the command line.",
default=True,
)
use_experimental_parser = click.option(
"--use-experimental-parser/--no-use-experimental-parser",
envvar="DBT_USE_EXPERIMENTAL_PARSER",
help="Enable experimental parsing features.",
)
vars = click.option(
"--vars",
envvar=None,
help="Supply variables to the project. This argument overrides variables defined in your dbt_project.yml file. This argument should be a YAML string, eg. '{my_variable: my_value}'",
type=YAML(),
)
version = click.option(
"--version",
envvar=None,
help="Show version information",
is_flag=True,
)
version_check = click.option(
"--version-check/--no-version-check",
envvar="DBT_VERSION_CHECK",
help="Ensure dbt's version matches the one specified in the dbt_project.yml file ('require-dbt-version')",
default=True,
)
warn_error = click.option(
"--warn-error",
envvar="DBT_WARN_ERROR",
help="If dbt would normally warn, instead raise an exception. Examples include --select that selects nothing, deprecations, configurations with no associated models, invalid test configurations, and missing sources/refs in tests.",
default=None,
flag_value=True,
)
warn_error_options = click.option(
"--warn-error-options",
envvar="DBT_WARN_ERROR_OPTIONS",
default=None,
help="""If dbt would normally warn, instead raise an exception based on include/exclude configuration. Examples include --select that selects nothing, deprecations, configurations with no associated models, invalid test configurations,
and missing sources/refs in tests. This argument should be a YAML string, with keys 'include' or 'exclude'. eg. '{"include": "all", "exclude": ["NoNodesForSelectionCriteria"]}'""",
type=WarnErrorOptionsType(),
)
write_json = click.option(
"--write-json/--no-write-json",
envvar="DBT_WRITE_JSON",
help="Writing the manifest and run_results.json files to disk",
default=True,
)
write_manifest = click.option(
"--write-manifest/--no-write-manifest",
envvar=None,
help="TODO: No help text currently available",
default=True,
)

View File

@@ -1,11 +0,0 @@
from pathlib import Path
def default_project_dir():
paths = list(Path.cwd().parents)
paths.insert(0, Path.cwd())
return next((x for x in paths if (x / "dbt_project.yml").exists()), Path.cwd())
def default_profiles_dir():
return Path.cwd() if (Path.cwd() / "profiles.yml").exists() else Path.home() / ".dbt"

View File

@@ -1,19 +1 @@
# Clients README # Clients README
### Jinja
#### How are materializations defined
Model materializations are kept in `core/dbt/include/global_project/macros/materializations/models/`. Materializations are defined using syntax that isn't part of the Jinja standard library. These tags are referenced internally, and materializations can be overridden in user projects when users have specific needs.
```
-- Pseudocode for arguments
{% materialization <name>, <target name := one_of{default, adapter}> %}'
{% endmaterialization %}
```
These blocks are referred to Jinja extensions. Extensions are defined as part of the accepted Jinja code encapsulated within a dbt project. This includes system code used internally by dbt and user space (i.e. user-defined) macros. Extensions exist to help Jinja users create reusable code blocks or abstract objects--for us, materializations are a great use-case since we pass these around as arguments within dbt system code.
The code that defines this extension is a class `MaterializationExtension` and a `parse` routine. That code lives in [clients/jinja.py](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/clients/jinja.py). The routine
enables Jinja to parse (i.e. recognize) the unique comma separated arg structure our `materialization` tags exhibit (the `table, default` as seen above).

View File

@@ -1,15 +1,7 @@
import re import re
from collections import namedtuple from collections import namedtuple
from dbt.exceptions import ( import dbt.exceptions
BlockDefinitionNotAtTopError,
DbtInternalError,
MissingCloseTagError,
MissingControlFlowStartTagError,
NestedTagsError,
UnexpectedControlFlowEndTagError,
UnexpectedMacroEOFError,
)
def regex(pat): def regex(pat):
@@ -147,7 +139,10 @@ class TagIterator:
def _expect_match(self, expected_name, *patterns, **kwargs): def _expect_match(self, expected_name, *patterns, **kwargs):
match = self._first_match(*patterns, **kwargs) match = self._first_match(*patterns, **kwargs)
if match is None: if match is None:
raise UnexpectedMacroEOFError(expected_name, self.data[self.pos :]) msg = 'unexpected EOF, expected {}, got "{}"'.format(
expected_name, self.data[self.pos :]
)
dbt.exceptions.raise_compiler_error(msg)
return match return match
def handle_expr(self, match): def handle_expr(self, match):
@@ -261,7 +256,7 @@ class TagIterator:
elif block_type_name is not None: elif block_type_name is not None:
yield self.handle_tag(match) yield self.handle_tag(match)
else: else:
raise DbtInternalError( raise dbt.exceptions.InternalException(
"Invalid regex match in next_block, expected block start, " "Invalid regex match in next_block, expected block start, "
"expr start, or comment start" "expr start, or comment start"
) )
@@ -270,6 +265,13 @@ class TagIterator:
return self.find_tags() return self.find_tags()
duplicate_tags = (
"Got nested tags: {outer.block_type_name} (started at {outer.start}) did "
"not have a matching {{% end{outer.block_type_name} %}} before a "
"subsequent {inner.block_type_name} was found (started at {inner.start})"
)
_CONTROL_FLOW_TAGS = { _CONTROL_FLOW_TAGS = {
"if": "endif", "if": "endif",
"for": "endfor", "for": "endfor",
@@ -317,16 +319,33 @@ class BlockIterator:
found = self.stack.pop() found = self.stack.pop()
else: else:
expected = _CONTROL_FLOW_END_TAGS[tag.block_type_name] expected = _CONTROL_FLOW_END_TAGS[tag.block_type_name]
raise UnexpectedControlFlowEndTagError(tag, expected, self.tag_parser) dbt.exceptions.raise_compiler_error(
(
"Got an unexpected control flow end tag, got {} but "
"never saw a preceeding {} (@ {})"
).format(tag.block_type_name, expected, self.tag_parser.linepos(tag.start))
)
expected = _CONTROL_FLOW_TAGS[found] expected = _CONTROL_FLOW_TAGS[found]
if expected != tag.block_type_name: if expected != tag.block_type_name:
raise MissingControlFlowStartTagError(tag, expected, self.tag_parser) dbt.exceptions.raise_compiler_error(
(
"Got an unexpected control flow end tag, got {} but "
"expected {} next (@ {})"
).format(tag.block_type_name, expected, self.tag_parser.linepos(tag.start))
)
if tag.block_type_name in allowed_blocks: if tag.block_type_name in allowed_blocks:
if self.stack: if self.stack:
raise BlockDefinitionNotAtTopError(self.tag_parser, tag.start) dbt.exceptions.raise_compiler_error(
(
"Got a block definition inside control flow at {}. "
"All dbt block definitions must be at the top level"
).format(self.tag_parser.linepos(tag.start))
)
if self.current is not None: if self.current is not None:
raise NestedTagsError(outer=self.current, inner=tag) dbt.exceptions.raise_compiler_error(
duplicate_tags.format(outer=self.current, inner=tag)
)
if collect_raw_data: if collect_raw_data:
raw_data = self.data[self.last_position : tag.start] raw_data = self.data[self.last_position : tag.start]
self.last_position = tag.start self.last_position = tag.start
@@ -347,7 +366,11 @@ class BlockIterator:
if self.current: if self.current:
linecount = self.data[: self.current.end].count("\n") + 1 linecount = self.data[: self.current.end].count("\n") + 1
raise MissingCloseTagError(self.current.block_type_name, linecount) dbt.exceptions.raise_compiler_error(
(
"Reached EOF without finding a close tag for " "{} (searched from line {})"
).format(self.current.block_type_name, linecount)
)
if collect_raw_data: if collect_raw_data:
raw_data = self.data[self.last_position :] raw_data = self.data[self.last_position :]

View File

@@ -7,7 +7,7 @@ import json
import dbt.utils import dbt.utils
from typing import Iterable, List, Dict, Union, Optional, Any from typing import Iterable, List, Dict, Union, Optional, Any
from dbt.exceptions import DbtRuntimeError from dbt.exceptions import RuntimeException
BOM = BOM_UTF8.decode("utf-8") # '\ufeff' BOM = BOM_UTF8.decode("utf-8") # '\ufeff'
@@ -168,7 +168,7 @@ class ColumnTypeBuilder(Dict[str, NullableAgateType]):
return return
elif not isinstance(value, type(existing_type)): elif not isinstance(value, type(existing_type)):
# actual type mismatch! # actual type mismatch!
raise DbtRuntimeError( raise RuntimeException(
f"Tables contain columns with the same names ({key}), " f"Tables contain columns with the same names ({key}), "
f"but different types ({value} vs {existing_type})" f"but different types ({value} vs {existing_type})"
) )

View File

@@ -14,10 +14,10 @@ from dbt.events.types import (
) )
from dbt.exceptions import ( from dbt.exceptions import (
CommandResultError, CommandResultError,
GitCheckoutError, RuntimeException,
GitCloningError, bad_package_spec,
UnknownGitCloningProblemError, raise_git_cloning_error,
DbtRuntimeError, raise_git_cloning_problem,
) )
from packaging import version from packaging import version
@@ -27,6 +27,16 @@ def _is_commit(revision: str) -> bool:
return bool(re.match(r"\b[0-9a-f]{40}\b", revision)) return bool(re.match(r"\b[0-9a-f]{40}\b", revision))
def _raise_git_cloning_error(repo, revision, error):
stderr = error.stderr.strip()
if "usage: git" in stderr:
stderr = stderr.split("\nusage: git")[0]
if re.match("fatal: destination path '(.+)' already exists", stderr):
raise_git_cloning_error(error)
bad_package_spec(repo, revision, stderr)
def clone(repo, cwd, dirname=None, remove_git_dir=False, revision=None, subdirectory=None): def clone(repo, cwd, dirname=None, remove_git_dir=False, revision=None, subdirectory=None):
has_revision = revision is not None has_revision = revision is not None
is_commit = _is_commit(revision or "") is_commit = _is_commit(revision or "")
@@ -54,7 +64,7 @@ def clone(repo, cwd, dirname=None, remove_git_dir=False, revision=None, subdirec
try: try:
result = run_cmd(cwd, clone_cmd, env={"LC_ALL": "C"}) result = run_cmd(cwd, clone_cmd, env={"LC_ALL": "C"})
except CommandResultError as exc: except CommandResultError as exc:
raise GitCloningError(repo, revision, exc) _raise_git_cloning_error(repo, revision, exc)
if subdirectory: if subdirectory:
cwd_subdir = os.path.join(cwd, dirname or "") cwd_subdir = os.path.join(cwd, dirname or "")
@@ -62,7 +72,7 @@ def clone(repo, cwd, dirname=None, remove_git_dir=False, revision=None, subdirec
try: try:
run_cmd(cwd_subdir, clone_cmd_subdir) run_cmd(cwd_subdir, clone_cmd_subdir)
except CommandResultError as exc: except CommandResultError as exc:
raise GitCloningError(repo, revision, exc) _raise_git_cloning_error(repo, revision, exc)
if remove_git_dir: if remove_git_dir:
rmdir(os.path.join(dirname, ".git")) rmdir(os.path.join(dirname, ".git"))
@@ -105,7 +115,8 @@ def checkout(cwd, repo, revision=None):
try: try:
return _checkout(cwd, repo, revision) return _checkout(cwd, repo, revision)
except CommandResultError as exc: except CommandResultError as exc:
raise GitCheckoutError(repo=repo, revision=revision, error=exc) stderr = exc.stderr.strip()
bad_package_spec(repo, revision, stderr)
def get_current_sha(cwd): def get_current_sha(cwd):
@@ -134,7 +145,7 @@ def clone_and_checkout(
err = exc.stderr err = exc.stderr
exists = re.match("fatal: destination path '(.+)' already exists", err) exists = re.match("fatal: destination path '(.+)' already exists", err)
if not exists: if not exists:
raise UnknownGitCloningProblemError(repo) raise_git_cloning_problem(repo)
directory = None directory = None
start_sha = None start_sha = None
@@ -144,7 +155,7 @@ def clone_and_checkout(
else: else:
matches = re.match("Cloning into '(.+)'", err.decode("utf-8")) matches = re.match("Cloning into '(.+)'", err.decode("utf-8"))
if matches is None: if matches is None:
raise DbtRuntimeError(f'Error cloning {repo} - never saw "Cloning into ..." from git') raise RuntimeException(f'Error cloning {repo} - never saw "Cloning into ..." from git')
directory = matches.group(1) directory = matches.group(1)
fire_event(GitProgressPullingNewDependency(dir=directory)) fire_event(GitProgressPullingNewDependency(dir=directory))
full_path = os.path.join(cwd, directory) full_path = os.path.join(cwd, directory)

View File

@@ -25,26 +25,18 @@ from dbt.utils import (
) )
from dbt.clients._jinja_blocks import BlockIterator, BlockData, BlockTag from dbt.clients._jinja_blocks import BlockIterator, BlockData, BlockTag
from dbt.contracts.graph.nodes import GenericTestNode from dbt.contracts.graph.compiled import CompiledGenericTestNode
from dbt.contracts.graph.parsed import ParsedGenericTestNode
from dbt.exceptions import ( from dbt.exceptions import (
CaughtMacroError, InternalException,
CaughtMacroErrorWithNodeError, raise_compiler_error,
CompilationError, CompilationException,
DbtInternalError, invalid_materialization_argument,
MaterializationArgError,
JinjaRenderingError,
MacroReturn, MacroReturn,
MaterializtionMacroNotUsedError, JinjaRenderingException,
NoSupportedLanguagesFoundError, UndefinedMacroException,
UndefinedCompilationError,
UndefinedMacroError,
) )
from dbt import flags from dbt import flags
from dbt.node_types import ModelLanguage
SUPPORTED_LANG_ARG = jinja2.nodes.Name("supported_languages", "param")
def _linecache_inject(source, write): def _linecache_inject(source, write):
@@ -161,15 +153,15 @@ def quoted_native_concat(nodes):
except (ValueError, SyntaxError, MemoryError): except (ValueError, SyntaxError, MemoryError):
result = raw result = raw
if isinstance(raw, BoolMarker) and not isinstance(result, bool): if isinstance(raw, BoolMarker) and not isinstance(result, bool):
raise JinjaRenderingError(f"Could not convert value '{raw!s}' into type 'bool'") raise JinjaRenderingException(f"Could not convert value '{raw!s}' into type 'bool'")
if isinstance(raw, NumberMarker) and not _is_number(result): if isinstance(raw, NumberMarker) and not _is_number(result):
raise JinjaRenderingError(f"Could not convert value '{raw!s}' into type 'number'") raise JinjaRenderingException(f"Could not convert value '{raw!s}' into type 'number'")
return result return result
class NativeSandboxTemplate(jinja2.nativetypes.NativeTemplate): # mypy: ignore class NativeSandboxTemplate(jinja2.nativetypes.NativeTemplate): # mypy: ignore
environment_class = NativeSandboxEnvironment # type: ignore environment_class = NativeSandboxEnvironment
def render(self, *args, **kwargs): def render(self, *args, **kwargs):
"""Render the template to produce a native Python type. If the """Render the template to produce a native Python type. If the
@@ -241,12 +233,12 @@ class BaseMacroGenerator:
try: try:
yield yield
except (TypeError, jinja2.exceptions.TemplateRuntimeError) as e: except (TypeError, jinja2.exceptions.TemplateRuntimeError) as e:
raise CaughtMacroError(e) raise_compiler_error(str(e))
def call_macro(self, *args, **kwargs): def call_macro(self, *args, **kwargs):
# called from __call__ methods # called from __call__ methods
if self.context is None: if self.context is None:
raise DbtInternalError("Context is still None in call_macro!") raise InternalException("Context is still None in call_macro!")
assert self.context is not None assert self.context is not None
macro = self.get_macro() macro = self.get_macro()
@@ -273,7 +265,7 @@ class MacroStack(threading.local):
def pop(self, name): def pop(self, name):
got = self.call_stack.pop() got = self.call_stack.pop()
if got != name: if got != name:
raise DbtInternalError(f"popped {got}, expected {name}") raise InternalException(f"popped {got}, expected {name}")
class MacroGenerator(BaseMacroGenerator): class MacroGenerator(BaseMacroGenerator):
@@ -300,8 +292,8 @@ class MacroGenerator(BaseMacroGenerator):
try: try:
yield yield
except (TypeError, jinja2.exceptions.TemplateRuntimeError) as e: except (TypeError, jinja2.exceptions.TemplateRuntimeError) as e:
raise CaughtMacroErrorWithNodeError(exc=e, node=self.macro) raise_compiler_error(str(e), self.macro)
except CompilationError as e: except CompilationException as e:
e.stack.append(self.macro) e.stack.append(self.macro)
raise e raise e
@@ -309,13 +301,13 @@ class MacroGenerator(BaseMacroGenerator):
@contextmanager @contextmanager
def track_call(self): def track_call(self):
# This is only called from __call__ # This is only called from __call__
if self.stack is None: if self.stack is None or self.node is None:
yield yield
else: else:
unique_id = self.macro.unique_id unique_id = self.macro.unique_id
depth = self.stack.depth depth = self.stack.depth
# only mark depth=0 as a dependency, when creating this dependency we don't pass in stack # only mark depth=0 as a dependency
if depth == 0 and self.node: if depth == 0:
self.node.depends_on.add_macro(unique_id) self.node.depends_on.add_macro(unique_id)
self.stack.push(unique_id) self.stack.push(unique_id)
try: try:
@@ -372,19 +364,8 @@ class MaterializationExtension(jinja2.ext.Extension):
value = parser.parse_expression() value = parser.parse_expression()
adapter_name = value.value adapter_name = value.value
elif target.name == "supported_languages":
target.set_ctx("param")
node.args.append(target)
parser.stream.expect("assign")
languages = parser.parse_expression()
node.defaults.append(languages)
else: else:
raise MaterializationArgError(materialization_name, target.name) invalid_materialization_argument(materialization_name, target.name)
if SUPPORTED_LANG_ARG not in node.args:
node.args.append(SUPPORTED_LANG_ARG)
node.defaults.append(jinja2.nodes.List([jinja2.nodes.Const("sql")]))
node.name = get_materialization_macro_name(materialization_name, adapter_name) node.name = get_materialization_macro_name(materialization_name, adapter_name)
@@ -455,7 +436,7 @@ def create_undefined(node=None):
return self return self
def __reduce__(self): def __reduce__(self):
raise UndefinedCompilationError(name=self.name, node=node) raise_compiler_error(f"{self.name} is undefined", node=node)
return Undefined return Undefined
@@ -513,10 +494,10 @@ def catch_jinja(node=None) -> Iterator[None]:
yield yield
except jinja2.exceptions.TemplateSyntaxError as e: except jinja2.exceptions.TemplateSyntaxError as e:
e.translated = False e.translated = False
raise CompilationError(str(e), node) from e raise CompilationException(str(e), node) from e
except jinja2.exceptions.UndefinedError as e: except jinja2.exceptions.UndefinedError as e:
raise UndefinedMacroError(str(e), node) from e raise UndefinedMacroException(str(e), node) from e
except CompilationError as exc: except CompilationException as exc:
exc.add_node(node) exc.add_node(node)
raise raise
@@ -623,7 +604,7 @@ GENERIC_TEST_KWARGS_NAME = "_dbt_generic_test_kwargs"
def add_rendered_test_kwargs( def add_rendered_test_kwargs(
context: Dict[str, Any], context: Dict[str, Any],
node: GenericTestNode, node: Union[ParsedGenericTestNode, CompiledGenericTestNode],
capture_macros: bool = False, capture_macros: bool = False,
) -> None: ) -> None:
"""Render each of the test kwargs in the given context using the native """Render each of the test kwargs in the given context using the native
@@ -651,21 +632,3 @@ def add_rendered_test_kwargs(
# when the test node was created in _parse_generic_test. # when the test node was created in _parse_generic_test.
kwargs = deep_map_render(_convert_function, node.test_metadata.kwargs) kwargs = deep_map_render(_convert_function, node.test_metadata.kwargs)
context[GENERIC_TEST_KWARGS_NAME] = kwargs context[GENERIC_TEST_KWARGS_NAME] = kwargs
def get_supported_languages(node: jinja2.nodes.Macro) -> List[ModelLanguage]:
if "materialization" not in node.name:
raise MaterializtionMacroNotUsedError(node=node)
no_kwargs = not node.defaults
no_langs_found = SUPPORTED_LANG_ARG not in node.args
if no_kwargs or no_langs_found:
raise NoSupportedLanguagesFoundError(node=node)
lang_idx = node.args.index(SUPPORTED_LANG_ARG)
# indexing defaults from the end
# since supported_languages is a kwarg, and kwargs are at always after args
return [
ModelLanguage[item.value] for item in node.defaults[-(len(node.args) - lang_idx)].items
]

View File

@@ -1,6 +1,6 @@
import jinja2 import jinja2
from dbt.clients.jinja import get_environment from dbt.clients.jinja import get_environment
from dbt.exceptions import MacroNamespaceNotStringError, MacroNameNotStringError from dbt.exceptions import raise_compiler_error
def statically_extract_macro_calls(string, ctx, db_wrapper=None): def statically_extract_macro_calls(string, ctx, db_wrapper=None):
@@ -15,7 +15,7 @@ def statically_extract_macro_calls(string, ctx, db_wrapper=None):
if hasattr(func_call, "node") and hasattr(func_call.node, "name"): if hasattr(func_call, "node") and hasattr(func_call.node, "name"):
func_name = func_call.node.name func_name = func_call.node.name
else: else:
# func_call for dbt.current_timestamp macro # func_call for dbt_utils.current_timestamp macro
# Call( # Call(
# node=Getattr( # node=Getattr(
# node=Name( # node=Name(
@@ -117,14 +117,20 @@ def statically_parse_adapter_dispatch(func_call, ctx, db_wrapper):
func_name = kwarg.value.value func_name = kwarg.value.value
possible_macro_calls.append(func_name) possible_macro_calls.append(func_name)
else: else:
raise MacroNameNotStringError(kwarg_value=kwarg.value.value) raise_compiler_error(
f"The macro_name parameter ({kwarg.value.value}) "
"to adapter.dispatch was not a string"
)
elif kwarg.key == "macro_namespace": elif kwarg.key == "macro_namespace":
# This will remain to enable static resolution # This will remain to enable static resolution
kwarg_type = type(kwarg.value).__name__ kwarg_type = type(kwarg.value).__name__
if kwarg_type == "Const": if kwarg_type == "Const":
macro_namespace = kwarg.value.value macro_namespace = kwarg.value.value
else: else:
raise MacroNamespaceNotStringError(kwarg_type) raise_compiler_error(
"The macro_namespace parameter to adapter.dispatch "
f"is a {kwarg_type}, not a string"
)
# positional arguments # positional arguments
if packages_arg: if packages_arg:

View File

@@ -3,9 +3,9 @@ from typing import Any, Dict, List
import requests import requests
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import ( from dbt.events.types import (
RegistryProgressGETRequest, RegistryProgressMakingGETRequest,
RegistryProgressGETResponse, RegistryProgressGETResponse,
RegistryIndexProgressGETRequest, RegistryIndexProgressMakingGETRequest,
RegistryIndexProgressGETResponse, RegistryIndexProgressGETResponse,
RegistryResponseUnexpectedType, RegistryResponseUnexpectedType,
RegistryResponseMissingTopKeys, RegistryResponseMissingTopKeys,
@@ -14,7 +14,6 @@ from dbt.events.types import (
) )
from dbt.utils import memoized, _connection_exception_retry as connection_exception_retry from dbt.utils import memoized, _connection_exception_retry as connection_exception_retry
from dbt import deprecations from dbt import deprecations
from dbt import semver
import os import os
if os.getenv("DBT_PACKAGE_HUB_URL"): if os.getenv("DBT_PACKAGE_HUB_URL"):
@@ -38,7 +37,7 @@ def _get_with_retries(package_name, registry_base_url=None):
def _get(package_name, registry_base_url=None): def _get(package_name, registry_base_url=None):
url = _get_url(package_name, registry_base_url) url = _get_url(package_name, registry_base_url)
fire_event(RegistryProgressGETRequest(url=url)) fire_event(RegistryProgressMakingGETRequest(url=url))
# all exceptions from requests get caught in the retry logic so no need to wrap this here # all exceptions from requests get caught in the retry logic so no need to wrap this here
resp = requests.get(url, timeout=30) resp = requests.get(url, timeout=30)
fire_event(RegistryProgressGETResponse(url=url, resp_code=resp.status_code)) fire_event(RegistryProgressGETResponse(url=url, resp_code=resp.status_code))
@@ -126,43 +125,16 @@ def package_version(package_name, version, registry_base_url=None) -> Dict[str,
return response[version] return response[version]
def is_compatible_version(package_spec, dbt_version) -> bool: def get_available_versions(package_name) -> List["str"]:
require_dbt_version = package_spec.get("require_dbt_version")
if not require_dbt_version:
# if version requirements are missing or empty, assume any version is compatible
return True
else:
# determine whether dbt_version satisfies this package's require-dbt-version config
if not isinstance(require_dbt_version, list):
require_dbt_version = [require_dbt_version]
supported_versions = [
semver.VersionSpecifier.from_version_string(v) for v in require_dbt_version
]
return semver.versions_compatible(dbt_version, *supported_versions)
def get_compatible_versions(package_name, dbt_version, should_version_check) -> List["str"]:
# returns a list of all available versions of a package # returns a list of all available versions of a package
response = package(package_name) response = package(package_name)
return list(response)
# if the user doesn't care about installing compatible versions, just return them all
if not should_version_check:
return list(response)
# otherwise, only return versions that are compatible with the installed version of dbt-core
else:
compatible_versions = [
pkg_version
for pkg_version, info in response.items()
if is_compatible_version(info, dbt_version)
]
return compatible_versions
def _get_index(registry_base_url=None): def _get_index(registry_base_url=None):
url = _get_url("index", registry_base_url) url = _get_url("index", registry_base_url)
fire_event(RegistryIndexProgressGETRequest(url=url)) fire_event(RegistryIndexProgressMakingGETRequest(url=url))
# all exceptions from requests get caught in the retry logic so no need to wrap this here # all exceptions from requests get caught in the retry logic so no need to wrap this here
resp = requests.get(url, timeout=30) resp = requests.get(url, timeout=30)
fire_event(RegistryIndexProgressGETResponse(url=url, resp_code=resp.status_code)) fire_event(RegistryIndexProgressGETResponse(url=url, resp_code=resp.status_code))

View File

@@ -12,15 +12,14 @@ import tarfile
import requests import requests
import stat import stat
from typing import Type, NoReturn, List, Optional, Dict, Any, Tuple, Callable, Union from typing import Type, NoReturn, List, Optional, Dict, Any, Tuple, Callable, Union
from pathspec import PathSpec # type: ignore
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import ( from dbt.events.types import (
SystemErrorRetrievingModTime, SystemErrorRetrievingModTime,
SystemCouldNotWrite, SystemCouldNotWrite,
SystemExecutingCmd, SystemExecutingCmd,
SystemStdOut, SystemStdOutMsg,
SystemStdErr, SystemStdErrMsg,
SystemReportReturnCode, SystemReportReturnCode,
) )
import dbt.exceptions import dbt.exceptions
@@ -37,7 +36,6 @@ def find_matching(
root_path: str, root_path: str,
relative_paths_to_search: List[str], relative_paths_to_search: List[str],
file_pattern: str, file_pattern: str,
ignore_spec: Optional[PathSpec] = None,
) -> List[Dict[str, Any]]: ) -> List[Dict[str, Any]]:
""" """
Given an absolute `root_path`, a list of relative paths to that Given an absolute `root_path`, a list of relative paths to that
@@ -59,30 +57,19 @@ def find_matching(
reobj = re.compile(regex, re.IGNORECASE) reobj = re.compile(regex, re.IGNORECASE)
for relative_path_to_search in relative_paths_to_search: for relative_path_to_search in relative_paths_to_search:
# potential speedup for ignore_spec
# if ignore_spec.matches(relative_path_to_search):
# continue
absolute_path_to_search = os.path.join(root_path, relative_path_to_search) absolute_path_to_search = os.path.join(root_path, relative_path_to_search)
walk_results = os.walk(absolute_path_to_search) walk_results = os.walk(absolute_path_to_search)
for current_path, subdirectories, local_files in walk_results: for current_path, subdirectories, local_files in walk_results:
# potential speedup for ignore_spec
# relative_dir = os.path.relpath(current_path, root_path) + os.sep
# if ignore_spec.match(relative_dir):
# continue
for local_file in local_files: for local_file in local_files:
absolute_path = os.path.join(current_path, local_file) absolute_path = os.path.join(current_path, local_file)
relative_path = os.path.relpath(absolute_path, absolute_path_to_search) relative_path = os.path.relpath(absolute_path, absolute_path_to_search)
relative_path_to_root = os.path.join(relative_path_to_search, relative_path)
modification_time = 0.0 modification_time = 0.0
try: try:
modification_time = os.path.getmtime(absolute_path) modification_time = os.path.getmtime(absolute_path)
except OSError: except OSError:
fire_event(SystemErrorRetrievingModTime(path=absolute_path)) fire_event(SystemErrorRetrievingModTime(path=absolute_path))
if reobj.match(local_file) and ( if reobj.match(local_file):
not ignore_spec or not ignore_spec.match_file(relative_path_to_root)
):
matching.append( matching.append(
{ {
"searched_path": relative_path_to_search, "searched_path": relative_path_to_search,
@@ -144,8 +131,7 @@ def make_symlink(source: str, link_path: str) -> None:
Create a symlink at `link_path` referring to `source`. Create a symlink at `link_path` referring to `source`.
""" """
if not supports_symlinks(): if not supports_symlinks():
# TODO: why not import these at top? dbt.exceptions.system_error("create a symbolic link")
raise dbt.exceptions.SymbolicLinkError()
os.symlink(source, link_path) os.symlink(source, link_path)
@@ -178,7 +164,7 @@ def write_file(path: str, contents: str = "") -> bool:
reason = "Path was possibly too long" reason = "Path was possibly too long"
# all our hard work and the path was still too long. Log and # all our hard work and the path was still too long. Log and
# continue. # continue.
fire_event(SystemCouldNotWrite(path=path, reason=reason, exc=str(exc))) fire_event(SystemCouldNotWrite(path=path, reason=reason, exc=exc))
else: else:
raise raise
return True return True
@@ -412,7 +398,7 @@ def _interpret_oserror(exc: OSError, cwd: str, cmd: List[str]) -> NoReturn:
_handle_posix_error(exc, cwd, cmd) _handle_posix_error(exc, cwd, cmd)
# this should not be reachable, raise _something_ at least! # this should not be reachable, raise _something_ at least!
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
"Unhandled exception in _interpret_oserror: {}".format(exc) "Unhandled exception in _interpret_oserror: {}".format(exc)
) )
@@ -441,8 +427,8 @@ def run_cmd(cwd: str, cmd: List[str], env: Optional[Dict[str, Any]] = None) -> T
except OSError as exc: except OSError as exc:
_interpret_oserror(exc, cwd, cmd) _interpret_oserror(exc, cwd, cmd)
fire_event(SystemStdOut(bmsg=out)) fire_event(SystemStdOutMsg(bmsg=out))
fire_event(SystemStdErr(bmsg=err)) fire_event(SystemStdErrMsg(bmsg=err))
if proc.returncode != 0: if proc.returncode != 0:
fire_event(SystemReportReturnCode(returncode=proc.returncode)) fire_event(SystemReportReturnCode(returncode=proc.returncode))

View File

@@ -60,4 +60,4 @@ def load_yaml_text(contents, path=None):
else: else:
error = str(e) error = str(e)
raise dbt.exceptions.DbtValidationError(error) raise dbt.exceptions.ValidationException(error)

View File

@@ -1,6 +1,6 @@
import os import os
from collections import defaultdict from collections import defaultdict
from typing import List, Dict, Any, Tuple, Optional from typing import List, Dict, Any, Tuple, cast, Optional
import networkx as nx # type: ignore import networkx as nx # type: ignore
import pickle import pickle
@@ -12,30 +12,38 @@ from dbt.clients import jinja
from dbt.clients.system import make_directory from dbt.clients.system import make_directory
from dbt.context.providers import generate_runtime_model_context from dbt.context.providers import generate_runtime_model_context
from dbt.contracts.graph.manifest import Manifest, UniqueID from dbt.contracts.graph.manifest import Manifest, UniqueID
from dbt.contracts.graph.nodes import ( from dbt.contracts.graph.compiled import (
ManifestNode, COMPILED_TYPES,
ManifestSQLNode, CompiledGenericTestNode,
GenericTestNode,
GraphMemberNode, GraphMemberNode,
InjectedCTE, InjectedCTE,
SeedNode, ManifestNode,
NonSourceCompiledNode,
) )
from dbt.contracts.graph.parsed import ParsedNode
from dbt.exceptions import ( from dbt.exceptions import (
GraphDependencyNotFoundError, dependency_not_found,
DbtInternalError, InternalException,
DbtRuntimeError, RuntimeException,
) )
from dbt.graph import Graph from dbt.graph import Graph
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import FoundStats, WritingInjectedSQLForNode from dbt.events.types import FoundStats, CompilingNode, WritingInjectedSQLForNode
from dbt.events.contextvars import get_node_info from dbt.node_types import NodeType
from dbt.node_types import NodeType, ModelLanguage
from dbt.events.format import pluralize from dbt.events.format import pluralize
import dbt.tracking import dbt.tracking
graph_file_name = "graph.gpickle" graph_file_name = "graph.gpickle"
def _compiled_type_for(model: ParsedNode):
if type(model) not in COMPILED_TYPES:
raise InternalException(
f"Asked to compile {type(model)} node, but it has no compiled form"
)
return COMPILED_TYPES[type(model)]
def print_compile_stats(stats): def print_compile_stats(stats):
names = { names = {
NodeType.Model: "model", NodeType.Model: "model",
@@ -48,7 +56,6 @@ def print_compile_stats(stats):
NodeType.Source: "source", NodeType.Source: "source",
NodeType.Exposure: "exposure", NodeType.Exposure: "exposure",
NodeType.Metric: "metric", NodeType.Metric: "metric",
NodeType.Entity: "entity",
} }
results = {k: 0 for k in names.keys()} results = {k: 0 for k in names.keys()}
@@ -84,8 +91,6 @@ def _generate_stats(manifest: Manifest):
stats[exposure.resource_type] += 1 stats[exposure.resource_type] += 1
for metric in manifest.metrics.values(): for metric in manifest.metrics.values():
stats[metric.resource_type] += 1 stats[metric.resource_type] += 1
for entity in manifest.entities.values():
stats[entity.resource_type] += 1
for macro in manifest.macros.values(): for macro in manifest.macros.values():
stats[macro.resource_type] += 1 stats[macro.resource_type] += 1
return stats return stats
@@ -171,15 +176,14 @@ class Compiler:
# a dict for jinja rendering of SQL # a dict for jinja rendering of SQL
def _create_node_context( def _create_node_context(
self, self,
node: ManifestSQLNode, node: NonSourceCompiledNode,
manifest: Manifest, manifest: Manifest,
extra_context: Dict[str, Any], extra_context: Dict[str, Any],
) -> Dict[str, Any]: ) -> Dict[str, Any]:
context = generate_runtime_model_context(node, self.config, manifest) context = generate_runtime_model_context(node, self.config, manifest)
context.update(extra_context) context.update(extra_context)
if isinstance(node, CompiledGenericTestNode):
if isinstance(node, GenericTestNode):
# for test nodes, add a special keyword args value to the context # for test nodes, add a special keyword args value to the context
jinja.add_rendered_test_kwargs(context, node) jinja.add_rendered_test_kwargs(context, node)
@@ -190,6 +194,14 @@ class Compiler:
relation_cls = adapter.Relation relation_cls = adapter.Relation
return relation_cls.add_ephemeral_prefix(name) return relation_cls.add_ephemeral_prefix(name)
def _get_relation_name(self, node: ParsedNode):
relation_name = None
if node.is_relational and not node.is_ephemeral_model:
adapter = get_adapter(self.config)
relation_cls = adapter.Relation
relation_name = str(relation_cls.create_from(self.config, node))
return relation_name
def _inject_ctes_into_sql(self, sql: str, ctes: List[InjectedCTE]) -> str: def _inject_ctes_into_sql(self, sql: str, ctes: List[InjectedCTE]) -> str:
""" """
`ctes` is a list of InjectedCTEs like: `ctes` is a list of InjectedCTEs like:
@@ -248,10 +260,10 @@ class Compiler:
def _recursively_prepend_ctes( def _recursively_prepend_ctes(
self, self,
model: ManifestSQLNode, model: NonSourceCompiledNode,
manifest: Manifest, manifest: Manifest,
extra_context: Optional[Dict[str, Any]], extra_context: Optional[Dict[str, Any]],
) -> Tuple[ManifestSQLNode, List[InjectedCTE]]: ) -> Tuple[NonSourceCompiledNode, List[InjectedCTE]]:
"""This method is called by the 'compile_node' method. Starting """This method is called by the 'compile_node' method. Starting
from the node that it is passed in, it will recursively call from the node that it is passed in, it will recursively call
itself using the 'extra_ctes'. The 'ephemeral' models do itself using the 'extra_ctes'. The 'ephemeral' models do
@@ -259,15 +271,14 @@ class Compiler:
are rolled up into the models that refer to them by are rolled up into the models that refer to them by
inserting CTEs into the SQL. inserting CTEs into the SQL.
""" """
if model.compiled_code is None: if model.compiled_sql is None:
raise DbtRuntimeError("Cannot inject ctes into an unparsed node", model) raise RuntimeException("Cannot inject ctes into an unparsed node", model)
if model.extra_ctes_injected: if model.extra_ctes_injected:
return (model, model.extra_ctes) return (model, model.extra_ctes)
# Just to make it plain that nothing is actually injected for this case # Just to make it plain that nothing is actually injected for this case
if not model.extra_ctes: if not model.extra_ctes:
if not isinstance(model, SeedNode): model.extra_ctes_injected = True
model.extra_ctes_injected = True
manifest.update_node(model) manifest.update_node(model)
return (model, model.extra_ctes) return (model, model.extra_ctes)
@@ -281,19 +292,20 @@ class Compiler:
# ephemeral model. # ephemeral model.
for cte in model.extra_ctes: for cte in model.extra_ctes:
if cte.id not in manifest.nodes: if cte.id not in manifest.nodes:
raise DbtInternalError( raise InternalException(
f"During compilation, found a cte reference that " f"During compilation, found a cte reference that "
f"could not be resolved: {cte.id}" f"could not be resolved: {cte.id}"
) )
cte_model = manifest.nodes[cte.id] cte_model = manifest.nodes[cte.id]
assert not isinstance(cte_model, SeedNode)
if not cte_model.is_ephemeral_model: if not cte_model.is_ephemeral_model:
raise DbtInternalError(f"{cte.id} is not ephemeral") raise InternalException(f"{cte.id} is not ephemeral")
# This model has already been compiled, so it's been # This model has already been compiled, so it's been
# through here before # through here before
if getattr(cte_model, "compiled", False): if getattr(cte_model, "compiled", False):
assert isinstance(cte_model, tuple(COMPILED_TYPES.values()))
cte_model = cast(NonSourceCompiledNode, cte_model)
new_prepended_ctes = cte_model.extra_ctes new_prepended_ctes = cte_model.extra_ctes
# if the cte_model isn't compiled, i.e. first time here # if the cte_model isn't compiled, i.e. first time here
@@ -312,78 +324,64 @@ class Compiler:
_extend_prepended_ctes(prepended_ctes, new_prepended_ctes) _extend_prepended_ctes(prepended_ctes, new_prepended_ctes)
new_cte_name = self.add_ephemeral_prefix(cte_model.name) new_cte_name = self.add_ephemeral_prefix(cte_model.name)
rendered_sql = cte_model._pre_injected_sql or cte_model.compiled_code rendered_sql = cte_model._pre_injected_sql or cte_model.compiled_sql
sql = f" {new_cte_name} as (\n{rendered_sql}\n)" sql = f" {new_cte_name} as (\n{rendered_sql}\n)"
_add_prepended_cte(prepended_ctes, InjectedCTE(id=cte.id, sql=sql)) _add_prepended_cte(prepended_ctes, InjectedCTE(id=cte.id, sql=sql))
injected_sql = self._inject_ctes_into_sql( injected_sql = self._inject_ctes_into_sql(
model.compiled_code, model.compiled_sql,
prepended_ctes, prepended_ctes,
) )
model._pre_injected_sql = model.compiled_code model._pre_injected_sql = model.compiled_sql
model.compiled_code = injected_sql model.compiled_sql = injected_sql
model.extra_ctes_injected = True model.extra_ctes_injected = True
model.extra_ctes = prepended_ctes model.extra_ctes = prepended_ctes
model.validate(model.to_dict(omit_none=True)) model.validate(model.to_dict(omit_none=True))
manifest.update_node(model) manifest.update_node(model)
return model, prepended_ctes return model, prepended_ctes
# Sets compiled fields in the ManifestSQLNode passed in, # creates a compiled_node from the ManifestNode passed in,
# creates a "context" dictionary for jinja rendering, # creates a "context" dictionary for jinja rendering,
# and then renders the "compiled_code" using the node, the # and then renders the "compiled_sql" using the node, the
# raw_code and the context. # raw_sql and the context.
def _compile_node( def _compile_node(
self, self,
node: ManifestSQLNode, node: ManifestNode,
manifest: Manifest, manifest: Manifest,
extra_context: Optional[Dict[str, Any]] = None, extra_context: Optional[Dict[str, Any]] = None,
) -> ManifestSQLNode: ) -> NonSourceCompiledNode:
if extra_context is None: if extra_context is None:
extra_context = {} extra_context = {}
fire_event(CompilingNode(unique_id=node.unique_id))
data = node.to_dict(omit_none=True) data = node.to_dict(omit_none=True)
data.update( data.update(
{ {
"compiled": False, "compiled": False,
"compiled_code": None, "compiled_sql": None,
"extra_ctes_injected": False, "extra_ctes_injected": False,
"extra_ctes": [], "extra_ctes": [],
} }
) )
compiled_node = _compiled_type_for(node).from_dict(data)
if node.language == ModelLanguage.python: context = self._create_node_context(compiled_node, manifest, extra_context)
# TODO could we also 'minify' this code at all? just aesthetic, not functional
# quoating seems like something very specific to sql so far compiled_node.compiled_sql = jinja.get_rendered(
# for all python implementations we are seeing there's no quating. node.raw_sql,
# TODO try to find better way to do this, given that context,
original_quoting = self.config.quoting node,
self.config.quoting = {key: False for key in original_quoting.keys()} )
context = self._create_node_context(node, manifest, extra_context)
postfix = jinja.get_rendered( compiled_node.relation_name = self._get_relation_name(node)
"{{ py_script_postfix(model) }}",
context,
node,
)
# we should NOT jinja render the python model's 'raw code'
node.compiled_code = f"{node.raw_code}\n\n{postfix}"
# restore quoting settings in the end since context is lazy evaluated
self.config.quoting = original_quoting
else: compiled_node.compiled = True
context = self._create_node_context(node, manifest, extra_context)
node.compiled_code = jinja.get_rendered(
node.raw_code,
context,
node,
)
node.compiled = True return compiled_node
return node
def write_graph_file(self, linker: Linker, manifest: Manifest): def write_graph_file(self, linker: Linker, manifest: Manifest):
filename = graph_file_name filename = graph_file_name
@@ -401,10 +399,8 @@ class Compiler:
linker.dependency(node.unique_id, (manifest.sources[dependency].unique_id)) linker.dependency(node.unique_id, (manifest.sources[dependency].unique_id))
elif dependency in manifest.metrics: elif dependency in manifest.metrics:
linker.dependency(node.unique_id, (manifest.metrics[dependency].unique_id)) linker.dependency(node.unique_id, (manifest.metrics[dependency].unique_id))
elif dependency in manifest.entities:
linker.dependency(node.unique_id, (manifest.entities[dependency].unique_id))
else: else:
raise GraphDependencyNotFoundError(node, dependency) dependency_not_found(node, dependency)
def link_graph(self, linker: Linker, manifest: Manifest, add_test_edges: bool = False): def link_graph(self, linker: Linker, manifest: Manifest, add_test_edges: bool = False):
for source in manifest.sources.values(): for source in manifest.sources.values():
@@ -415,8 +411,6 @@ class Compiler:
self.link_node(linker, exposure, manifest) self.link_node(linker, exposure, manifest)
for metric in manifest.metrics.values(): for metric in manifest.metrics.values():
self.link_node(linker, metric, manifest) self.link_node(linker, metric, manifest)
for entity in manifest.entities.values():
self.link_node(linker, entity, manifest)
cycle = linker.find_cycles() cycle = linker.find_cycles()
@@ -493,28 +487,25 @@ class Compiler:
return Graph(linker.graph) return Graph(linker.graph)
# writes the "compiled_code" into the target/compiled directory # writes the "compiled_sql" into the target/compiled directory
def _write_node(self, node: ManifestSQLNode) -> ManifestSQLNode: def _write_node(self, node: NonSourceCompiledNode) -> ManifestNode:
if not node.extra_ctes_injected or node.resource_type in ( if not node.extra_ctes_injected or node.resource_type == NodeType.Snapshot:
NodeType.Snapshot,
NodeType.Seed,
):
return node return node
fire_event(WritingInjectedSQLForNode(node_info=get_node_info())) fire_event(WritingInjectedSQLForNode(unique_id=node.unique_id))
if node.compiled_code: if node.compiled_sql:
node.compiled_path = node.write_node( node.compiled_path = node.write_node(
self.config.target_path, "compiled", node.compiled_code self.config.target_path, "compiled", node.compiled_sql
) )
return node return node
def compile_node( def compile_node(
self, self,
node: ManifestSQLNode, node: ManifestNode,
manifest: Manifest, manifest: Manifest,
extra_context: Optional[Dict[str, Any]] = None, extra_context: Optional[Dict[str, Any]] = None,
write: bool = True, write: bool = True,
) -> ManifestSQLNode: ) -> NonSourceCompiledNode:
"""This is the main entry point into this code. It's called by """This is the main entry point into this code. It's called by
CompileRunner.compile, GenericRPCRunner.compile, and CompileRunner.compile, GenericRPCRunner.compile, and
RunTask.get_hook_sql. It calls '_compile_node' to convert RunTask.get_hook_sql. It calls '_compile_node' to convert

View File

@@ -9,14 +9,12 @@ from dbt.clients.system import load_file_contents
from dbt.clients.yaml_helper import load_yaml_text from dbt.clients.yaml_helper import load_yaml_text
from dbt.contracts.connection import Credentials, HasCredentials from dbt.contracts.connection import Credentials, HasCredentials
from dbt.contracts.project import ProfileConfig, UserConfig from dbt.contracts.project import ProfileConfig, UserConfig
from dbt.exceptions import ( from dbt.exceptions import CompilationException
CompilationError, from dbt.exceptions import DbtProfileError
DbtProfileError, from dbt.exceptions import DbtProjectError
DbtProjectError, from dbt.exceptions import ValidationException
DbtValidationError, from dbt.exceptions import RuntimeException
DbtRuntimeError, from dbt.exceptions import validator_error_message
ProfileConfigError,
)
from dbt.events.types import MissingProfileTarget from dbt.events.types import MissingProfileTarget
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.utils import coerce_dict_str from dbt.utils import coerce_dict_str
@@ -25,6 +23,8 @@ from .renderer import ProfileRenderer
DEFAULT_THREADS = 1 DEFAULT_THREADS = 1
DEFAULT_PROFILES_DIR = os.path.join(os.path.expanduser("~"), ".dbt")
INVALID_PROFILE_MESSAGE = """ INVALID_PROFILE_MESSAGE = """
dbt encountered an error while trying to read your profiles.yml file. dbt encountered an error while trying to read your profiles.yml file.
@@ -44,7 +44,7 @@ defined in your profiles.yml file. You can find profiles.yml here:
{profiles_file}/profiles.yml {profiles_file}/profiles.yml
""".format( """.format(
profiles_file=flags.DEFAULT_PROFILES_DIR profiles_file=DEFAULT_PROFILES_DIR
) )
@@ -60,9 +60,9 @@ def read_profile(profiles_dir: str) -> Dict[str, Any]:
msg = f"The profiles.yml file at {path} is empty" msg = f"The profiles.yml file at {path} is empty"
raise DbtProfileError(INVALID_PROFILE_MESSAGE.format(error_string=msg)) raise DbtProfileError(INVALID_PROFILE_MESSAGE.format(error_string=msg))
return yaml_content return yaml_content
except DbtValidationError as e: except ValidationException as e:
msg = INVALID_PROFILE_MESSAGE.format(error_string=e) msg = INVALID_PROFILE_MESSAGE.format(error_string=e)
raise DbtValidationError(msg) from e raise ValidationException(msg) from e
return {} return {}
@@ -75,7 +75,7 @@ def read_user_config(directory: str) -> UserConfig:
if user_config is not None: if user_config is not None:
UserConfig.validate(user_config) UserConfig.validate(user_config)
return UserConfig.from_dict(user_config) return UserConfig.from_dict(user_config)
except (DbtRuntimeError, ValidationError): except (RuntimeException, ValidationError):
pass pass
return UserConfig() return UserConfig()
@@ -158,7 +158,7 @@ class Profile(HasCredentials):
dct = self.to_profile_info(serialize_credentials=True) dct = self.to_profile_info(serialize_credentials=True)
ProfileConfig.validate(dct) ProfileConfig.validate(dct)
except ValidationError as exc: except ValidationError as exc:
raise ProfileConfigError(exc) from exc raise DbtProfileError(validator_error_message(exc)) from exc
@staticmethod @staticmethod
def _credentials_from_profile( def _credentials_from_profile(
@@ -182,8 +182,8 @@ class Profile(HasCredentials):
data = cls.translate_aliases(profile) data = cls.translate_aliases(profile)
cls.validate(data) cls.validate(data)
credentials = cls.from_dict(data) credentials = cls.from_dict(data)
except (DbtRuntimeError, ValidationError) as e: except (RuntimeException, ValidationError) as e:
msg = str(e) if isinstance(e, DbtRuntimeError) else e.message msg = str(e) if isinstance(e, RuntimeException) else e.message
raise DbtProfileError( raise DbtProfileError(
'Credentials in profile "{}", target "{}" invalid: {}'.format( 'Credentials in profile "{}", target "{}" invalid: {}'.format(
profile_name, target_name, msg profile_name, target_name, msg
@@ -299,7 +299,7 @@ class Profile(HasCredentials):
try: try:
profile_data = renderer.render_data(raw_profile_data) profile_data = renderer.render_data(raw_profile_data)
except CompilationError as exc: except CompilationException as exc:
raise DbtProfileError(str(exc)) from exc raise DbtProfileError(str(exc)) from exc
return target_name, profile_data return target_name, profile_data

View File

@@ -16,19 +16,19 @@ import hashlib
import os import os
from dbt import flags, deprecations from dbt import flags, deprecations
from dbt.clients.system import path_exists, resolve_path_from_base, load_file_contents from dbt.clients.system import resolve_path_from_base
from dbt.clients.system import path_exists
from dbt.clients.system import load_file_contents
from dbt.clients.yaml_helper import load_yaml_text from dbt.clients.yaml_helper import load_yaml_text
from dbt.contracts.connection import QueryComment from dbt.contracts.connection import QueryComment
from dbt.exceptions import ( from dbt.exceptions import DbtProjectError
DbtProjectError, from dbt.exceptions import SemverException
SemverError, from dbt.exceptions import validator_error_message
ProjectContractBrokenError, from dbt.exceptions import RuntimeException
ProjectContractError,
DbtRuntimeError,
)
from dbt.graph import SelectionSpec from dbt.graph import SelectionSpec
from dbt.helper_types import NoValue from dbt.helper_types import NoValue
from dbt.semver import VersionSpecifier, versions_compatible from dbt.semver import VersionSpecifier
from dbt.semver import versions_compatible
from dbt.version import get_installed_version from dbt.version import get_installed_version
from dbt.utils import MultiDict from dbt.utils import MultiDict
from dbt.node_types import NodeType from dbt.node_types import NodeType
@@ -219,7 +219,7 @@ def _get_required_version(
try: try:
dbt_version = _parse_versions(dbt_raw_version) dbt_version = _parse_versions(dbt_raw_version)
except SemverError as e: except SemverException as e:
raise DbtProjectError(str(e)) from e raise DbtProjectError(str(e)) from e
if verify_version: if verify_version:
@@ -248,7 +248,7 @@ class PartialProject(RenderComponents):
project_name: Optional[str] = field( project_name: Optional[str] = field(
metadata=dict( metadata=dict(
description=( description=(
"The name of the project. This should always be set and will not be rendered" "The name of the project. This should always be set and will not " "be rendered"
) )
) )
) )
@@ -325,7 +325,7 @@ class PartialProject(RenderComponents):
ProjectContract.validate(rendered.project_dict) ProjectContract.validate(rendered.project_dict)
cfg = ProjectContract.from_dict(rendered.project_dict) cfg = ProjectContract.from_dict(rendered.project_dict)
except ValidationError as e: except ValidationError as e:
raise ProjectContractError(e) from e raise DbtProjectError(validator_error_message(e)) from e
# name/version are required in the Project definition, so we can assume # name/version are required in the Project definition, so we can assume
# they are present # they are present
name = cfg.name name = cfg.name
@@ -380,9 +380,6 @@ class PartialProject(RenderComponents):
snapshots: Dict[str, Any] snapshots: Dict[str, Any]
sources: Dict[str, Any] sources: Dict[str, Any]
tests: Dict[str, Any] tests: Dict[str, Any]
metrics: Dict[str, Any]
entities: Dict[str, Any]
exposures: Dict[str, Any]
vars_value: VarProvider vars_value: VarProvider
dispatch = cfg.dispatch dispatch = cfg.dispatch
@@ -391,9 +388,6 @@ class PartialProject(RenderComponents):
snapshots = cfg.snapshots snapshots = cfg.snapshots
sources = cfg.sources sources = cfg.sources
tests = cfg.tests tests = cfg.tests
metrics = cfg.metrics
entities = cfg.entities
exposures = cfg.exposures
if cfg.vars is None: if cfg.vars is None:
vars_dict: Dict[str, Any] = {} vars_dict: Dict[str, Any] = {}
else: else:
@@ -447,9 +441,6 @@ class PartialProject(RenderComponents):
query_comment=query_comment, query_comment=query_comment,
sources=sources, sources=sources,
tests=tests, tests=tests,
metrics=metrics,
entities=entities,
exposures=exposures,
vars=vars_value, vars=vars_value,
config_version=cfg.config_version, config_version=cfg.config_version,
unrendered=unrendered, unrendered=unrendered,
@@ -552,9 +543,6 @@ class Project:
snapshots: Dict[str, Any] snapshots: Dict[str, Any]
sources: Dict[str, Any] sources: Dict[str, Any]
tests: Dict[str, Any] tests: Dict[str, Any]
metrics: Dict[str, Any]
entities: Dict[str, Any]
exposures: Dict[str, Any]
vars: VarProvider vars: VarProvider
dbt_version: List[VersionSpecifier] dbt_version: List[VersionSpecifier]
packages: Dict[str, Any] packages: Dict[str, Any]
@@ -627,9 +615,6 @@ class Project:
"snapshots": self.snapshots, "snapshots": self.snapshots,
"sources": self.sources, "sources": self.sources,
"tests": self.tests, "tests": self.tests,
"metrics": self.metrics,
"entities": self.entities,
"exposures": self.exposures,
"vars": self.vars.to_dict(), "vars": self.vars.to_dict(),
"require-dbt-version": [v.to_version_string() for v in self.dbt_version], "require-dbt-version": [v.to_version_string() for v in self.dbt_version],
"config-version": self.config_version, "config-version": self.config_version,
@@ -647,7 +632,7 @@ class Project:
try: try:
ProjectContract.validate(self.to_project_config()) ProjectContract.validate(self.to_project_config())
except ValidationError as e: except ValidationError as e:
raise ProjectContractBrokenError(e) from e raise DbtProjectError(validator_error_message(e)) from e
@classmethod @classmethod
def partial_load(cls, project_root: str, *, verify_version: bool = False) -> PartialProject: def partial_load(cls, project_root: str, *, verify_version: bool = False) -> PartialProject:
@@ -672,8 +657,8 @@ class Project:
def get_selector(self, name: str) -> Union[SelectionSpec, bool]: def get_selector(self, name: str) -> Union[SelectionSpec, bool]:
if name not in self.selectors: if name not in self.selectors:
raise DbtRuntimeError( raise RuntimeException(
f"Could not find selector named {name}, expected one of {list(self.selectors)}" f"Could not find selector named {name}, expected one of " f"{list(self.selectors)}"
) )
return self.selectors[name]["definition"] return self.selectors[name]["definition"]

View File

@@ -3,13 +3,13 @@ import re
import os import os
from dbt.clients.jinja import get_rendered, catch_jinja from dbt.clients.jinja import get_rendered, catch_jinja
from dbt.constants import SECRET_ENV_PREFIX
from dbt.context.target import TargetContext from dbt.context.target import TargetContext
from dbt.context.secret import SecretContext, SECRET_PLACEHOLDER from dbt.context.secret import SecretContext, SECRET_PLACEHOLDER
from dbt.context.base import BaseContext from dbt.context.base import BaseContext
from dbt.contracts.connection import HasCredentials from dbt.contracts.connection import HasCredentials
from dbt.exceptions import DbtProjectError, CompilationError, RecursionError from dbt.exceptions import DbtProjectError, CompilationException, RecursionException
from dbt.utils import deep_map_render from dbt.utils import deep_map_render
from dbt.logger import SECRET_ENV_PREFIX
Keypath = Tuple[Union[str, int], ...] Keypath = Tuple[Union[str, int], ...]
@@ -40,14 +40,14 @@ class BaseRenderer:
try: try:
with catch_jinja(): with catch_jinja():
return get_rendered(value, self.context, native=True) return get_rendered(value, self.context, native=True)
except CompilationError as exc: except CompilationException as exc:
msg = f"Could not render {value}: {exc.msg}" msg = f"Could not render {value}: {exc.msg}"
raise CompilationError(msg) from exc raise CompilationException(msg) from exc
def render_data(self, data: Dict[str, Any]) -> Dict[str, Any]: def render_data(self, data: Dict[str, Any]) -> Dict[str, Any]:
try: try:
return deep_map_render(self.render_entry, data) return deep_map_render(self.render_entry, data)
except RecursionError: except RecursionException:
raise DbtProjectError( raise DbtProjectError(
f"Cycle detected: {self.name} input has a reference to itself", project=data f"Cycle detected: {self.name} input has a reference to itself", project=data
) )
@@ -159,8 +159,7 @@ class DbtProjectYamlRenderer(BaseRenderer):
if first in {"seeds", "models", "snapshots", "tests"}: if first in {"seeds", "models", "snapshots", "tests"}:
keypath_parts = {(k.lstrip("+ ") if isinstance(k, str) else k) for k in keypath} keypath_parts = {(k.lstrip("+ ") if isinstance(k, str) else k) for k in keypath}
# model-level hooks # model-level hooks
late_rendered_hooks = {"pre-hook", "post-hook", "pre_hook", "post_hook"} if "pre-hook" in keypath_parts or "post-hook" in keypath_parts:
if keypath_parts.intersection(late_rendered_hooks):
return False return False
return True return True

View File

@@ -8,6 +8,7 @@ from typing import (
Dict, Dict,
Iterable, Iterable,
Iterator, Iterator,
List,
Mapping, Mapping,
MutableSet, MutableSet,
Optional, Optional,
@@ -25,15 +26,14 @@ from dbt.contracts.project import Configuration, UserConfig
from dbt.contracts.relation import ComponentName from dbt.contracts.relation import ComponentName
from dbt.dataclass_schema import ValidationError from dbt.dataclass_schema import ValidationError
from dbt.exceptions import ( from dbt.exceptions import (
ConfigContractBrokenError,
DbtProjectError, DbtProjectError,
NonUniquePackageNameError, RuntimeException,
DbtRuntimeError, raise_compiler_error,
UninstalledPackagesFoundError, validator_error_message,
warn_or_error,
) )
from dbt.events.functions import warn_or_error
from dbt.events.types import UnusedResourceConfigPath
from dbt.helper_types import DictDefaultEmptyStr, FQNPath, PathSet from dbt.helper_types import DictDefaultEmptyStr, FQNPath, PathSet
from dbt.ui import warning_tag
from .profile import Profile from .profile import Profile
from .project import Project, PartialProject from .project import Project, PartialProject
@@ -116,9 +116,6 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
query_comment=project.query_comment, query_comment=project.query_comment,
sources=project.sources, sources=project.sources,
tests=project.tests, tests=project.tests,
metrics=project.metrics,
entities=project.entities,
exposures=project.exposures,
vars=project.vars, vars=project.vars,
config_version=project.config_version, config_version=project.config_version,
unrendered=project.unrendered, unrendered=project.unrendered,
@@ -188,7 +185,7 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
try: try:
Configuration.validate(self.serialize()) Configuration.validate(self.serialize())
except ValidationError as e: except ValidationError as e:
raise ConfigContractBrokenError(e) from e raise DbtProjectError(validator_error_message(e)) from e
@classmethod @classmethod
def _get_rendered_profile( def _get_rendered_profile(
@@ -259,7 +256,7 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
:param args: The arguments as parsed from the cli. :param args: The arguments as parsed from the cli.
:raises DbtProjectError: If the project is invalid or missing. :raises DbtProjectError: If the project is invalid or missing.
:raises DbtProfileError: If the profile is invalid or missing. :raises DbtProfileError: If the profile is invalid or missing.
:raises DbtValidationError: If the cli variables are invalid. :raises ValidationException: If the cli variables are invalid.
""" """
project, profile = cls.collect_parts(args) project, profile = cls.collect_parts(args)
@@ -312,16 +309,13 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
"snapshots": self._get_config_paths(self.snapshots), "snapshots": self._get_config_paths(self.snapshots),
"sources": self._get_config_paths(self.sources), "sources": self._get_config_paths(self.sources),
"tests": self._get_config_paths(self.tests), "tests": self._get_config_paths(self.tests),
"metrics": self._get_config_paths(self.metrics),
"entities": self._get_config_paths(self.entities),
"exposures": self._get_config_paths(self.exposures),
} }
def warn_for_unused_resource_config_paths( def get_unused_resource_config_paths(
self, self,
resource_fqns: Mapping[str, PathSet], resource_fqns: Mapping[str, PathSet],
disabled: PathSet, disabled: PathSet,
) -> None: ) -> List[FQNPath]:
"""Return a list of lists of strings, where each inner list of strings """Return a list of lists of strings, where each inner list of strings
represents a type + FQN path of a resource configuration that is not represents a type + FQN path of a resource configuration that is not
used. used.
@@ -335,13 +329,23 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
for config_path in config_paths: for config_path in config_paths:
if not _is_config_used(config_path, fqns): if not _is_config_used(config_path, fqns):
resource_path = ".".join(i for i in ((resource_type,) + config_path)) unused_resource_config_paths.append((resource_type,) + config_path)
unused_resource_config_paths.append(resource_path) return unused_resource_config_paths
if len(unused_resource_config_paths) == 0: def warn_for_unused_resource_config_paths(
self,
resource_fqns: Mapping[str, PathSet],
disabled: PathSet,
) -> None:
unused = self.get_unused_resource_config_paths(resource_fqns, disabled)
if len(unused) == 0:
return return
warn_or_error(UnusedResourceConfigPath(unused_config_paths=unused_resource_config_paths)) msg = UNUSED_RESOURCE_CONFIGURATION_PATH_MESSAGE.format(
len(unused), "\n".join("- {}".format(".".join(u)) for u in unused)
)
warn_or_error(msg, log_fmt=warning_tag("{}"))
def load_dependencies(self, base_only=False) -> Mapping[str, "RuntimeConfig"]: def load_dependencies(self, base_only=False) -> Mapping[str, "RuntimeConfig"]:
if self.dependencies is None: if self.dependencies is None:
@@ -355,15 +359,22 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
count_packages_specified = len(self.packages.packages) # type: ignore count_packages_specified = len(self.packages.packages) # type: ignore
count_packages_installed = len(tuple(self._get_project_directories())) count_packages_installed = len(tuple(self._get_project_directories()))
if count_packages_specified > count_packages_installed: if count_packages_specified > count_packages_installed:
raise UninstalledPackagesFoundError( raise_compiler_error(
count_packages_specified, f"dbt found {count_packages_specified} package(s) "
count_packages_installed, f"specified in packages.yml, but only "
self.packages_install_path, f"{count_packages_installed} package(s) installed "
f'in {self.packages_install_path}. Run "dbt deps" to '
f"install package dependencies."
) )
project_paths = itertools.chain(internal_packages, self._get_project_directories()) project_paths = itertools.chain(internal_packages, self._get_project_directories())
for project_name, project in self.load_projects(project_paths): for project_name, project in self.load_projects(project_paths):
if project_name in all_projects: if project_name in all_projects:
raise NonUniquePackageNameError(project_name) raise_compiler_error(
f"dbt found more than one package with the name "
f'"{project_name}" included in this project. Package '
f"names must be unique in a project. Please rename "
f"one of these packages."
)
all_projects[project_name] = project all_projects[project_name] = project
self.dependencies = all_projects self.dependencies = all_projects
return self.dependencies return self.dependencies
@@ -428,7 +439,7 @@ class UnsetProfile(Profile):
def __getattribute__(self, name): def __getattribute__(self, name):
if name in {"profile_name", "target_name", "threads"}: if name in {"profile_name", "target_name", "threads"}:
raise DbtRuntimeError(f'Error: disallowed attribute "{name}" - no profile!') raise RuntimeException(f'Error: disallowed attribute "{name}" - no profile!')
return Profile.__getattribute__(self, name) return Profile.__getattribute__(self, name)
@@ -455,7 +466,7 @@ class UnsetProfileConfig(RuntimeConfig):
def __getattribute__(self, name): def __getattribute__(self, name):
# Override __getattribute__ to check that the attribute isn't 'banned'. # Override __getattribute__ to check that the attribute isn't 'banned'.
if name in {"profile_name", "target_name"}: if name in {"profile_name", "target_name"}:
raise DbtRuntimeError(f'Error: disallowed attribute "{name}" - no profile!') raise RuntimeException(f'Error: disallowed attribute "{name}" - no profile!')
# avoid every attribute access triggering infinite recursion # avoid every attribute access triggering infinite recursion
return RuntimeConfig.__getattribute__(self, name) return RuntimeConfig.__getattribute__(self, name)
@@ -501,9 +512,6 @@ class UnsetProfileConfig(RuntimeConfig):
"snapshots": self.snapshots, "snapshots": self.snapshots,
"sources": self.sources, "sources": self.sources,
"tests": self.tests, "tests": self.tests,
"metrics": self.metrics,
"entities": self.entities,
"exposures": self.exposures,
"vars": self.vars.to_dict(), "vars": self.vars.to_dict(),
"require-dbt-version": [v.to_version_string() for v in self.dbt_version], "require-dbt-version": [v.to_version_string() for v in self.dbt_version],
"config-version": self.config_version, "config-version": self.config_version,
@@ -564,9 +572,6 @@ class UnsetProfileConfig(RuntimeConfig):
query_comment=project.query_comment, query_comment=project.query_comment,
sources=project.sources, sources=project.sources,
tests=project.tests, tests=project.tests,
metrics=project.metrics,
entities=project.entities,
exposures=project.exposures,
vars=project.vars, vars=project.vars,
config_version=project.config_version, config_version=project.config_version,
unrendered=project.unrendered, unrendered=project.unrendered,
@@ -606,13 +611,21 @@ class UnsetProfileConfig(RuntimeConfig):
:param args: The arguments as parsed from the cli. :param args: The arguments as parsed from the cli.
:raises DbtProjectError: If the project is invalid or missing. :raises DbtProjectError: If the project is invalid or missing.
:raises DbtProfileError: If the profile is invalid or missing. :raises DbtProfileError: If the profile is invalid or missing.
:raises DbtValidationError: If the cli variables are invalid. :raises ValidationException: If the cli variables are invalid.
""" """
project, profile = cls.collect_parts(args) project, profile = cls.collect_parts(args)
return cls.from_parts(project=project, profile=profile, args=args) return cls.from_parts(project=project, profile=profile, args=args)
UNUSED_RESOURCE_CONFIGURATION_PATH_MESSAGE = """\
Configuration paths exist in your dbt_project.yml file which do not \
apply to any resources.
There are {} unused configuration paths:
{}
"""
def _is_config_used(path, fqns): def _is_config_used(path, fqns):
if fqns: if fqns:
for fqn in fqns: for fqn in fqns:

View File

@@ -12,7 +12,7 @@ from dbt.clients.system import (
resolve_path_from_base, resolve_path_from_base,
) )
from dbt.contracts.selection import SelectorFile from dbt.contracts.selection import SelectorFile
from dbt.exceptions import DbtSelectorsError, DbtRuntimeError from dbt.exceptions import DbtSelectorsError, RuntimeException
from dbt.graph import parse_from_selectors_definition, SelectionSpec from dbt.graph import parse_from_selectors_definition, SelectionSpec
from dbt.graph.selector_spec import SelectionCriteria from dbt.graph.selector_spec import SelectionCriteria
@@ -46,7 +46,7 @@ class SelectorConfig(Dict[str, Dict[str, Union[SelectionSpec, bool]]]):
f"yaml-selectors", f"yaml-selectors",
result_type="invalid_selector", result_type="invalid_selector",
) from exc ) from exc
except DbtRuntimeError as exc: except RuntimeException as exc:
raise DbtSelectorsError( raise DbtSelectorsError(
f"Could not read selector file data: {exc}", f"Could not read selector file data: {exc}",
result_type="invalid_selector", result_type="invalid_selector",
@@ -62,7 +62,7 @@ class SelectorConfig(Dict[str, Dict[str, Union[SelectionSpec, bool]]]):
) -> "SelectorConfig": ) -> "SelectorConfig":
try: try:
rendered = renderer.render_data(data) rendered = renderer.render_data(data)
except (ValidationError, DbtRuntimeError) as exc: except (ValidationError, RuntimeException) as exc:
raise DbtSelectorsError( raise DbtSelectorsError(
f"Could not render selector data: {exc}", f"Could not render selector data: {exc}",
result_type="invalid_selector", result_type="invalid_selector",
@@ -77,7 +77,7 @@ class SelectorConfig(Dict[str, Dict[str, Union[SelectionSpec, bool]]]):
) -> "SelectorConfig": ) -> "SelectorConfig":
try: try:
data = load_yaml_text(load_file_contents(str(path))) data = load_yaml_text(load_file_contents(str(path)))
except (ValidationError, DbtRuntimeError) as exc: except (ValidationError, RuntimeException) as exc:
raise DbtSelectorsError( raise DbtSelectorsError(
f"Could not read selector file: {exc}", f"Could not read selector file: {exc}",
result_type="invalid_selector", result_type="invalid_selector",

View File

@@ -8,24 +8,24 @@ from dbt.clients import yaml_helper
from dbt.config import Profile, Project, read_user_config from dbt.config import Profile, Project, read_user_config
from dbt.config.renderer import DbtProjectYamlRenderer, ProfileRenderer from dbt.config.renderer import DbtProjectYamlRenderer, ProfileRenderer
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import InvalidOptionYAML from dbt.events.types import InvalidVarsYAML
from dbt.exceptions import DbtValidationError, OptionNotYamlDictError from dbt.exceptions import ValidationException, raise_compiler_error
def parse_cli_vars(var_string: str) -> Dict[str, Any]: def parse_cli_vars(var_string: str) -> Dict[str, Any]:
return parse_cli_yaml_string(var_string, "vars")
def parse_cli_yaml_string(var_string: str, cli_option_name: str) -> Dict[str, Any]:
try: try:
cli_vars = yaml_helper.load_yaml_text(var_string) cli_vars = yaml_helper.load_yaml_text(var_string)
var_type = type(cli_vars) var_type = type(cli_vars)
if var_type is dict: if var_type is dict:
return cli_vars return cli_vars
else: else:
raise OptionNotYamlDictError(var_type, cli_option_name) type_name = var_type.__name__
except DbtValidationError: raise_compiler_error(
fire_event(InvalidOptionYAML(option_name=cli_option_name)) "The --vars argument must be a YAML dictionary, but was "
"of type '{}'".format(type_name)
)
except ValidationException:
fire_event(InvalidVarsYAML())
raise raise

View File

@@ -1,10 +0,0 @@
SECRET_ENV_PREFIX = "DBT_ENV_SECRET_"
DEFAULT_ENV_PLACEHOLDER = "DBT_DEFAULT_PLACEHOLDER"
METADATA_ENV_PREFIX = "DBT_ENV_CUSTOM_ENV_"
MAXIMUM_SEED_SIZE = 1 * 1024 * 1024
MAXIMUM_SEED_SIZE_NAME = "1MB"
PIN_PACKAGE_URL = (
"https://docs.getdbt.com/docs/package-management#section-specifying-package-versions"
)

View File

@@ -2,7 +2,7 @@
Contexts are used for Jinja rendering. They include context methods, executable macros, and various settings that are available in Jinja. Contexts are used for Jinja rendering. They include context methods, executable macros, and various settings that are available in Jinja.
The most common entrypoint to Jinja rendering in dbt is a method named `get_rendered`, which takes two arguments: templated code (string), and a context used to render it (dictionary). The most common entrypoint to Jinja rendering in dbt is a method named `get_rendered`, which takes two arguments: templated code (string), and a context used to render it (dictionary).
The context is the bundle of information that is in "scope" when rendering Jinja-templated code. For instance, imagine a simple Jinja template: The context is the bundle of information that is in "scope" when rendering Jinja-templated code. For instance, imagine a simple Jinja template:
``` ```

View File

@@ -4,22 +4,19 @@ from typing import Any, Dict, NoReturn, Optional, Mapping, Iterable, Set, List
from dbt import flags from dbt import flags
from dbt import tracking from dbt import tracking
from dbt import utils
from dbt.clients.jinja import get_rendered from dbt.clients.jinja import get_rendered
from dbt.clients.yaml_helper import yaml, safe_load, SafeLoader, Loader, Dumper # noqa: F401 from dbt.clients.yaml_helper import yaml, safe_load, SafeLoader, Loader, Dumper # noqa: F401
from dbt.constants import SECRET_ENV_PREFIX, DEFAULT_ENV_PLACEHOLDER from dbt.contracts.graph.compiled import CompiledResource
from dbt.contracts.graph.nodes import Resource
from dbt.exceptions import ( from dbt.exceptions import (
SecretEnvVarLocationError, CompilationException,
EnvVarMissingError,
MacroReturn, MacroReturn,
RequiredVarNotFoundError, raise_compiler_error,
SetStrictWrongTypeError, raise_parsing_error,
ZipStrictWrongTypeError, disallow_secret_env_var,
) )
from dbt.logger import SECRET_ENV_PREFIX
from dbt.events.functions import fire_event, get_invocation_id from dbt.events.functions import fire_event, get_invocation_id
from dbt.events.types import JinjaLogInfo, JinjaLogDebug from dbt.events.types import MacroEventInfo, MacroEventDebug
from dbt.events.contextvars import get_node_info
from dbt.version import __version__ as dbt_version from dbt.version import __version__ as dbt_version
# These modules are added to the context. Consider alternative # These modules are added to the context. Consider alternative
@@ -129,17 +126,18 @@ class ContextMeta(type):
class Var: class Var:
UndefinedVarError = "Required var '{}' not found in config:\nVars " "supplied to {} = {}"
_VAR_NOTSET = object() _VAR_NOTSET = object()
def __init__( def __init__(
self, self,
context: Mapping[str, Any], context: Mapping[str, Any],
cli_vars: Mapping[str, Any], cli_vars: Mapping[str, Any],
node: Optional[Resource] = None, node: Optional[CompiledResource] = None,
) -> None: ) -> None:
self._context: Mapping[str, Any] = context self._context: Mapping[str, Any] = context
self._cli_vars: Mapping[str, Any] = cli_vars self._cli_vars: Mapping[str, Any] = cli_vars
self._node: Optional[Resource] = node self._node: Optional[CompiledResource] = node
self._merged: Mapping[str, Any] = self._generate_merged() self._merged: Mapping[str, Any] = self._generate_merged()
def _generate_merged(self) -> Mapping[str, Any]: def _generate_merged(self) -> Mapping[str, Any]:
@@ -153,7 +151,10 @@ class Var:
return "<Configuration>" return "<Configuration>"
def get_missing_var(self, var_name): def get_missing_var(self, var_name):
raise RequiredVarNotFoundError(var_name, self._merged, self._node) dct = {k: self._merged[k] for k in self._merged}
pretty_vars = json.dumps(dct, sort_keys=True, indent=4)
msg = self.UndefinedVarError.format(var_name, self.node_name, pretty_vars)
raise_compiler_error(msg, self._node)
def has_var(self, var_name: str): def has_var(self, var_name: str):
return var_name in self._merged return var_name in self._merged
@@ -297,22 +298,18 @@ class BaseContext(metaclass=ContextMeta):
""" """
return_value = None return_value = None
if var.startswith(SECRET_ENV_PREFIX): if var.startswith(SECRET_ENV_PREFIX):
raise SecretEnvVarLocationError(var) disallow_secret_env_var(var)
if var in os.environ: if var in os.environ:
return_value = os.environ[var] return_value = os.environ[var]
elif default is not None: elif default is not None:
return_value = default return_value = default
if return_value is not None: if return_value is not None:
# If the environment variable is set from a default, store a string indicating self.env_vars[var] = return_value
# that so we can skip partial parsing. Otherwise the file will be scheduled for
# reparsing. If the default changes, the file will have been updated and therefore
# will be scheduled for reparsing anyways.
self.env_vars[var] = return_value if var in os.environ else DEFAULT_ENV_PLACEHOLDER
return return_value return return_value
else: else:
raise EnvVarMissingError(var) msg = f"Env var required but not provided: '{var}'"
raise_parsing_error(msg)
if os.environ.get("DBT_MACRO_DEBUGGING"): if os.environ.get("DBT_MACRO_DEBUGGING"):
@@ -493,7 +490,7 @@ class BaseContext(metaclass=ContextMeta):
try: try:
return set(value) return set(value)
except TypeError as e: except TypeError as e:
raise SetStrictWrongTypeError(e) raise CompilationException(e)
@contextmember("zip") @contextmember("zip")
@staticmethod @staticmethod
@@ -537,7 +534,7 @@ class BaseContext(metaclass=ContextMeta):
try: try:
return zip(*args) return zip(*args)
except TypeError as e: except TypeError as e:
raise ZipStrictWrongTypeError(e) raise CompilationException(e)
@contextmember @contextmember
@staticmethod @staticmethod
@@ -555,9 +552,9 @@ class BaseContext(metaclass=ContextMeta):
{% endmacro %}" {% endmacro %}"
""" """
if info: if info:
fire_event(JinjaLogInfo(msg=msg, node_info=get_node_info())) fire_event(MacroEventInfo(msg=msg))
else: else:
fire_event(JinjaLogDebug(msg=msg, node_info=get_node_info())) fire_event(MacroEventDebug(msg=msg))
return "" return ""
@contextproperty @contextproperty
@@ -634,8 +631,9 @@ class BaseContext(metaclass=ContextMeta):
{% endif %} {% endif %}
This supports all flags defined in flags submodule (core/dbt/flags.py) This supports all flags defined in flags submodule (core/dbt/flags.py)
TODO: Replace with object that provides read-only access to flag values
""" """
return flags.get_flag_obj() return flags
@contextmember @contextmember
@staticmethod @staticmethod
@@ -684,19 +682,6 @@ class BaseContext(metaclass=ContextMeta):
dict_diff.update({k: dict_a[k]}) dict_diff.update({k: dict_a[k]})
return dict_diff return dict_diff
@contextmember
@staticmethod
def local_md5(value: str) -> str:
"""Calculates an MD5 hash of the given string.
It's called "local_md5" to emphasize that it runs locally in dbt (in jinja context) and not an MD5 SQL command.
:param value: The value to hash
Usage:
{% set value_hash = local_md5("hello world") %}
"""
return utils.md5(value)
def generate_base_context(cli_vars: Dict[str, Any]) -> Dict[str, Any]: def generate_base_context(cli_vars: Dict[str, Any]) -> Dict[str, Any]:
ctx = BaseContext(cli_vars) ctx = BaseContext(cli_vars)

View File

@@ -1,14 +1,14 @@
import os import os
from typing import Any, Dict, Optional from typing import Any, Dict, Optional
from dbt.constants import SECRET_ENV_PREFIX, DEFAULT_ENV_PLACEHOLDER
from dbt.contracts.connection import AdapterRequiredConfig from dbt.contracts.connection import AdapterRequiredConfig
from dbt.logger import SECRET_ENV_PREFIX
from dbt.node_types import NodeType from dbt.node_types import NodeType
from dbt.utils import MultiDict from dbt.utils import MultiDict
from dbt.context.base import contextproperty, contextmember, Var from dbt.context.base import contextproperty, contextmember, Var
from dbt.context.target import TargetContext from dbt.context.target import TargetContext
from dbt.exceptions import EnvVarMissingError, SecretEnvVarLocationError from dbt.exceptions import raise_parsing_error, disallow_secret_env_var
class ConfiguredContext(TargetContext): class ConfiguredContext(TargetContext):
@@ -86,7 +86,7 @@ class SchemaYamlContext(ConfiguredContext):
def env_var(self, var: str, default: Optional[str] = None) -> str: def env_var(self, var: str, default: Optional[str] = None) -> str:
return_value = None return_value = None
if var.startswith(SECRET_ENV_PREFIX): if var.startswith(SECRET_ENV_PREFIX):
raise SecretEnvVarLocationError(var) disallow_secret_env_var(var)
if var in os.environ: if var in os.environ:
return_value = os.environ[var] return_value = os.environ[var]
elif default is not None: elif default is not None:
@@ -94,17 +94,11 @@ class SchemaYamlContext(ConfiguredContext):
if return_value is not None: if return_value is not None:
if self.schema_yaml_vars: if self.schema_yaml_vars:
# If the environment variable is set from a default, store a string indicating self.schema_yaml_vars.env_vars[var] = return_value
# that so we can skip partial parsing. Otherwise the file will be scheduled for
# reparsing. If the default changes, the file will have been updated and therefore
# will be scheduled for reparsing anyways.
self.schema_yaml_vars.env_vars[var] = (
return_value if var in os.environ else DEFAULT_ENV_PLACEHOLDER
)
return return_value return return_value
else: else:
raise EnvVarMissingError(var) msg = f"Env var required but not provided: '{var}'"
raise_parsing_error(msg)
class MacroResolvingContext(ConfiguredContext): class MacroResolvingContext(ConfiguredContext):

View File

@@ -5,7 +5,7 @@ from typing import List, Iterator, Dict, Any, TypeVar, Generic
from dbt.config import RuntimeConfig, Project, IsFQNResource from dbt.config import RuntimeConfig, Project, IsFQNResource
from dbt.contracts.graph.model_config import BaseConfig, get_config_for, _listify from dbt.contracts.graph.model_config import BaseConfig, get_config_for, _listify
from dbt.exceptions import DbtInternalError from dbt.exceptions import InternalException
from dbt.node_types import NodeType from dbt.node_types import NodeType
from dbt.utils import fqn_search from dbt.utils import fqn_search
@@ -43,14 +43,9 @@ class UnrenderedConfig(ConfigSource):
model_configs = unrendered.get("sources") model_configs = unrendered.get("sources")
elif resource_type == NodeType.Test: elif resource_type == NodeType.Test:
model_configs = unrendered.get("tests") model_configs = unrendered.get("tests")
elif resource_type == NodeType.Metric:
model_configs = unrendered.get("metrics")
elif resource_type == NodeType.Entity:
model_configs = unrendered.get("entities")
elif resource_type == NodeType.Exposure:
model_configs = unrendered.get("exposures")
else: else:
model_configs = unrendered.get("models") model_configs = unrendered.get("models")
if model_configs is None: if model_configs is None:
return {} return {}
else: else:
@@ -70,12 +65,6 @@ class RenderedConfig(ConfigSource):
model_configs = self.project.sources model_configs = self.project.sources
elif resource_type == NodeType.Test: elif resource_type == NodeType.Test:
model_configs = self.project.tests model_configs = self.project.tests
elif resource_type == NodeType.Metric:
model_configs = self.project.metrics
elif resource_type == NodeType.Entity:
model_configs = self.project.entities
elif resource_type == NodeType.Exposure:
model_configs = self.project.exposures
else: else:
model_configs = self.project.models model_configs = self.project.models
return model_configs return model_configs
@@ -93,7 +82,7 @@ class BaseContextConfigGenerator(Generic[T]):
return self._active_project return self._active_project
dependencies = self._active_project.load_dependencies() dependencies = self._active_project.load_dependencies()
if project_name not in dependencies: if project_name not in dependencies:
raise DbtInternalError( raise InternalException(
f"Project name {project_name} not found in dependencies " f"Project name {project_name} not found in dependencies "
f"(found {list(dependencies)})" f"(found {list(dependencies)})"
) )
@@ -291,14 +280,14 @@ class ContextConfig:
elif k in BaseConfig.mergebehavior["update"]: elif k in BaseConfig.mergebehavior["update"]:
if not isinstance(v, dict): if not isinstance(v, dict):
raise DbtInternalError(f"expected dict, got {v}") raise InternalException(f"expected dict, got {v}")
if k in config_call_dict and isinstance(config_call_dict[k], dict): if k in config_call_dict and isinstance(config_call_dict[k], dict):
config_call_dict[k].update(v) config_call_dict[k].update(v)
else: else:
config_call_dict[k] = v config_call_dict[k] = v
elif k in BaseConfig.mergebehavior["dict_key_append"]: elif k in BaseConfig.mergebehavior["dict_key_append"]:
if not isinstance(v, dict): if not isinstance(v, dict):
raise DbtInternalError(f"expected dict, got {v}") raise InternalException(f"expected dict, got {v}")
if k in config_call_dict: # should always be a dict if k in config_call_dict: # should always be a dict
for key, value in v.items(): for key, value in v.items():
extend = False extend = False

View File

@@ -1,12 +1,13 @@
from typing import Any, Dict, Union from typing import Any, Dict, Union
from dbt.exceptions import ( from dbt.exceptions import (
DocTargetNotFoundError, doc_invalid_args,
DocArgsError, doc_target_not_found,
) )
from dbt.config.runtime import RuntimeConfig from dbt.config.runtime import RuntimeConfig
from dbt.contracts.graph.compiled import CompileResultNode
from dbt.contracts.graph.manifest import Manifest from dbt.contracts.graph.manifest import Manifest
from dbt.contracts.graph.nodes import Macro, ResultNode from dbt.contracts.graph.parsed import ParsedMacro
from dbt.context.base import contextmember from dbt.context.base import contextmember
from dbt.context.configured import SchemaYamlContext from dbt.context.configured import SchemaYamlContext
@@ -16,7 +17,7 @@ class DocsRuntimeContext(SchemaYamlContext):
def __init__( def __init__(
self, self,
config: RuntimeConfig, config: RuntimeConfig,
node: Union[Macro, ResultNode], node: Union[ParsedMacro, CompileResultNode],
manifest: Manifest, manifest: Manifest,
current_project: str, current_project: str,
) -> None: ) -> None:
@@ -52,9 +53,9 @@ class DocsRuntimeContext(SchemaYamlContext):
elif len(args) == 2: elif len(args) == 2:
doc_package_name, doc_name = args doc_package_name, doc_name = args
else: else:
raise DocArgsError(self.node, args) doc_invalid_args(self.node, args)
# Documentation # ParsedDocumentation
target_doc = self.manifest.resolve_doc( target_doc = self.manifest.resolve_doc(
doc_name, doc_name,
doc_package_name, doc_package_name,
@@ -68,9 +69,7 @@ class DocsRuntimeContext(SchemaYamlContext):
# TODO CT-211 # TODO CT-211
source_file.add_node(self.node.unique_id) # type: ignore[union-attr] source_file.add_node(self.node.unique_id) # type: ignore[union-attr]
else: else:
raise DocTargetNotFoundError( doc_target_not_found(self.node, doc_name, doc_package_name)
node=self.node, target_doc_name=doc_name, target_doc_package=doc_package_name
)
return target_doc.block_contents return target_doc.block_contents

View File

@@ -1,144 +0,0 @@
import functools
from typing import NoReturn
from dbt.events.functions import warn_or_error
from dbt.events.helpers import env_secrets, scrub_secrets
from dbt.events.types import JinjaLogWarning
from dbt.exceptions import (
DbtRuntimeError,
MissingConfigError,
MissingMaterializationError,
MissingRelationError,
AmbiguousAliasError,
AmbiguousCatalogMatchError,
CacheInconsistencyError,
DataclassNotDictError,
CompilationError,
DbtDatabaseError,
DependencyNotFoundError,
DependencyError,
DuplicatePatchPathError,
DuplicateResourceNameError,
PropertyYMLError,
NotImplementedError,
RelationWrongTypeError,
)
def warn(msg, node=None):
warn_or_error(JinjaLogWarning(msg=msg), node=node)
return ""
def missing_config(model, name) -> NoReturn:
raise MissingConfigError(unique_id=model.unique_id, name=name)
def missing_materialization(model, adapter_type) -> NoReturn:
raise MissingMaterializationError(
materialization=model.config.materialized, adapter_type=adapter_type
)
def missing_relation(relation, model=None) -> NoReturn:
raise MissingRelationError(relation, model)
def raise_ambiguous_alias(node_1, node_2, duped_name=None) -> NoReturn:
raise AmbiguousAliasError(node_1, node_2, duped_name)
def raise_ambiguous_catalog_match(unique_id, match_1, match_2) -> NoReturn:
raise AmbiguousCatalogMatchError(unique_id, match_1, match_2)
def raise_cache_inconsistent(message) -> NoReturn:
raise CacheInconsistencyError(message)
def raise_dataclass_not_dict(obj) -> NoReturn:
raise DataclassNotDictError(obj)
def raise_compiler_error(msg, node=None) -> NoReturn:
raise CompilationError(msg, node)
def raise_database_error(msg, node=None) -> NoReturn:
raise DbtDatabaseError(msg, node)
def raise_dep_not_found(node, node_description, required_pkg) -> NoReturn:
raise DependencyNotFoundError(node, node_description, required_pkg)
def raise_dependency_error(msg) -> NoReturn:
raise DependencyError(scrub_secrets(msg, env_secrets()))
def raise_duplicate_patch_name(patch_1, existing_patch_path) -> NoReturn:
raise DuplicatePatchPathError(patch_1, existing_patch_path)
def raise_duplicate_resource_name(node_1, node_2) -> NoReturn:
raise DuplicateResourceNameError(node_1, node_2)
def raise_invalid_property_yml_version(path, issue) -> NoReturn:
raise PropertyYMLError(path, issue)
def raise_not_implemented(msg) -> NoReturn:
raise NotImplementedError(msg)
def relation_wrong_type(relation, expected_type, model=None) -> NoReturn:
raise RelationWrongTypeError(relation, expected_type, model)
# Update this when a new function should be added to the
# dbt context's `exceptions` key!
CONTEXT_EXPORTS = {
fn.__name__: fn
for fn in [
warn,
missing_config,
missing_materialization,
missing_relation,
raise_ambiguous_alias,
raise_ambiguous_catalog_match,
raise_cache_inconsistent,
raise_dataclass_not_dict,
raise_compiler_error,
raise_database_error,
raise_dep_not_found,
raise_dependency_error,
raise_duplicate_patch_name,
raise_duplicate_resource_name,
raise_invalid_property_yml_version,
raise_not_implemented,
relation_wrong_type,
]
}
# wraps context based exceptions in node info
def wrapper(model):
def wrap(func):
@functools.wraps(func)
def inner(*args, **kwargs):
try:
return func(*args, **kwargs)
except DbtRuntimeError as exc:
exc.add_node(model)
raise exc
return inner
return wrap
def wrapped_exports(model):
wrap = wrapper(model)
return {name: wrap(export) for name, export in CONTEXT_EXPORTS.items()}

View File

@@ -1,10 +1,10 @@
from typing import Dict, MutableMapping, Optional from typing import Dict, MutableMapping, Optional
from dbt.contracts.graph.nodes import Macro from dbt.contracts.graph.parsed import ParsedMacro
from dbt.exceptions import DuplicateMacroNameError, PackageNotFoundForMacroError from dbt.exceptions import raise_duplicate_macro_name, raise_compiler_error
from dbt.include.global_project import PROJECT_NAME as GLOBAL_PROJECT_NAME from dbt.include.global_project import PROJECT_NAME as GLOBAL_PROJECT_NAME
from dbt.clients.jinja import MacroGenerator from dbt.clients.jinja import MacroGenerator
MacroNamespace = Dict[str, Macro] MacroNamespace = Dict[str, ParsedMacro]
# This class builds the MacroResolver by adding macros # This class builds the MacroResolver by adding macros
@@ -21,7 +21,7 @@ MacroNamespace = Dict[str, Macro]
class MacroResolver: class MacroResolver:
def __init__( def __init__(
self, self,
macros: MutableMapping[str, Macro], macros: MutableMapping[str, ParsedMacro],
root_project_name: str, root_project_name: str,
internal_package_names, internal_package_names,
) -> None: ) -> None:
@@ -77,7 +77,7 @@ class MacroResolver:
def _add_macro_to( def _add_macro_to(
self, self,
package_namespaces: Dict[str, MacroNamespace], package_namespaces: Dict[str, MacroNamespace],
macro: Macro, macro: ParsedMacro,
): ):
if macro.package_name in package_namespaces: if macro.package_name in package_namespaces:
namespace = package_namespaces[macro.package_name] namespace = package_namespaces[macro.package_name]
@@ -86,10 +86,10 @@ class MacroResolver:
package_namespaces[macro.package_name] = namespace package_namespaces[macro.package_name] = namespace
if macro.name in namespace: if macro.name in namespace:
raise DuplicateMacroNameError(macro, macro, macro.package_name) raise_duplicate_macro_name(macro, macro, macro.package_name)
package_namespaces[macro.package_name][macro.name] = macro package_namespaces[macro.package_name][macro.name] = macro
def add_macro(self, macro: Macro): def add_macro(self, macro: ParsedMacro):
macro_name: str = macro.name macro_name: str = macro.name
# internal macros (from plugins) will be processed separately from # internal macros (from plugins) will be processed separately from
@@ -109,15 +109,9 @@ class MacroResolver:
def get_macro(self, local_package, macro_name): def get_macro(self, local_package, macro_name):
local_package_macros = {} local_package_macros = {}
# If the macro is explicitly prefixed with an internal namespace
# (e.g. 'dbt.some_macro'), look there first
if local_package in self.internal_package_names:
local_package_macros = self.internal_packages[local_package]
# If the macro is explicitly prefixed with a different package name
# (e.g. 'dbt_utils.some_macro'), look there first
if local_package not in self.internal_package_names and local_package in self.packages: if local_package not in self.internal_package_names and local_package in self.packages:
local_package_macros = self.packages[local_package] local_package_macros = self.packages[local_package]
# First: search the specified package for this macro # First: search the local packages for this macro
if macro_name in local_package_macros: if macro_name in local_package_macros:
return local_package_macros[macro_name] return local_package_macros[macro_name]
# Now look up in the standard search order # Now look up in the standard search order
@@ -187,7 +181,7 @@ class TestMacroNamespace:
elif package_name in self.macro_resolver.packages: elif package_name in self.macro_resolver.packages:
macro = self.macro_resolver.packages[package_name].get(name) macro = self.macro_resolver.packages[package_name].get(name)
else: else:
raise PackageNotFoundForMacroError(package_name) raise_compiler_error(f"Could not find package '{package_name}'")
if not macro: if not macro:
return None return None
macro_func = MacroGenerator(macro, self.ctx, self.node, self.thread_ctx) macro_func = MacroGenerator(macro, self.ctx, self.node, self.thread_ctx)

View File

@@ -1,9 +1,9 @@
from typing import Any, Dict, Iterable, Union, Optional, List, Iterator, Mapping, Set from typing import Any, Dict, Iterable, Union, Optional, List, Iterator, Mapping, Set
from dbt.clients.jinja import MacroGenerator, MacroStack from dbt.clients.jinja import MacroGenerator, MacroStack
from dbt.contracts.graph.nodes import Macro from dbt.contracts.graph.parsed import ParsedMacro
from dbt.include.global_project import PROJECT_NAME as GLOBAL_PROJECT_NAME from dbt.include.global_project import PROJECT_NAME as GLOBAL_PROJECT_NAME
from dbt.exceptions import DuplicateMacroNameError, PackageNotFoundForMacroError from dbt.exceptions import raise_duplicate_macro_name, raise_compiler_error
FlatNamespace = Dict[str, MacroGenerator] FlatNamespace = Dict[str, MacroGenerator]
@@ -75,7 +75,7 @@ class MacroNamespace(Mapping):
elif package_name in self.packages: elif package_name in self.packages:
return self.packages[package_name].get(name) return self.packages[package_name].get(name)
else: else:
raise PackageNotFoundForMacroError(package_name) raise_compiler_error(f"Could not find package '{package_name}'")
# This class builds the MacroNamespace by adding macros to # This class builds the MacroNamespace by adding macros to
@@ -112,7 +112,7 @@ class MacroNamespaceBuilder:
def _add_macro_to( def _add_macro_to(
self, self,
hierarchy: Dict[str, FlatNamespace], hierarchy: Dict[str, FlatNamespace],
macro: Macro, macro: ParsedMacro,
macro_func: MacroGenerator, macro_func: MacroGenerator,
): ):
if macro.package_name in hierarchy: if macro.package_name in hierarchy:
@@ -122,10 +122,10 @@ class MacroNamespaceBuilder:
hierarchy[macro.package_name] = namespace hierarchy[macro.package_name] = namespace
if macro.name in namespace: if macro.name in namespace:
raise DuplicateMacroNameError(macro_func.macro, macro, macro.package_name) raise_duplicate_macro_name(macro_func.macro, macro, macro.package_name)
hierarchy[macro.package_name][macro.name] = macro_func hierarchy[macro.package_name][macro.name] = macro_func
def add_macro(self, macro: Macro, ctx: Dict[str, Any]): def add_macro(self, macro: ParsedMacro, ctx: Dict[str, Any]):
macro_name: str = macro.name macro_name: str = macro.name
# MacroGenerator is in clients/jinja.py # MacroGenerator is in clients/jinja.py
@@ -147,11 +147,13 @@ class MacroNamespaceBuilder:
elif macro.package_name == self.root_package: elif macro.package_name == self.root_package:
self.globals[macro_name] = macro_func self.globals[macro_name] = macro_func
def add_macros(self, macros: Iterable[Macro], ctx: Dict[str, Any]): def add_macros(self, macros: Iterable[ParsedMacro], ctx: Dict[str, Any]):
for macro in macros: for macro in macros:
self.add_macro(macro, ctx) self.add_macro(macro, ctx)
def build_namespace(self, macros: Iterable[Macro], ctx: Dict[str, Any]) -> MacroNamespace: def build_namespace(
self, macros: Iterable[ParsedMacro], ctx: Dict[str, Any]
) -> MacroNamespace:
self.add_macros(macros, ctx) self.add_macros(macros, ctx)
# Iterate in reverse-order and overwrite: the packages that are first # Iterate in reverse-order and overwrite: the packages that are first

View File

@@ -4,7 +4,6 @@ from dbt.clients.jinja import MacroStack
from dbt.contracts.connection import AdapterRequiredConfig from dbt.contracts.connection import AdapterRequiredConfig
from dbt.contracts.graph.manifest import Manifest from dbt.contracts.graph.manifest import Manifest
from dbt.context.macro_resolver import TestMacroNamespace from dbt.context.macro_resolver import TestMacroNamespace
from .base import contextproperty
from .configured import ConfiguredContext from .configured import ConfiguredContext
@@ -67,10 +66,6 @@ class ManifestContext(ConfiguredContext):
dct.update(self.namespace) dct.update(self.namespace)
return dct return dct
@contextproperty
def context_macro_stack(self):
return self.macro_stack
class QueryHeaderContext(ManifestContext): class QueryHeaderContext(ManifestContext):
def __init__(self, config: AdapterRequiredConfig, manifest: Manifest) -> None: def __init__(self, config: AdapterRequiredConfig, manifest: Manifest) -> None:

View File

@@ -19,56 +19,51 @@ from dbt.adapters.factory import get_adapter, get_adapter_package_names, get_ada
from dbt.clients import agate_helper from dbt.clients import agate_helper
from dbt.clients.jinja import get_rendered, MacroGenerator, MacroStack from dbt.clients.jinja import get_rendered, MacroGenerator, MacroStack
from dbt.config import RuntimeConfig, Project from dbt.config import RuntimeConfig, Project
from dbt.constants import SECRET_ENV_PREFIX, DEFAULT_ENV_PLACEHOLDER from .base import contextmember, contextproperty, Var
from dbt.context.base import contextmember, contextproperty, Var from .configured import FQNLookup
from dbt.context.configured import FQNLookup from .context_config import ContextConfig
from dbt.context.context_config import ContextConfig from dbt.logger import SECRET_ENV_PREFIX
from dbt.context.exceptions_jinja import wrapped_exports
from dbt.context.macro_resolver import MacroResolver, TestMacroNamespace from dbt.context.macro_resolver import MacroResolver, TestMacroNamespace
from dbt.context.macros import MacroNamespaceBuilder, MacroNamespace from .macros import MacroNamespaceBuilder, MacroNamespace
from dbt.context.manifest import ManifestContext from .manifest import ManifestContext
from dbt.contracts.connection import AdapterResponse from dbt.contracts.connection import AdapterResponse
from dbt.contracts.graph.manifest import Manifest, Disabled from dbt.contracts.graph.manifest import Manifest, Disabled
from dbt.contracts.graph.nodes import ( from dbt.contracts.graph.compiled import (
Macro, CompiledResource,
Exposure, CompiledSeedNode,
Metric,
Entity,
SeedNode,
SourceDefinition,
Resource,
ManifestNode, ManifestNode,
) )
from dbt.contracts.graph.parsed import (
ParsedMacro,
ParsedExposure,
ParsedMetric,
ParsedSeedNode,
ParsedSourceDefinition,
)
from dbt.contracts.graph.metrics import MetricReference, ResolvedMetricReference from dbt.contracts.graph.metrics import MetricReference, ResolvedMetricReference
from dbt.events.functions import get_metadata_vars
from dbt.exceptions import ( from dbt.exceptions import (
CompilationError, CompilationException,
ConflictingConfigKeysError, ParsingException,
SecretEnvVarLocationError, InternalException,
EnvVarMissingError, ValidationException,
DbtInternalError, RuntimeException,
InlineModelConfigError, macro_invalid_dispatch_arg,
NumberSourceArgsError, missing_config,
PersistDocsValueTypeError, raise_compiler_error,
LoadAgateTableNotSeedError, ref_invalid_args,
LoadAgateTableValueError, metric_invalid_args,
MacroDispatchArgError, ref_target_not_found,
MacrosSourcesUnWriteableError, metric_target_not_found,
MetricArgsError, ref_bad_context,
MissingConfigError, source_target_not_found,
OperationsCannotRefEphemeralNodesError, wrapped_exports,
PackageNotInDepsError, raise_parsing_error,
ParsingError, disallow_secret_env_var,
RefBadContextError,
RefArgsError,
DbtRuntimeError,
TargetNotFoundError,
DbtValidationError,
) )
from dbt.config import IsFQNResource from dbt.config import IsFQNResource
from dbt.node_types import NodeType, ModelLanguage from dbt.node_types import NodeType
from dbt.utils import merge, AttrDict, MultiDict, args_to_dict from dbt.utils import merge, AttrDict, MultiDict
from dbt import selected_resources from dbt import selected_resources
@@ -145,10 +140,10 @@ class BaseDatabaseWrapper:
f'`adapter.dispatch("{suggest_macro_name}", ' f'`adapter.dispatch("{suggest_macro_name}", '
f'macro_namespace="{suggest_macro_namespace}")`?' f'macro_namespace="{suggest_macro_namespace}")`?'
) )
raise CompilationError(msg) raise CompilationException(msg)
if packages is not None: if packages is not None:
raise MacroDispatchArgError(macro_name) raise macro_invalid_dispatch_arg(macro_name)
namespace = macro_namespace namespace = macro_namespace
@@ -160,7 +155,7 @@ class BaseDatabaseWrapper:
search_packages = [self.config.project_name, namespace] search_packages = [self.config.project_name, namespace]
else: else:
# Not a string and not None so must be a list # Not a string and not None so must be a list
raise CompilationError( raise CompilationException(
f"In adapter.dispatch, got a list macro_namespace argument " f"In adapter.dispatch, got a list macro_namespace argument "
f'("{macro_namespace}"), but macro_namespace should be None or a string.' f'("{macro_namespace}"), but macro_namespace should be None or a string.'
) )
@@ -173,8 +168,8 @@ class BaseDatabaseWrapper:
try: try:
# this uses the namespace from the context # this uses the namespace from the context
macro = self._namespace.get_from_package(package_name, search_name) macro = self._namespace.get_from_package(package_name, search_name)
except CompilationError: except CompilationException:
# Only raise CompilationError if macro is not found in # Only raise CompilationException if macro is not found in
# any package # any package
macro = None macro = None
@@ -187,8 +182,8 @@ class BaseDatabaseWrapper:
return macro return macro
searched = ", ".join(repr(a) for a in attempts) searched = ", ".join(repr(a) for a in attempts)
msg = f"In dispatch: No macro named '{macro_name}' found\n Searched for: {searched}" msg = f"In dispatch: No macro named '{macro_name}' found\n" f" Searched for: {searched}"
raise CompilationError(msg) raise CompilationException(msg)
class BaseResolver(metaclass=abc.ABCMeta): class BaseResolver(metaclass=abc.ABCMeta):
@@ -224,13 +219,13 @@ class BaseRefResolver(BaseResolver):
def validate_args(self, name: str, package: Optional[str]): def validate_args(self, name: str, package: Optional[str]):
if not isinstance(name, str): if not isinstance(name, str):
raise CompilationError( raise CompilationException(
f"The name argument to ref() must be a string, got {type(name)}" f"The name argument to ref() must be a string, got " f"{type(name)}"
) )
if package is not None and not isinstance(package, str): if package is not None and not isinstance(package, str):
raise CompilationError( raise CompilationException(
f"The package argument to ref() must be a string or None, got {type(package)}" f"The package argument to ref() must be a string or None, got " f"{type(package)}"
) )
def __call__(self, *args: str) -> RelationProxy: def __call__(self, *args: str) -> RelationProxy:
@@ -242,7 +237,7 @@ class BaseRefResolver(BaseResolver):
elif len(args) == 2: elif len(args) == 2:
package, name = args package, name = args
else: else:
raise RefArgsError(node=self.model, args=args) ref_invalid_args(self.model, args)
self.validate_args(name, package) self.validate_args(name, package)
return self.resolve(name, package) return self.resolve(name, package)
@@ -254,19 +249,21 @@ class BaseSourceResolver(BaseResolver):
def validate_args(self, source_name: str, table_name: str): def validate_args(self, source_name: str, table_name: str):
if not isinstance(source_name, str): if not isinstance(source_name, str):
raise CompilationError( raise CompilationException(
f"The source name (first) argument to source() must be a " f"The source name (first) argument to source() must be a "
f"string, got {type(source_name)}" f"string, got {type(source_name)}"
) )
if not isinstance(table_name, str): if not isinstance(table_name, str):
raise CompilationError( raise CompilationException(
f"The table name (second) argument to source() must be a " f"The table name (second) argument to source() must be a "
f"string, got {type(table_name)}" f"string, got {type(table_name)}"
) )
def __call__(self, *args: str) -> RelationProxy: def __call__(self, *args: str) -> RelationProxy:
if len(args) != 2: if len(args) != 2:
raise NumberSourceArgsError(args, node=self.model) raise_compiler_error(
f"source() takes exactly two arguments ({len(args)} given)", self.model
)
self.validate_args(args[0], args[1]) self.validate_args(args[0], args[1])
return self.resolve(args[0], args[1]) return self.resolve(args[0], args[1])
@@ -283,12 +280,12 @@ class BaseMetricResolver(BaseResolver):
def validate_args(self, name: str, package: Optional[str]): def validate_args(self, name: str, package: Optional[str]):
if not isinstance(name, str): if not isinstance(name, str):
raise CompilationError( raise CompilationException(
f"The name argument to metric() must be a string, got {type(name)}" f"The name argument to metric() must be a string, got {type(name)}"
) )
if package is not None and not isinstance(package, str): if package is not None and not isinstance(package, str):
raise CompilationError( raise CompilationException(
f"The package argument to metric() must be a string or None, got {type(package)}" f"The package argument to metric() must be a string or None, got {type(package)}"
) )
@@ -301,7 +298,7 @@ class BaseMetricResolver(BaseResolver):
elif len(args) == 2: elif len(args) == 2:
package, name = args package, name = args
else: else:
raise MetricArgsError(node=self.model, args=args) metric_invalid_args(self.model, args)
self.validate_args(name, package) self.validate_args(name, package)
return self.resolve(name, package) return self.resolve(name, package)
@@ -322,7 +319,12 @@ class ParseConfigObject(Config):
if oldkey in config: if oldkey in config:
newkey = oldkey.replace("_", "-") newkey = oldkey.replace("_", "-")
if newkey in config: if newkey in config:
raise ConflictingConfigKeysError(oldkey, newkey, node=self.model) raise_compiler_error(
'Invalid config, has conflicting keys "{}" and "{}"'.format(
oldkey, newkey
),
self.model,
)
config[newkey] = config.pop(oldkey) config[newkey] = config.pop(oldkey)
return config return config
@@ -332,14 +334,14 @@ class ParseConfigObject(Config):
elif len(args) == 0 and len(kwargs) > 0: elif len(args) == 0 and len(kwargs) > 0:
opts = kwargs opts = kwargs
else: else:
raise InlineModelConfigError(node=self.model) raise_compiler_error("Invalid inline model config", self.model)
opts = self._transform_config(opts) opts = self._transform_config(opts)
# it's ok to have a parse context with no context config, but you must # it's ok to have a parse context with no context config, but you must
# not call it! # not call it!
if self.context_config is None: if self.context_config is None:
raise DbtRuntimeError("At parse time, did not receive a context config") raise RuntimeException("At parse time, did not receive a context config")
self.context_config.add_config_call(opts) self.context_config.add_config_call(opts)
return "" return ""
@@ -380,7 +382,7 @@ class RuntimeConfigObject(Config):
else: else:
result = self.model.config.get(name, default) result = self.model.config.get(name, default)
if result is _MISSING: if result is _MISSING:
raise MissingConfigError(unique_id=self.model.unique_id, name=name) missing_config(self.model, name)
return result return result
def require(self, name, validator=None): def require(self, name, validator=None):
@@ -402,14 +404,20 @@ class RuntimeConfigObject(Config):
def persist_relation_docs(self) -> bool: def persist_relation_docs(self) -> bool:
persist_docs = self.get("persist_docs", default={}) persist_docs = self.get("persist_docs", default={})
if not isinstance(persist_docs, dict): if not isinstance(persist_docs, dict):
raise PersistDocsValueTypeError(persist_docs) raise_compiler_error(
f"Invalid value provided for 'persist_docs'. Expected dict "
f"but received {type(persist_docs)}"
)
return persist_docs.get("relation", False) return persist_docs.get("relation", False)
def persist_column_docs(self) -> bool: def persist_column_docs(self) -> bool:
persist_docs = self.get("persist_docs", default={}) persist_docs = self.get("persist_docs", default={})
if not isinstance(persist_docs, dict): if not isinstance(persist_docs, dict):
raise PersistDocsValueTypeError(persist_docs) raise_compiler_error(
f"Invalid value provided for 'persist_docs'. Expected dict "
f"but received {type(persist_docs)}"
)
return persist_docs.get("columns", False) return persist_docs.get("columns", False)
@@ -468,11 +476,10 @@ class RuntimeRefResolver(BaseRefResolver):
) )
if target_model is None or isinstance(target_model, Disabled): if target_model is None or isinstance(target_model, Disabled):
raise TargetNotFoundError( ref_target_not_found(
node=self.model, self.model,
target_name=target_name, target_name,
target_kind="node", target_package,
target_package=target_package,
disabled=isinstance(target_model, Disabled), disabled=isinstance(target_model, Disabled),
) )
self.validate(target_model, target_name, target_package) self.validate(target_model, target_name, target_package)
@@ -490,7 +497,7 @@ class RuntimeRefResolver(BaseRefResolver):
) -> None: ) -> None:
if resolved.unique_id not in self.model.depends_on.nodes: if resolved.unique_id not in self.model.depends_on.nodes:
args = self._repack_args(target_name, target_package) args = self._repack_args(target_name, target_package)
raise RefBadContextError(node=self.model, args=args) ref_bad_context(self.model, args)
class OperationRefResolver(RuntimeRefResolver): class OperationRefResolver(RuntimeRefResolver):
@@ -505,8 +512,13 @@ class OperationRefResolver(RuntimeRefResolver):
def create_relation(self, target_model: ManifestNode, name: str) -> RelationProxy: def create_relation(self, target_model: ManifestNode, name: str) -> RelationProxy:
if target_model.is_ephemeral_model: if target_model.is_ephemeral_model:
# In operations, we can't ref() ephemeral nodes, because # In operations, we can't ref() ephemeral nodes, because
# Macros do not support set_cte # ParsedMacros do not support set_cte
raise OperationsCannotRefEphemeralNodesError(target_model.name, node=self.model) raise_compiler_error(
"Operations can not ref() ephemeral nodes, but {} is ephemeral".format(
target_model.name
),
self.model,
)
else: else:
return super().create_relation(target_model, name) return super().create_relation(target_model, name)
@@ -529,11 +541,10 @@ class RuntimeSourceResolver(BaseSourceResolver):
) )
if target_source is None or isinstance(target_source, Disabled): if target_source is None or isinstance(target_source, Disabled):
raise TargetNotFoundError( source_target_not_found(
node=self.model, self.model,
target_name=f"{source_name}.{table_name}", source_name,
target_kind="source", table_name,
disabled=(isinstance(target_source, Disabled)),
) )
return self.Relation.create_from_source(target_source) return self.Relation.create_from_source(target_source)
@@ -556,11 +567,11 @@ class RuntimeMetricResolver(BaseMetricResolver):
) )
if target_metric is None or isinstance(target_metric, Disabled): if target_metric is None or isinstance(target_metric, Disabled):
raise TargetNotFoundError( # TODO : Use a different exception!!
node=self.model, metric_target_not_found(
target_name=target_name, self.model,
target_kind="metric", target_name,
target_package=target_package, target_package,
) )
return ResolvedMetricReference(target_metric, self.manifest, self.Relation) return ResolvedMetricReference(target_metric, self.manifest, self.Relation)
@@ -572,9 +583,9 @@ class ModelConfiguredVar(Var):
self, self,
context: Dict[str, Any], context: Dict[str, Any],
config: RuntimeConfig, config: RuntimeConfig,
node: Resource, node: CompiledResource,
) -> None: ) -> None:
self._node: Resource self._node: CompiledResource
self._config: RuntimeConfig = config self._config: RuntimeConfig = config
super().__init__(context, config.cli_vars, node=node) super().__init__(context, config.cli_vars, node=node)
@@ -585,7 +596,7 @@ class ModelConfiguredVar(Var):
if package_name != self._config.project_name: if package_name != self._config.project_name:
if package_name not in dependencies: if package_name not in dependencies:
# I don't think this is actually reachable # I don't think this is actually reachable
raise PackageNotInDepsError(package_name, node=self._node) raise_compiler_error(f"Node package named {package_name} not found!", self._node)
yield dependencies[package_name] yield dependencies[package_name]
yield self._config yield self._config
@@ -675,10 +686,10 @@ class ProviderContext(ManifestContext):
context_config: Optional[ContextConfig], context_config: Optional[ContextConfig],
) -> None: ) -> None:
if provider is None: if provider is None:
raise DbtInternalError(f"Invalid provider given to context: {provider}") raise InternalException(f"Invalid provider given to context: {provider}")
# mypy appeasement - we know it'll be a RuntimeConfig # mypy appeasement - we know it'll be a RuntimeConfig
self.config: RuntimeConfig self.config: RuntimeConfig
self.model: Union[Macro, ManifestNode] = model self.model: Union[ParsedMacro, ManifestNode] = model
super().__init__(config, manifest, model.package_name) super().__init__(config, manifest, model.package_name)
self.sql_results: Dict[str, AttrDict] = {} self.sql_results: Dict[str, AttrDict] = {}
self.context_config: Optional[ContextConfig] = context_config self.context_config: Optional[ContextConfig] = context_config
@@ -699,14 +710,6 @@ class ProviderContext(ManifestContext):
self.model, self.model,
) )
@contextproperty
def dbt_metadata_envs(self) -> Dict[str, str]:
return get_metadata_vars()
@contextproperty
def invocation_args_dict(self):
return args_to_dict(self.config.args)
@contextproperty @contextproperty
def _sql_results(self) -> Dict[str, AttrDict]: def _sql_results(self) -> Dict[str, AttrDict]:
return self.sql_results return self.sql_results
@@ -752,7 +755,7 @@ class ProviderContext(ManifestContext):
return return
elif value == arg: elif value == arg:
return return
raise DbtValidationError( raise ValidationException(
'Expected value "{}" to be one of {}'.format(value, ",".join(map(str, args))) 'Expected value "{}" to be one of {}'.format(value, ",".join(map(str, args)))
) )
@@ -767,8 +770,8 @@ class ProviderContext(ManifestContext):
@contextmember @contextmember
def write(self, payload: str) -> str: def write(self, payload: str) -> str:
# macros/source defs aren't 'writeable'. # macros/source defs aren't 'writeable'.
if isinstance(self.model, (Macro, SourceDefinition)): if isinstance(self.model, (ParsedMacro, ParsedSourceDefinition)):
raise MacrosSourcesUnWriteableError(node=self.model) raise_compiler_error('cannot "write" macros or sources')
self.model.build_path = self.model.write_node(self.config.target_path, "run", payload) self.model.build_path = self.model.write_node(self.config.target_path, "run", payload)
return "" return ""
@@ -783,19 +786,20 @@ class ProviderContext(ManifestContext):
try: try:
return func(*args, **kwargs) return func(*args, **kwargs)
except Exception: except Exception:
raise CompilationError(message_if_exception, self.model) raise_compiler_error(message_if_exception, self.model)
@contextmember @contextmember
def load_agate_table(self) -> agate.Table: def load_agate_table(self) -> agate.Table:
if not isinstance(self.model, SeedNode): if not isinstance(self.model, (ParsedSeedNode, CompiledSeedNode)):
raise LoadAgateTableNotSeedError(self.model.resource_type, node=self.model) raise_compiler_error(
assert self.model.root_path "can only load_agate_table for seeds (got a {})".format(self.model.resource_type)
)
path = os.path.join(self.model.root_path, self.model.original_file_path) path = os.path.join(self.model.root_path, self.model.original_file_path)
column_types = self.model.config.column_types column_types = self.model.config.column_types
try: try:
table = agate_helper.from_csv(path, text_columns=column_types) table = agate_helper.from_csv(path, text_columns=column_types)
except ValueError as e: except ValueError as e:
raise LoadAgateTableValueError(e, node=self.model) raise_compiler_error(str(e))
table.original_abspath = os.path.abspath(path) table.original_abspath = os.path.abspath(path)
return table return table
@@ -1153,12 +1157,7 @@ class ProviderContext(ManifestContext):
@contextproperty("model") @contextproperty("model")
def ctx_model(self) -> Dict[str, Any]: def ctx_model(self) -> Dict[str, Any]:
ret = self.model.to_dict(omit_none=True) return self.model.to_dict(omit_none=True)
# Maintain direct use of compiled_sql
# TODO add depreciation logic[CT-934]
if "compiled_code" in ret:
ret["compiled_sql"] = ret["compiled_code"]
return ret
@contextproperty @contextproperty
def pre_hooks(self) -> Optional[List[Dict[str, Any]]]: def pre_hooks(self) -> Optional[List[Dict[str, Any]]]:
@@ -1186,7 +1185,7 @@ class ProviderContext(ManifestContext):
"https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch)" "https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch)"
" adapter_macro was called for: {macro_name}".format(macro_name=name) " adapter_macro was called for: {macro_name}".format(macro_name=name)
) )
raise CompilationError(msg) raise CompilationException(msg)
@contextmember @contextmember
def env_var(self, var: str, default: Optional[str] = None) -> str: def env_var(self, var: str, default: Optional[str] = None) -> str:
@@ -1197,7 +1196,7 @@ class ProviderContext(ManifestContext):
""" """
return_value = None return_value = None
if var.startswith(SECRET_ENV_PREFIX): if var.startswith(SECRET_ENV_PREFIX):
raise SecretEnvVarLocationError(var) disallow_secret_env_var(var)
if var in os.environ: if var in os.environ:
return_value = os.environ[var] return_value = os.environ[var]
elif default is not None: elif default is not None:
@@ -1206,21 +1205,8 @@ class ProviderContext(ManifestContext):
if return_value is not None: if return_value is not None:
# Save the env_var value in the manifest and the var name in the source_file. # Save the env_var value in the manifest and the var name in the source_file.
# If this is compiling, do not save because it's irrelevant to parsing. # If this is compiling, do not save because it's irrelevant to parsing.
compiling = ( if self.model and not hasattr(self.model, "compiled"):
True self.manifest.env_vars[var] = return_value
if hasattr(self.model, "compiled")
and getattr(self.model, "compiled", False) is True
else False
)
if self.model and not compiling:
# If the environment variable is set from a default, store a string indicating
# that so we can skip partial parsing. Otherwise the file will be scheduled for
# reparsing. If the default changes, the file will have been updated and therefore
# will be scheduled for reparsing anyways.
self.manifest.env_vars[var] = (
return_value if var in os.environ else DEFAULT_ENV_PLACEHOLDER
)
# hooks come from dbt_project.yml which doesn't have a real file_id # hooks come from dbt_project.yml which doesn't have a real file_id
if self.model.file_id in self.manifest.files: if self.model.file_id in self.manifest.files:
source_file = self.manifest.files[self.model.file_id] source_file = self.manifest.files[self.model.file_id]
@@ -1230,7 +1216,8 @@ class ProviderContext(ManifestContext):
source_file.env_vars.append(var) # type: ignore[union-attr] source_file.env_vars.append(var) # type: ignore[union-attr]
return return_value return return_value
else: else:
raise EnvVarMissingError(var) msg = f"Env var required but not provided: '{var}'"
raise_parsing_error(msg)
@contextproperty @contextproperty
def selected_resources(self) -> List[str]: def selected_resources(self) -> List[str]:
@@ -1241,19 +1228,6 @@ class ProviderContext(ManifestContext):
""" """
return selected_resources.SELECTED_RESOURCES return selected_resources.SELECTED_RESOURCES
@contextmember
def submit_python_job(self, parsed_model: Dict, compiled_code: str) -> AdapterResponse:
# Check macro_stack and that the unique id is for a materialization macro
if not (
self.context_macro_stack.depth == 2
and self.context_macro_stack.call_stack[1] == "macro.dbt.statement"
and "materialization" in self.context_macro_stack.call_stack[0]
):
raise DbtRuntimeError(
f"submit_python_job is not intended to be called here, at model {parsed_model['alias']}, with macro call_stack {self.context_macro_stack.call_stack}."
)
return self.adapter.submit_python_job(parsed_model, compiled_code)
class MacroContext(ProviderContext): class MacroContext(ProviderContext):
"""Internally, macros can be executed like nodes, with some restrictions: """Internally, macros can be executed like nodes, with some restrictions:
@@ -1266,7 +1240,7 @@ class MacroContext(ProviderContext):
def __init__( def __init__(
self, self,
model: Macro, model: ParsedMacro,
config: RuntimeConfig, config: RuntimeConfig,
manifest: Manifest, manifest: Manifest,
provider: Provider, provider: Provider,
@@ -1304,20 +1278,9 @@ class ModelContext(ProviderContext):
@contextproperty @contextproperty
def sql(self) -> Optional[str]: def sql(self) -> Optional[str]:
# only doing this in sql model for backward compatible
if (
getattr(self.model, "extra_ctes_injected", None)
and self.model.language == ModelLanguage.sql # type: ignore[union-attr]
):
# TODO CT-211
return self.model.compiled_code # type: ignore[union-attr]
return None
@contextproperty
def compiled_code(self) -> Optional[str]:
if getattr(self.model, "extra_ctes_injected", None): if getattr(self.model, "extra_ctes_injected", None):
# TODO CT-211 # TODO CT-211
return self.model.compiled_code # type: ignore[union-attr] return self.model.compiled_sql # type: ignore[union-attr]
return None return None
@contextproperty @contextproperty
@@ -1381,7 +1344,7 @@ def generate_parser_model_context(
def generate_generate_name_macro_context( def generate_generate_name_macro_context(
macro: Macro, macro: ParsedMacro,
config: RuntimeConfig, config: RuntimeConfig,
manifest: Manifest, manifest: Manifest,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
@@ -1399,7 +1362,7 @@ def generate_runtime_model_context(
def generate_runtime_macro_context( def generate_runtime_macro_context(
macro: Macro, macro: ParsedMacro,
config: RuntimeConfig, config: RuntimeConfig,
manifest: Manifest, manifest: Manifest,
package_name: Optional[str], package_name: Optional[str],
@@ -1411,7 +1374,7 @@ def generate_runtime_macro_context(
class ExposureRefResolver(BaseResolver): class ExposureRefResolver(BaseResolver):
def __call__(self, *args) -> str: def __call__(self, *args) -> str:
if len(args) not in (1, 2): if len(args) not in (1, 2):
raise RefArgsError(node=self.model, args=args) ref_invalid_args(self.model, args)
self.model.refs.append(list(args)) self.model.refs.append(list(args))
return "" return ""
@@ -1419,21 +1382,15 @@ class ExposureRefResolver(BaseResolver):
class ExposureSourceResolver(BaseResolver): class ExposureSourceResolver(BaseResolver):
def __call__(self, *args) -> str: def __call__(self, *args) -> str:
if len(args) != 2: if len(args) != 2:
raise NumberSourceArgsError(args, node=self.model) raise_compiler_error(
f"source() takes exactly two arguments ({len(args)} given)", self.model
)
self.model.sources.append(list(args)) self.model.sources.append(list(args))
return "" return ""
class ExposureMetricResolver(BaseResolver):
def __call__(self, *args) -> str:
if len(args) not in (1, 2):
raise MetricArgsError(node=self.model, args=args)
self.model.metrics.append(list(args))
return ""
def generate_parse_exposure( def generate_parse_exposure(
exposure: Exposure, exposure: ParsedExposure,
config: RuntimeConfig, config: RuntimeConfig,
manifest: Manifest, manifest: Manifest,
package_name: str, package_name: str,
@@ -1452,12 +1409,6 @@ def generate_parse_exposure(
project, project,
manifest, manifest,
), ),
"metric": ExposureMetricResolver(
None,
exposure,
project,
manifest,
),
} }
@@ -1469,21 +1420,21 @@ class MetricRefResolver(BaseResolver):
elif len(args) == 2: elif len(args) == 2:
package, name = args package, name = args
else: else:
raise RefArgsError(node=self.model, args=args) ref_invalid_args(self.model, args)
self.validate_args(name, package) self.validate_args(name, package)
self.model.refs.append(list(args)) self.model.refs.append(list(args))
return "" return ""
def validate_args(self, name, package): def validate_args(self, name, package):
if not isinstance(name, str): if not isinstance(name, str):
raise ParsingError( raise ParsingException(
f"In a metrics section in {self.model.original_file_path} " f"In a metrics section in {self.model.original_file_path} "
"the name argument to ref() must be a string" "the name argument to ref() must be a string"
) )
def generate_parse_metrics( def generate_parse_metrics(
metric: Metric, metric: ParsedMetric,
config: RuntimeConfig, config: RuntimeConfig,
manifest: Manifest, manifest: Manifest,
package_name: str, package_name: str,
@@ -1505,44 +1456,6 @@ def generate_parse_metrics(
} }
class EntityRefResolver(BaseResolver):
def __call__(self, *args) -> str:
package = None
if len(args) == 1:
name = args[0]
elif len(args) == 2:
package, name = args
else:
raise RefArgsError(node=self.model, args=args)
self.validate_args(name, package)
self.model.refs.append(list(args))
return ""
def validate_args(self, name, package):
if not isinstance(name, str):
raise ParsingError(
f"In the entity associated with {self.model.original_file_path} "
"the name argument to ref() must be a string"
)
def generate_parse_entities(
entity: Entity,
config: RuntimeConfig,
manifest: Manifest,
package_name: str,
) -> Dict[str, Any]:
project = config.load_dependencies()[package_name]
return {
"ref": EntityRefResolver(
None,
entity,
project,
manifest,
),
}
# This class is currently used by the schema parser in order # This class is currently used by the schema parser in order
# to limit the number of macros in the context by using # to limit the number of macros in the context by using
# the TestMacroNamespace # the TestMacroNamespace
@@ -1597,7 +1510,7 @@ class TestContext(ProviderContext):
def env_var(self, var: str, default: Optional[str] = None) -> str: def env_var(self, var: str, default: Optional[str] = None) -> str:
return_value = None return_value = None
if var.startswith(SECRET_ENV_PREFIX): if var.startswith(SECRET_ENV_PREFIX):
raise SecretEnvVarLocationError(var) disallow_secret_env_var(var)
if var in os.environ: if var in os.environ:
return_value = os.environ[var] return_value = os.environ[var]
elif default is not None: elif default is not None:
@@ -1606,13 +1519,7 @@ class TestContext(ProviderContext):
if return_value is not None: if return_value is not None:
# Save the env_var value in the manifest and the var name in the source_file # Save the env_var value in the manifest and the var name in the source_file
if self.model: if self.model:
# If the environment variable is set from a default, store a string indicating self.manifest.env_vars[var] = return_value
# that so we can skip partial parsing. Otherwise the file will be scheduled for
# reparsing. If the default changes, the file will have been updated and therefore
# will be scheduled for reparsing anyways.
self.manifest.env_vars[var] = (
return_value if var in os.environ else DEFAULT_ENV_PLACEHOLDER
)
# the "model" should only be test nodes, but just in case, check # the "model" should only be test nodes, but just in case, check
# TODO CT-211 # TODO CT-211
if self.model.resource_type == NodeType.Test and self.model.file_key_name: # type: ignore[union-attr] # noqa if self.model.resource_type == NodeType.Test and self.model.file_key_name: # type: ignore[union-attr] # noqa
@@ -1623,7 +1530,8 @@ class TestContext(ProviderContext):
source_file.add_env_var(var, yaml_key, name) # type: ignore[union-attr] source_file.add_env_var(var, yaml_key, name) # type: ignore[union-attr]
return return_value return return_value
else: else:
raise EnvVarMissingError(var) msg = f"Env var required but not provided: '{var}'"
raise_parsing_error(msg)
def generate_test_context( def generate_test_context(

View File

@@ -3,8 +3,8 @@ from typing import Any, Dict, Optional
from .base import BaseContext, contextmember from .base import BaseContext, contextmember
from dbt.constants import SECRET_ENV_PREFIX, DEFAULT_ENV_PLACEHOLDER from dbt.exceptions import raise_parsing_error
from dbt.exceptions import EnvVarMissingError from dbt.logger import SECRET_ENV_PREFIX
SECRET_PLACEHOLDER = "$$$DBT_SECRET_START$$${}$$$DBT_SECRET_END$$$" SECRET_PLACEHOLDER = "$$$DBT_SECRET_START$$${}$$$DBT_SECRET_END$$$"
@@ -43,14 +43,11 @@ class SecretContext(BaseContext):
# if it's a 'secret' env var, we shouldn't even get here # if it's a 'secret' env var, we shouldn't even get here
# but just to be safe — don't save secrets # but just to be safe — don't save secrets
if not var.startswith(SECRET_ENV_PREFIX): if not var.startswith(SECRET_ENV_PREFIX):
# If the environment variable is set from a default, store a string indicating self.env_vars[var] = return_value
# that so we can skip partial parsing. Otherwise the file will be scheduled for
# reparsing. If the default changes, the file will have been updated and therefore
# will be scheduled for reparsing anyways.
self.env_vars[var] = return_value if var in os.environ else DEFAULT_ENV_PLACEHOLDER
return return_value return return_value
else: else:
raise EnvVarMissingError(var) msg = f"Env var required but not provided: '{var}'"
raise_parsing_error(msg)
def generate_secret_context(cli_vars: Dict[str, Any]) -> Dict[str, Any]: def generate_secret_context(cli_vars: Dict[str, Any]) -> Dict[str, Any]:

View File

@@ -12,11 +12,10 @@ from typing import (
List, List,
Callable, Callable,
) )
from dbt.exceptions import DbtInternalError from dbt.exceptions import InternalException
from dbt.utils import translate_aliases from dbt.utils import translate_aliases
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import NewConnectionOpening from dbt.events.types import NewConnectionOpening
from dbt.events.contextvars import get_node_info
from typing_extensions import Protocol from typing_extensions import Protocol
from dbt.dataclass_schema import ( from dbt.dataclass_schema import (
dbtClassMixin, dbtClassMixin,
@@ -94,8 +93,8 @@ class Connection(ExtensibleDbtClassMixin, Replaceable):
# this will actually change 'self._handle'. # this will actually change 'self._handle'.
self._handle.resolve(self) self._handle.resolve(self)
except RecursionError as exc: except RecursionError as exc:
raise DbtInternalError( raise InternalException(
"A connection's open() method attempted to read the handle value" "A connection's open() method attempted to read the " "handle value"
) from exc ) from exc
return self._handle return self._handle
@@ -113,9 +112,7 @@ class LazyHandle:
self.opener = opener self.opener = opener
def resolve(self, connection: Connection) -> Connection: def resolve(self, connection: Connection) -> Connection:
fire_event( fire_event(NewConnectionOpening(connection_state=connection.state))
NewConnectionOpening(connection_state=connection.state, node_info=get_node_info())
)
return self.opener(connection) return self.opener(connection)

View File

@@ -1,16 +1,18 @@
import hashlib import hashlib
import os import os
from dataclasses import dataclass, field from dataclasses import dataclass, field
from mashumaro.types import SerializableType from mashumaro.types import SerializableType
from typing import List, Optional, Union, Dict, Any from typing import List, Optional, Union, Dict, Any
from dbt.constants import MAXIMUM_SEED_SIZE
from dbt.dataclass_schema import dbtClassMixin, StrEnum from dbt.dataclass_schema import dbtClassMixin, StrEnum
from .util import SourceKey from .util import SourceKey
MAXIMUM_SEED_SIZE = 1 * 1024 * 1024
MAXIMUM_SEED_SIZE_NAME = "1MB"
class ParseFileType(StrEnum): class ParseFileType(StrEnum):
Macro = "macro" Macro = "macro"
Model = "model" Model = "model"
@@ -112,34 +114,25 @@ class FileHash(dbtClassMixin):
@dataclass @dataclass
class RemoteFile(dbtClassMixin): class RemoteFile(dbtClassMixin):
def __init__(self, language) -> None:
if language == "sql":
self.path_end = ".sql"
elif language == "python":
self.path_end = ".py"
else:
raise RuntimeError(f"Invalid language for remote File {language}")
self.path = f"from remote system{self.path_end}"
@property @property
def searched_path(self) -> str: def searched_path(self) -> str:
return self.path return "from remote system"
@property @property
def relative_path(self) -> str: def relative_path(self) -> str:
return self.path return "from remote system"
@property @property
def absolute_path(self) -> str: def absolute_path(self) -> str:
return self.path return "from remote system"
@property @property
def original_file_path(self): def original_file_path(self):
return self.path return "from remote system"
@property @property
def modification_time(self): def modification_time(self):
return self.path return "from remote system"
@dataclass @dataclass
@@ -209,9 +202,9 @@ class SourceFile(BaseSourceFile):
# TODO: do this a different way. This remote file kludge isn't going # TODO: do this a different way. This remote file kludge isn't going
# to work long term # to work long term
@classmethod @classmethod
def remote(cls, contents: str, project_name: str, language: str) -> "SourceFile": def remote(cls, contents: str, project_name: str) -> "SourceFile":
self = cls( self = cls(
path=RemoteFile(language), path=RemoteFile(),
checksum=FileHash.from_contents(contents), checksum=FileHash.from_contents(contents),
project_name=project_name, project_name=project_name,
contents=contents, contents=contents,
@@ -227,7 +220,6 @@ class SchemaSourceFile(BaseSourceFile):
sources: List[str] = field(default_factory=list) sources: List[str] = field(default_factory=list)
exposures: List[str] = field(default_factory=list) exposures: List[str] = field(default_factory=list)
metrics: List[str] = field(default_factory=list) metrics: List[str] = field(default_factory=list)
entities: List[str] = field(default_factory=list)
# node patches contain models, seeds, snapshots, analyses # node patches contain models, seeds, snapshots, analyses
ndp: List[str] = field(default_factory=list) ndp: List[str] = field(default_factory=list)
# any macro patches in this file by macro unique_id. # any macro patches in this file by macro unique_id.
@@ -276,13 +268,11 @@ class SchemaSourceFile(BaseSourceFile):
self.tests[key][name] = [] self.tests[key][name] = []
self.tests[key][name].append(node_unique_id) self.tests[key][name].append(node_unique_id)
# this is only used in unit tests
def remove_tests(self, yaml_key, name): def remove_tests(self, yaml_key, name):
if yaml_key in self.tests: if yaml_key in self.tests:
if name in self.tests[yaml_key]: if name in self.tests[yaml_key]:
del self.tests[yaml_key][name] del self.tests[yaml_key][name]
# this is only used in tests (unit + functional)
def get_tests(self, yaml_key, name): def get_tests(self, yaml_key, name):
if yaml_key in self.tests: if yaml_key in self.tests:
if name in self.tests[yaml_key]: if name in self.tests[yaml_key]:

View File

@@ -0,0 +1,235 @@
from dbt.contracts.graph.parsed import (
HasTestMetadata,
ParsedNode,
ParsedAnalysisNode,
ParsedSingularTestNode,
ParsedHookNode,
ParsedModelNode,
ParsedExposure,
ParsedMetric,
ParsedResource,
ParsedRPCNode,
ParsedSqlNode,
ParsedGenericTestNode,
ParsedSeedNode,
ParsedSnapshotNode,
ParsedSourceDefinition,
SeedConfig,
TestConfig,
same_seeds,
)
from dbt.node_types import NodeType
from dbt.contracts.util import Replaceable
from dbt.dataclass_schema import dbtClassMixin
from dataclasses import dataclass, field
from typing import Optional, List, Union, Dict, Type
@dataclass
class InjectedCTE(dbtClassMixin, Replaceable):
id: str
sql: str
@dataclass
class CompiledNodeMixin(dbtClassMixin):
# this is a special mixin class to provide a required argument. If a node
# is missing a `compiled` flag entirely, it must not be a CompiledNode.
compiled: bool
@dataclass
class CompiledNode(ParsedNode, CompiledNodeMixin):
compiled_sql: Optional[str] = None
extra_ctes_injected: bool = False
extra_ctes: List[InjectedCTE] = field(default_factory=list)
relation_name: Optional[str] = None
_pre_injected_sql: Optional[str] = None
def set_cte(self, cte_id: str, sql: str):
"""This is the equivalent of what self.extra_ctes[cte_id] = sql would
do if extra_ctes were an OrderedDict
"""
for cte in self.extra_ctes:
if cte.id == cte_id:
cte.sql = sql
break
else:
self.extra_ctes.append(InjectedCTE(id=cte_id, sql=sql))
def __post_serialize__(self, dct):
dct = super().__post_serialize__(dct)
if "_pre_injected_sql" in dct:
del dct["_pre_injected_sql"]
return dct
@dataclass
class CompiledAnalysisNode(CompiledNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Analysis]})
@dataclass
class CompiledHookNode(CompiledNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Operation]})
index: Optional[int] = None
@dataclass
class CompiledModelNode(CompiledNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Model]})
# TODO: rm?
@dataclass
class CompiledRPCNode(CompiledNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.RPCCall]})
@dataclass
class CompiledSqlNode(CompiledNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.SqlOperation]})
@dataclass
class CompiledSeedNode(CompiledNode):
# keep this in sync with ParsedSeedNode!
resource_type: NodeType = field(metadata={"restrict": [NodeType.Seed]})
config: SeedConfig = field(default_factory=SeedConfig)
@property
def empty(self):
"""Seeds are never empty"""
return False
def same_body(self, other) -> bool:
return same_seeds(self, other)
@dataclass
class CompiledSnapshotNode(CompiledNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Snapshot]})
@dataclass
class CompiledSingularTestNode(CompiledNode):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Test]})
# Was not able to make mypy happy and keep the code working. We need to
# refactor the various configs.
config: TestConfig = field(default_factory=TestConfig) # type:ignore
@dataclass
class CompiledGenericTestNode(CompiledNode, HasTestMetadata):
# keep this in sync with ParsedGenericTestNode!
resource_type: NodeType = field(metadata={"restrict": [NodeType.Test]})
column_name: Optional[str] = None
file_key_name: Optional[str] = None
# Was not able to make mypy happy and keep the code working. We need to
# refactor the various configs.
config: TestConfig = field(default_factory=TestConfig) # type:ignore
def same_contents(self, other) -> bool:
if other is None:
return False
return self.same_config(other) and self.same_fqn(other) and True
CompiledTestNode = Union[CompiledSingularTestNode, CompiledGenericTestNode]
PARSED_TYPES: Dict[Type[CompiledNode], Type[ParsedResource]] = {
CompiledAnalysisNode: ParsedAnalysisNode,
CompiledModelNode: ParsedModelNode,
CompiledHookNode: ParsedHookNode,
CompiledRPCNode: ParsedRPCNode,
CompiledSqlNode: ParsedSqlNode,
CompiledSeedNode: ParsedSeedNode,
CompiledSnapshotNode: ParsedSnapshotNode,
CompiledSingularTestNode: ParsedSingularTestNode,
CompiledGenericTestNode: ParsedGenericTestNode,
}
COMPILED_TYPES: Dict[Type[ParsedResource], Type[CompiledNode]] = {
ParsedAnalysisNode: CompiledAnalysisNode,
ParsedModelNode: CompiledModelNode,
ParsedHookNode: CompiledHookNode,
ParsedRPCNode: CompiledRPCNode,
ParsedSqlNode: CompiledSqlNode,
ParsedSeedNode: CompiledSeedNode,
ParsedSnapshotNode: CompiledSnapshotNode,
ParsedSingularTestNode: CompiledSingularTestNode,
ParsedGenericTestNode: CompiledGenericTestNode,
}
# for some types, the compiled type is the parsed type, so make this easy
CompiledType = Union[Type[CompiledNode], Type[ParsedResource]]
CompiledResource = Union[ParsedResource, CompiledNode]
def compiled_type_for(parsed: ParsedNode) -> CompiledType:
if type(parsed) in COMPILED_TYPES:
return COMPILED_TYPES[type(parsed)]
else:
return type(parsed)
def parsed_instance_for(compiled: CompiledNode) -> ParsedResource:
cls = PARSED_TYPES.get(type(compiled))
if cls is None:
# how???
raise ValueError("invalid resource_type: {}".format(compiled.resource_type))
return cls.from_dict(compiled.to_dict(omit_none=True))
NonSourceCompiledNode = Union[
CompiledAnalysisNode,
CompiledSingularTestNode,
CompiledModelNode,
CompiledHookNode,
CompiledRPCNode,
CompiledSqlNode,
CompiledGenericTestNode,
CompiledSeedNode,
CompiledSnapshotNode,
]
NonSourceParsedNode = Union[
ParsedAnalysisNode,
ParsedSingularTestNode,
ParsedHookNode,
ParsedModelNode,
ParsedRPCNode,
ParsedSqlNode,
ParsedGenericTestNode,
ParsedSeedNode,
ParsedSnapshotNode,
]
# This is anything that can be in manifest.nodes.
ManifestNode = Union[
NonSourceCompiledNode,
NonSourceParsedNode,
]
# We allow either parsed or compiled nodes, or parsed sources, as some
# 'compile()' calls in the runner actually just return the original parsed
# node they were given.
CompileResultNode = Union[
ManifestNode,
ParsedSourceDefinition,
]
# anything that participates in the graph: sources, exposures, metrics,
# or manifest nodes
GraphMemberNode = Union[
CompileResultNode,
ParsedExposure,
ParsedMetric,
]

View File

@@ -1,7 +1,7 @@
import enum import enum
from dataclasses import dataclass, field from dataclasses import dataclass, field
from itertools import chain, islice from itertools import chain, islice
from mashumaro.mixins.msgpack import DataClassMessagePackMixin from mashumaro import DataClassMessagePackMixin
from multiprocessing.synchronize import Lock from multiprocessing.synchronize import Lock
from typing import ( from typing import (
Dict, Dict,
@@ -16,40 +16,43 @@ from typing import (
TypeVar, TypeVar,
Callable, Callable,
Generic, Generic,
cast,
AbstractSet, AbstractSet,
ClassVar, ClassVar,
) )
from typing_extensions import Protocol from typing_extensions import Protocol
from uuid import UUID from uuid import UUID
from dbt.contracts.graph.nodes import ( from dbt.contracts.graph.compiled import (
Macro, CompileResultNode,
Documentation,
SourceDefinition,
GenericTestNode,
Exposure,
Metric,
Entity,
UnpatchedSourceDefinition,
ManifestNode, ManifestNode,
NonSourceCompiledNode,
GraphMemberNode, GraphMemberNode,
ResultNode, )
BaseNode, from dbt.contracts.graph.parsed import (
ParsedMacro,
ParsedDocumentation,
ParsedSourceDefinition,
ParsedExposure,
ParsedMetric,
HasUniqueID,
UnpatchedSourceDefinition,
ManifestNodes,
) )
from dbt.contracts.graph.unparsed import SourcePatch from dbt.contracts.graph.unparsed import SourcePatch
from dbt.contracts.files import SourceFile, SchemaSourceFile, FileHash, AnySourceFile from dbt.contracts.files import SourceFile, SchemaSourceFile, FileHash, AnySourceFile
from dbt.contracts.util import BaseArtifactMetadata, SourceKey, ArtifactMixin, schema_version from dbt.contracts.util import BaseArtifactMetadata, SourceKey, ArtifactMixin, schema_version
from dbt.dataclass_schema import dbtClassMixin from dbt.dataclass_schema import dbtClassMixin
from dbt.exceptions import ( from dbt.exceptions import (
CompilationError, CompilationException,
DuplicateResourceNameError, raise_duplicate_resource_name,
DuplicateMacroInPackageError, raise_compiler_error,
DuplicateMaterializationNameError,
) )
from dbt.helper_types import PathSet from dbt.helper_types import PathSet
from dbt.events.functions import fire_event from dbt.events.functions import fire_event
from dbt.events.types import MergedFromState from dbt.events.types import MergedFromState
from dbt.node_types import NodeType from dbt.node_types import NodeType
from dbt.ui import line_wrap_message
from dbt import flags from dbt import flags
from dbt import tracking from dbt import tracking
import dbt.utils import dbt.utils
@@ -92,7 +95,7 @@ class DocLookup(dbtClassMixin):
return self.perform_lookup(unique_id, manifest) return self.perform_lookup(unique_id, manifest)
return None return None
def add_doc(self, doc: Documentation): def add_doc(self, doc: ParsedDocumentation):
if doc.name not in self.storage: if doc.name not in self.storage:
self.storage[doc.name] = {} self.storage[doc.name] = {}
self.storage[doc.name][doc.package_name] = doc.unique_id self.storage[doc.name][doc.package_name] = doc.unique_id
@@ -101,9 +104,9 @@ class DocLookup(dbtClassMixin):
for doc in manifest.docs.values(): for doc in manifest.docs.values():
self.add_doc(doc) self.add_doc(doc)
def perform_lookup(self, unique_id: UniqueID, manifest) -> Documentation: def perform_lookup(self, unique_id: UniqueID, manifest) -> ParsedDocumentation:
if unique_id not in manifest.docs: if unique_id not in manifest.docs:
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
f"Doc {unique_id} found in cache but not found in manifest" f"Doc {unique_id} found in cache but not found in manifest"
) )
return manifest.docs[unique_id] return manifest.docs[unique_id]
@@ -123,7 +126,7 @@ class SourceLookup(dbtClassMixin):
return self.perform_lookup(unique_id, manifest) return self.perform_lookup(unique_id, manifest)
return None return None
def add_source(self, source: SourceDefinition): def add_source(self, source: ParsedSourceDefinition):
if source.search_name not in self.storage: if source.search_name not in self.storage:
self.storage[source.search_name] = {} self.storage[source.search_name] = {}
@@ -134,9 +137,9 @@ class SourceLookup(dbtClassMixin):
if hasattr(source, "source_name"): if hasattr(source, "source_name"):
self.add_source(source) self.add_source(source)
def perform_lookup(self, unique_id: UniqueID, manifest: "Manifest") -> SourceDefinition: def perform_lookup(self, unique_id: UniqueID, manifest: "Manifest") -> ParsedSourceDefinition:
if unique_id not in manifest.sources: if unique_id not in manifest.sources:
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
f"Source {unique_id} found in cache but not found in manifest" f"Source {unique_id} found in cache but not found in manifest"
) )
return manifest.sources[unique_id] return manifest.sources[unique_id]
@@ -174,7 +177,7 @@ class RefableLookup(dbtClassMixin):
def perform_lookup(self, unique_id: UniqueID, manifest) -> ManifestNode: def perform_lookup(self, unique_id: UniqueID, manifest) -> ManifestNode:
if unique_id not in manifest.nodes: if unique_id not in manifest.nodes:
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
f"Node {unique_id} found in cache but not found in manifest" f"Node {unique_id} found in cache but not found in manifest"
) )
return manifest.nodes[unique_id] return manifest.nodes[unique_id]
@@ -194,7 +197,7 @@ class MetricLookup(dbtClassMixin):
return self.perform_lookup(unique_id, manifest) return self.perform_lookup(unique_id, manifest)
return None return None
def add_metric(self, metric: Metric): def add_metric(self, metric: ParsedMetric):
if metric.search_name not in self.storage: if metric.search_name not in self.storage:
self.storage[metric.search_name] = {} self.storage[metric.search_name] = {}
@@ -205,48 +208,15 @@ class MetricLookup(dbtClassMixin):
if hasattr(metric, "name"): if hasattr(metric, "name"):
self.add_metric(metric) self.add_metric(metric)
def perform_lookup(self, unique_id: UniqueID, manifest: "Manifest") -> Metric: def perform_lookup(self, unique_id: UniqueID, manifest: "Manifest") -> ParsedMetric:
if unique_id not in manifest.metrics: if unique_id not in manifest.metrics:
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
f"Metric {unique_id} found in cache but not found in manifest" f"Metric {unique_id} found in cache but not found in manifest"
) )
return manifest.metrics[unique_id] return manifest.metrics[unique_id]
class EntityLookup(dbtClassMixin): # This handles both models/seeds/snapshots and sources
def __init__(self, manifest: "Manifest"):
self.storage: Dict[str, Dict[PackageName, UniqueID]] = {}
self.populate(manifest)
def get_unique_id(self, search_name, package: Optional[PackageName]):
return find_unique_id_for_package(self.storage, search_name, package)
def find(self, search_name, package: Optional[PackageName], manifest: "Manifest"):
unique_id = self.get_unique_id(search_name, package)
if unique_id is not None:
return self.perform_lookup(unique_id, manifest)
return None
def add_entity(self, entity: Entity):
if entity.search_name not in self.storage:
self.storage[entity.search_name] = {}
self.storage[entity.search_name][entity.package_name] = entity.unique_id
def populate(self, manifest):
for entity in manifest.entities.values():
if hasattr(entity, "name"):
self.add_entity(entity)
def perform_lookup(self, unique_id: UniqueID, manifest: "Manifest") -> Entity:
if unique_id not in manifest.entities:
raise dbt.exceptions.DbtInternalError(
f"Entity {unique_id} found in cache but not found in manifest"
)
return manifest.entities[unique_id]
# This handles both models/seeds/snapshots and sources/metrics/exposures
class DisabledLookup(dbtClassMixin): class DisabledLookup(dbtClassMixin):
def __init__(self, manifest: "Manifest"): def __init__(self, manifest: "Manifest"):
self.storage: Dict[str, Dict[PackageName, List[Any]]] = {} self.storage: Dict[str, Dict[PackageName, List[Any]]] = {}
@@ -354,7 +324,7 @@ def _sort_values(dct):
def build_node_edges(nodes: List[ManifestNode]): def build_node_edges(nodes: List[ManifestNode]):
"""Build the forward and backward edges on the given list of ManifestNodes """Build the forward and backward edges on the given list of ParsedNodes
and return them as two separate dictionaries, each mapping unique IDs to and return them as two separate dictionaries, each mapping unique IDs to
lists of edges. lists of edges.
""" """
@@ -372,10 +342,10 @@ def build_node_edges(nodes: List[ManifestNode]):
# Build a map of children of macros and generic tests # Build a map of children of macros and generic tests
def build_macro_edges(nodes: List[Any]): def build_macro_edges(nodes: List[Any]):
forward_edges: Dict[str, List[str]] = { forward_edges: Dict[str, List[str]] = {
n.unique_id: [] for n in nodes if n.unique_id.startswith("macro") or n.depends_on_macros n.unique_id: [] for n in nodes if n.unique_id.startswith("macro") or n.depends_on.macros
} }
for node in nodes: for node in nodes:
for unique_id in node.depends_on_macros: for unique_id in node.depends_on.macros:
if unique_id in forward_edges.keys(): if unique_id in forward_edges.keys():
forward_edges[unique_id].append(node.unique_id) forward_edges[unique_id].append(node.unique_id)
return _sort_values(forward_edges) return _sort_values(forward_edges)
@@ -394,7 +364,7 @@ class Locality(enum.IntEnum):
@dataclass @dataclass
class MacroCandidate: class MacroCandidate:
locality: Locality locality: Locality
macro: Macro macro: ParsedMacro
def __eq__(self, other: object) -> bool: def __eq__(self, other: object) -> bool:
if not isinstance(other, MacroCandidate): if not isinstance(other, MacroCandidate):
@@ -432,7 +402,12 @@ class MaterializationCandidate(MacroCandidate):
return NotImplemented return NotImplemented
equal = self.specificity == other.specificity and self.locality == other.locality equal = self.specificity == other.specificity and self.locality == other.locality
if equal: if equal:
raise DuplicateMaterializationNameError(self.macro, other) raise_compiler_error(
"Found two materializations with the name {} (packages {} and "
"{}). dbt cannot resolve this ambiguity".format(
self.macro.name, self.macro.package_name, other.macro.package_name
)
)
return equal return equal
@@ -454,14 +429,16 @@ M = TypeVar("M", bound=MacroCandidate)
class CandidateList(List[M]): class CandidateList(List[M]):
def last(self) -> Optional[Macro]: def last(self) -> Optional[ParsedMacro]:
if not self: if not self:
return None return None
self.sort() self.sort()
return self[-1].macro return self[-1].macro
def _get_locality(macro: Macro, root_project_name: str, internal_packages: Set[str]) -> Locality: def _get_locality(
macro: ParsedMacro, root_project_name: str, internal_packages: Set[str]
) -> Locality:
if macro.package_name == root_project_name: if macro.package_name == root_project_name:
return Locality.Root return Locality.Root
elif macro.package_name in internal_packages: elif macro.package_name in internal_packages:
@@ -487,19 +464,16 @@ class Disabled(Generic[D]):
target: D target: D
MaybeMetricNode = Optional[Union[Metric, Disabled[Metric]]] MaybeMetricNode = Optional[ParsedMetric]
MaybeEntityNode = Optional[Union[Entity, Disabled[Entity]]] MaybeDocumentation = Optional[ParsedDocumentation]
MaybeDocumentation = Optional[Documentation]
MaybeParsedSource = Optional[ MaybeParsedSource = Optional[
Union[ Union[
SourceDefinition, ParsedSourceDefinition,
Disabled[SourceDefinition], Disabled[ParsedSourceDefinition],
] ]
] ]
@@ -517,14 +491,14 @@ def _update_into(dest: MutableMapping[str, T], new_item: T):
""" """
unique_id = new_item.unique_id unique_id = new_item.unique_id
if unique_id not in dest: if unique_id not in dest:
raise dbt.exceptions.DbtRuntimeError( raise dbt.exceptions.RuntimeException(
f"got an update_{new_item.resource_type} call with an " f"got an update_{new_item.resource_type} call with an "
f"unrecognized {new_item.resource_type}: {new_item.unique_id}" f"unrecognized {new_item.resource_type}: {new_item.unique_id}"
) )
existing = dest[unique_id] existing = dest[unique_id]
if new_item.original_file_path != existing.original_file_path: if new_item.original_file_path != existing.original_file_path:
raise dbt.exceptions.DbtRuntimeError( raise dbt.exceptions.RuntimeException(
f"cannot update a {new_item.resource_type} to have a new file path!" f"cannot update a {new_item.resource_type} to have a new file " f"path!"
) )
dest[unique_id] = new_item dest[unique_id] = new_item
@@ -539,7 +513,7 @@ class MacroMethods:
def find_macro_by_name( def find_macro_by_name(
self, name: str, root_project_name: str, package: Optional[str] self, name: str, root_project_name: str, package: Optional[str]
) -> Optional[Macro]: ) -> Optional[ParsedMacro]:
"""Find a macro in the graph by its name and package name, or None for """Find a macro in the graph by its name and package name, or None for
any package. The root project name is used to determine priority: any package. The root project name is used to determine priority:
- locally defined macros come first - locally defined macros come first
@@ -562,7 +536,7 @@ class MacroMethods:
def find_generate_macro_by_name( def find_generate_macro_by_name(
self, component: str, root_project_name: str self, component: str, root_project_name: str
) -> Optional[Macro]: ) -> Optional[ParsedMacro]:
""" """
The `generate_X_name` macros are similar to regular ones, but ignore The `generate_X_name` macros are similar to regular ones, but ignore
imported packages. imported packages.
@@ -631,19 +605,18 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
# is added it must all be added in the __reduce_ex__ method in the # is added it must all be added in the __reduce_ex__ method in the
# args tuple in the right position. # args tuple in the right position.
nodes: MutableMapping[str, ManifestNode] = field(default_factory=dict) nodes: MutableMapping[str, ManifestNode] = field(default_factory=dict)
sources: MutableMapping[str, SourceDefinition] = field(default_factory=dict) sources: MutableMapping[str, ParsedSourceDefinition] = field(default_factory=dict)
macros: MutableMapping[str, Macro] = field(default_factory=dict) macros: MutableMapping[str, ParsedMacro] = field(default_factory=dict)
docs: MutableMapping[str, Documentation] = field(default_factory=dict) docs: MutableMapping[str, ParsedDocumentation] = field(default_factory=dict)
exposures: MutableMapping[str, Exposure] = field(default_factory=dict) exposures: MutableMapping[str, ParsedExposure] = field(default_factory=dict)
metrics: MutableMapping[str, Metric] = field(default_factory=dict) metrics: MutableMapping[str, ParsedMetric] = field(default_factory=dict)
entities: MutableMapping[str, Entity] = field(default_factory=dict)
selectors: MutableMapping[str, Any] = field(default_factory=dict) selectors: MutableMapping[str, Any] = field(default_factory=dict)
files: MutableMapping[str, AnySourceFile] = field(default_factory=dict) files: MutableMapping[str, AnySourceFile] = field(default_factory=dict)
metadata: ManifestMetadata = field(default_factory=ManifestMetadata) metadata: ManifestMetadata = field(default_factory=ManifestMetadata)
flat_graph: Dict[str, Any] = field(default_factory=dict) flat_graph: Dict[str, Any] = field(default_factory=dict)
state_check: ManifestStateCheck = field(default_factory=ManifestStateCheck) state_check: ManifestStateCheck = field(default_factory=ManifestStateCheck)
source_patches: MutableMapping[SourceKey, SourcePatch] = field(default_factory=dict) source_patches: MutableMapping[SourceKey, SourcePatch] = field(default_factory=dict)
disabled: MutableMapping[str, List[GraphMemberNode]] = field(default_factory=dict) disabled: MutableMapping[str, List[CompileResultNode]] = field(default_factory=dict)
env_vars: MutableMapping[str, str] = field(default_factory=dict) env_vars: MutableMapping[str, str] = field(default_factory=dict)
_doc_lookup: Optional[DocLookup] = field( _doc_lookup: Optional[DocLookup] = field(
@@ -658,9 +631,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
_metric_lookup: Optional[MetricLookup] = field( _metric_lookup: Optional[MetricLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None} default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
) )
_entity_lookup: Optional[EntityLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
)
_disabled_lookup: Optional[DisabledLookup] = field( _disabled_lookup: Optional[DisabledLookup] = field(
default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None} default=None, metadata={"serialize": lambda x: None, "deserialize": lambda x: None}
) )
@@ -687,7 +657,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
obj._lock = flags.MP_CONTEXT.Lock() obj._lock = flags.MP_CONTEXT.Lock()
return obj return obj
def sync_update_node(self, new_node: ManifestNode) -> ManifestNode: def sync_update_node(self, new_node: NonSourceCompiledNode) -> NonSourceCompiledNode:
"""update the node with a lock. The only time we should want to lock is """update the node with a lock. The only time we should want to lock is
when compiling an ephemeral ancestor of a node at runtime, because when compiling an ephemeral ancestor of a node at runtime, because
multiple threads could be just-in-time compiling the same ephemeral multiple threads could be just-in-time compiling the same ephemeral
@@ -700,24 +670,21 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
with self._lock: with self._lock:
existing = self.nodes[new_node.unique_id] existing = self.nodes[new_node.unique_id]
if getattr(existing, "compiled", False): if getattr(existing, "compiled", False):
# already compiled # already compiled -> must be a NonSourceCompiledNode
return existing return cast(NonSourceCompiledNode, existing)
_update_into(self.nodes, new_node) _update_into(self.nodes, new_node)
return new_node return new_node
def update_exposure(self, new_exposure: Exposure): def update_exposure(self, new_exposure: ParsedExposure):
_update_into(self.exposures, new_exposure) _update_into(self.exposures, new_exposure)
def update_metric(self, new_metric: Metric): def update_metric(self, new_metric: ParsedMetric):
_update_into(self.metrics, new_metric) _update_into(self.metrics, new_metric)
def update_entity(self, new_entity: Entity):
_update_into(self.entities, new_entity)
def update_node(self, new_node: ManifestNode): def update_node(self, new_node: ManifestNode):
_update_into(self.nodes, new_node) _update_into(self.nodes, new_node)
def update_source(self, new_source: SourceDefinition): def update_source(self, new_source: ParsedSourceDefinition):
_update_into(self.sources, new_source) _update_into(self.sources, new_source)
def build_flat_graph(self): def build_flat_graph(self):
@@ -729,7 +696,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.flat_graph = { self.flat_graph = {
"exposures": {k: v.to_dict(omit_none=False) for k, v in self.exposures.items()}, "exposures": {k: v.to_dict(omit_none=False) for k, v in self.exposures.items()},
"metrics": {k: v.to_dict(omit_none=False) for k, v in self.metrics.items()}, "metrics": {k: v.to_dict(omit_none=False) for k, v in self.metrics.items()},
"entities": {k: v.to_dict(omit_none=False) for k, v in self.entities.items()},
"nodes": {k: v.to_dict(omit_none=False) for k, v in self.nodes.items()}, "nodes": {k: v.to_dict(omit_none=False) for k, v in self.nodes.items()},
"sources": {k: v.to_dict(omit_none=False) for k, v in self.sources.items()}, "sources": {k: v.to_dict(omit_none=False) for k, v in self.sources.items()},
} }
@@ -771,7 +737,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
def find_materialization_macro_by_name( def find_materialization_macro_by_name(
self, project_name: str, materialization_name: str, adapter_type: str self, project_name: str, materialization_name: str, adapter_type: str
) -> Optional[Macro]: ) -> Optional[ParsedMacro]:
candidates: CandidateList = CandidateList( candidates: CandidateList = CandidateList(
chain.from_iterable( chain.from_iterable(
self._materialization_candidates_for( self._materialization_candidates_for(
@@ -792,7 +758,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.nodes.values(), self.nodes.values(),
self.sources.values(), self.sources.values(),
self.metrics.values(), self.metrics.values(),
self.entities.values(),
) )
for resource in all_resources: for resource in all_resources:
resource_type_plural = resource.resource_type.pluralize() resource_type_plural = resource.resource_type.pluralize()
@@ -814,22 +779,19 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
return frozenset(x.database for x in chain(self.nodes.values(), self.sources.values())) return frozenset(x.database for x in chain(self.nodes.values(), self.sources.values()))
def deepcopy(self): def deepcopy(self):
copy = Manifest( return Manifest(
nodes={k: _deepcopy(v) for k, v in self.nodes.items()}, nodes={k: _deepcopy(v) for k, v in self.nodes.items()},
sources={k: _deepcopy(v) for k, v in self.sources.items()}, sources={k: _deepcopy(v) for k, v in self.sources.items()},
macros={k: _deepcopy(v) for k, v in self.macros.items()}, macros={k: _deepcopy(v) for k, v in self.macros.items()},
docs={k: _deepcopy(v) for k, v in self.docs.items()}, docs={k: _deepcopy(v) for k, v in self.docs.items()},
exposures={k: _deepcopy(v) for k, v in self.exposures.items()}, exposures={k: _deepcopy(v) for k, v in self.exposures.items()},
metrics={k: _deepcopy(v) for k, v in self.metrics.items()}, metrics={k: _deepcopy(v) for k, v in self.metrics.items()},
entities={k: _deepcopy(v) for k, v in self.entities.items()},
selectors={k: _deepcopy(v) for k, v in self.selectors.items()}, selectors={k: _deepcopy(v) for k, v in self.selectors.items()},
metadata=self.metadata, metadata=self.metadata,
disabled={k: _deepcopy(v) for k, v in self.disabled.items()}, disabled={k: _deepcopy(v) for k, v in self.disabled.items()},
files={k: _deepcopy(v) for k, v in self.files.items()}, files={k: _deepcopy(v) for k, v in self.files.items()},
state_check=_deepcopy(self.state_check), state_check=_deepcopy(self.state_check),
) )
copy.build_flat_graph()
return copy
def build_parent_and_child_maps(self): def build_parent_and_child_maps(self):
edge_members = list( edge_members = list(
@@ -838,7 +800,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.sources.values(), self.sources.values(),
self.exposures.values(), self.exposures.values(),
self.metrics.values(), self.metrics.values(),
self.entities.values(),
) )
) )
forward_edges, backward_edges = build_node_edges(edge_members) forward_edges, backward_edges = build_node_edges(edge_members)
@@ -864,7 +825,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
docs=self.docs, docs=self.docs,
exposures=self.exposures, exposures=self.exposures,
metrics=self.metrics, metrics=self.metrics,
entities=self.entities,
selectors=self.selectors, selectors=self.selectors,
metadata=self.metadata, metadata=self.metadata,
disabled=self.disabled, disabled=self.disabled,
@@ -886,11 +846,9 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
return self.exposures[unique_id] return self.exposures[unique_id]
elif unique_id in self.metrics: elif unique_id in self.metrics:
return self.metrics[unique_id] return self.metrics[unique_id]
elif unique_id in self.entities:
return self.entities[unique_id]
else: else:
# something terrible has happened # something terrible has happened
raise dbt.exceptions.DbtInternalError( raise dbt.exceptions.InternalException(
"Expected node {} not found in manifest".format(unique_id) "Expected node {} not found in manifest".format(unique_id)
) )
@@ -924,12 +882,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self._metric_lookup = MetricLookup(self) self._metric_lookup = MetricLookup(self)
return self._metric_lookup return self._metric_lookup
@property
def entity_lookup(self) -> EntityLookup:
if self._entity_lookup is None:
self._entity_lookup = EntityLookup(self)
return self._entity_lookup
def rebuild_ref_lookup(self): def rebuild_ref_lookup(self):
self._ref_lookup = RefableLookup(self) self._ref_lookup = RefableLookup(self)
@@ -988,8 +940,8 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
search_name = f"{target_source_name}.{target_table_name}" search_name = f"{target_source_name}.{target_table_name}"
candidates = _search_packages(current_project, node_package) candidates = _search_packages(current_project, node_package)
source: Optional[SourceDefinition] = None source: Optional[ParsedSourceDefinition] = None
disabled: Optional[List[SourceDefinition]] = None disabled: Optional[List[ParsedSourceDefinition]] = None
for pkg in candidates: for pkg in candidates:
source = self.source_lookup.find(search_name, pkg, self) source = self.source_lookup.find(search_name, pkg, self)
@@ -1012,47 +964,13 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
current_project: str, current_project: str,
node_package: str, node_package: str,
) -> MaybeMetricNode: ) -> MaybeMetricNode:
metric: Optional[ParsedMetric] = None
metric: Optional[Metric] = None
disabled: Optional[List[Metric]] = None
candidates = _search_packages(current_project, node_package, target_metric_package) candidates = _search_packages(current_project, node_package, target_metric_package)
for pkg in candidates: for pkg in candidates:
metric = self.metric_lookup.find(target_metric_name, pkg, self) metric = self.metric_lookup.find(target_metric_name, pkg, self)
if metric is not None:
if metric is not None and metric.config.enabled:
return metric return metric
# it's possible that the node is disabled
if disabled is None:
disabled = self.disabled_lookup.find(f"{target_metric_name}", pkg)
if disabled:
return Disabled(disabled[0])
return None
def resolve_entity(
self,
target_entity_name: str,
target_entity_package: Optional[str],
current_project: str,
node_package: str,
) -> MaybeEntityNode:
entity: Optional[Entity] = None
disabled: Optional[List[Entity]] = None
candidates = _search_packages(current_project, node_package, target_entity_package)
for pkg in candidates:
entity = self.entity_lookup.find(target_entity_name, pkg, self)
if entity is not None and entity.config.enabled:
return entity
# it's possible that the node is disabled
if disabled is None:
disabled = self.disabled_lookup.find(f"{target_entity_name}", pkg)
if disabled:
return Disabled(disabled[0])
return None return None
# Called by DocsRuntimeContext.doc # Called by DocsRuntimeContext.doc
@@ -1062,7 +980,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
package: Optional[str], package: Optional[str],
current_project: str, current_project: str,
node_package: str, node_package: str,
) -> Optional[Documentation]: ) -> Optional[ParsedDocumentation]:
"""Resolve the given documentation. This follows the same algorithm as """Resolve the given documentation. This follows the same algorithm as
resolve_ref except the is_enabled checks are unnecessary as docs are resolve_ref except the is_enabled checks are unnecessary as docs are
always enabled. always enabled.
@@ -1081,7 +999,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
adapter, adapter,
other: "WritableManifest", other: "WritableManifest",
selected: AbstractSet[UniqueID], selected: AbstractSet[UniqueID],
favor_state: bool = False,
) -> None: ) -> None:
"""Given the selected unique IDs and a writable manifest, update this """Given the selected unique IDs and a writable manifest, update this
manifest by replacing any unselected nodes with their counterpart. manifest by replacing any unselected nodes with their counterpart.
@@ -1096,28 +1013,40 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
node.resource_type in refables node.resource_type in refables
and not node.is_ephemeral and not node.is_ephemeral
and unique_id not in selected and unique_id not in selected
and ( and not adapter.get_relation(current.database, current.schema, current.identifier)
not adapter.get_relation(current.database, current.schema, current.identifier)
or favor_state
)
): ):
merged.add(unique_id) merged.add(unique_id)
self.nodes[unique_id] = node.replace(deferred=True) self.nodes[unique_id] = node.replace(deferred=True)
# Rebuild the flat_graph, which powers the 'graph' context variable,
# now that we've deferred some nodes
self.build_flat_graph()
# log up to 5 items # log up to 5 items
sample = list(islice(merged, 5)) sample = list(islice(merged, 5))
fire_event(MergedFromState(num_merged=len(merged), sample=sample)) fire_event(MergedFromState(nbr_merged=len(merged), sample=sample))
# Methods that were formerly in ParseResult # Methods that were formerly in ParseResult
def add_macro(self, source_file: SourceFile, macro: Macro): def add_macro(self, source_file: SourceFile, macro: ParsedMacro):
if macro.unique_id in self.macros: if macro.unique_id in self.macros:
# detect that the macro exists and emit an error # detect that the macro exists and emit an error
raise DuplicateMacroInPackageError(macro=macro, macro_mapping=self.macros) other_path = self.macros[macro.unique_id].original_file_path
# subtract 2 for the "Compilation Error" indent
# note that the line wrap eats newlines, so if you want newlines,
# this is the result :(
msg = line_wrap_message(
f"""\
dbt found two macros named "{macro.name}" in the project
"{macro.package_name}".
To fix this error, rename or remove one of the following
macros:
- {macro.original_file_path}
- {other_path}
""",
subtract=2,
)
raise_compiler_error(msg)
self.macros[macro.unique_id] = macro self.macros[macro.unique_id] = macro
source_file.macros.append(macro.unique_id) source_file.macros.append(macro.unique_id)
@@ -1137,62 +1066,45 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.sources[source.unique_id] = source # type: ignore self.sources[source.unique_id] = source # type: ignore
source_file.sources.append(source.unique_id) source_file.sources.append(source.unique_id)
def add_node_nofile(self, node: ManifestNode): def add_node_nofile(self, node: ManifestNodes):
# nodes can't be overwritten! # nodes can't be overwritten!
_check_duplicates(node, self.nodes) _check_duplicates(node, self.nodes)
self.nodes[node.unique_id] = node self.nodes[node.unique_id] = node
def add_node(self, source_file: AnySourceFile, node: ManifestNode, test_from=None): def add_node(self, source_file: AnySourceFile, node: ManifestNodes, test_from=None):
self.add_node_nofile(node) self.add_node_nofile(node)
if isinstance(source_file, SchemaSourceFile): if isinstance(source_file, SchemaSourceFile):
if isinstance(node, GenericTestNode): assert test_from
assert test_from source_file.add_test(node.unique_id, test_from)
source_file.add_test(node.unique_id, test_from)
if isinstance(node, Metric):
source_file.metrics.append(node.unique_id)
if isinstance(node, Exposure):
source_file.exposures.append(node.unique_id)
else: else:
source_file.nodes.append(node.unique_id) source_file.nodes.append(node.unique_id)
def add_exposure(self, source_file: SchemaSourceFile, exposure: Exposure): def add_exposure(self, source_file: SchemaSourceFile, exposure: ParsedExposure):
_check_duplicates(exposure, self.exposures) _check_duplicates(exposure, self.exposures)
self.exposures[exposure.unique_id] = exposure self.exposures[exposure.unique_id] = exposure
source_file.exposures.append(exposure.unique_id) source_file.exposures.append(exposure.unique_id)
def add_metric(self, source_file: SchemaSourceFile, metric: Metric): def add_metric(self, source_file: SchemaSourceFile, metric: ParsedMetric):
_check_duplicates(metric, self.metrics) _check_duplicates(metric, self.metrics)
self.metrics[metric.unique_id] = metric self.metrics[metric.unique_id] = metric
source_file.metrics.append(metric.unique_id) source_file.metrics.append(metric.unique_id)
def add_entity(self, source_file: SchemaSourceFile, entity: Entity): def add_disabled_nofile(self, node: CompileResultNode):
_check_duplicates(entity, self.entities)
self.entities[entity.unique_id] = entity
source_file.entities.append(entity.unique_id)
def add_disabled_nofile(self, node: GraphMemberNode):
# There can be multiple disabled nodes for the same unique_id # There can be multiple disabled nodes for the same unique_id
if node.unique_id in self.disabled: if node.unique_id in self.disabled:
self.disabled[node.unique_id].append(node) self.disabled[node.unique_id].append(node)
else: else:
self.disabled[node.unique_id] = [node] self.disabled[node.unique_id] = [node]
def add_disabled(self, source_file: AnySourceFile, node: ResultNode, test_from=None): def add_disabled(self, source_file: AnySourceFile, node: CompileResultNode, test_from=None):
self.add_disabled_nofile(node) self.add_disabled_nofile(node)
if isinstance(source_file, SchemaSourceFile): if isinstance(source_file, SchemaSourceFile):
if isinstance(node, GenericTestNode): assert test_from
assert test_from source_file.add_test(node.unique_id, test_from)
source_file.add_test(node.unique_id, test_from)
if isinstance(node, Metric):
source_file.metrics.append(node.unique_id)
if isinstance(node, Entity):
source_file.entities.append(node.unique_id)
if isinstance(node, Exposure):
source_file.exposures.append(node.unique_id)
else: else:
source_file.nodes.append(node.unique_id) source_file.nodes.append(node.unique_id)
def add_doc(self, source_file: SourceFile, doc: Documentation): def add_doc(self, source_file: SourceFile, doc: ParsedDocumentation):
_check_duplicates(doc, self.docs) _check_duplicates(doc, self.docs)
self.docs[doc.unique_id] = doc self.docs[doc.unique_id] = doc
source_file.docs.append(doc.unique_id) source_file.docs.append(doc.unique_id)
@@ -1214,7 +1126,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self.docs, self.docs,
self.exposures, self.exposures,
self.metrics, self.metrics,
self.entities,
self.selectors, self.selectors,
self.files, self.files,
self.metadata, self.metadata,
@@ -1227,7 +1138,6 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
self._source_lookup, self._source_lookup,
self._ref_lookup, self._ref_lookup,
self._metric_lookup, self._metric_lookup,
self._entity_lookup,
self._disabled_lookup, self._disabled_lookup,
self._analysis_lookup, self._analysis_lookup,
) )
@@ -1247,35 +1157,32 @@ AnyManifest = Union[Manifest, MacroManifest]
@dataclass @dataclass
@schema_version("manifest", 8) @schema_version("manifest", 6)
class WritableManifest(ArtifactMixin): class WritableManifest(ArtifactMixin):
nodes: Mapping[UniqueID, ManifestNode] = field( nodes: Mapping[UniqueID, ManifestNode] = field(
metadata=dict(description=("The nodes defined in the dbt project and its dependencies")) metadata=dict(description=("The nodes defined in the dbt project and its dependencies"))
) )
sources: Mapping[UniqueID, SourceDefinition] = field( sources: Mapping[UniqueID, ParsedSourceDefinition] = field(
metadata=dict(description=("The sources defined in the dbt project and its dependencies")) metadata=dict(description=("The sources defined in the dbt project and its dependencies"))
) )
macros: Mapping[UniqueID, Macro] = field( macros: Mapping[UniqueID, ParsedMacro] = field(
metadata=dict(description=("The macros defined in the dbt project and its dependencies")) metadata=dict(description=("The macros defined in the dbt project and its dependencies"))
) )
docs: Mapping[UniqueID, Documentation] = field( docs: Mapping[UniqueID, ParsedDocumentation] = field(
metadata=dict(description=("The docs defined in the dbt project and its dependencies")) metadata=dict(description=("The docs defined in the dbt project and its dependencies"))
) )
exposures: Mapping[UniqueID, Exposure] = field( exposures: Mapping[UniqueID, ParsedExposure] = field(
metadata=dict( metadata=dict(
description=("The exposures defined in the dbt project and its dependencies") description=("The exposures defined in the dbt project and its dependencies")
) )
) )
metrics: Mapping[UniqueID, Metric] = field( metrics: Mapping[UniqueID, ParsedMetric] = field(
metadata=dict(description=("The metrics defined in the dbt project and its dependencies")) metadata=dict(description=("The metrics defined in the dbt project and its dependencies"))
) )
entities: Mapping[UniqueID, Entity] = field(
metadata=dict(description=("The entities defined in the dbt project and its dependencies"))
)
selectors: Mapping[UniqueID, Any] = field( selectors: Mapping[UniqueID, Any] = field(
metadata=dict(description=("The selectors defined in selectors.yml")) metadata=dict(description=("The selectors defined in selectors.yml"))
) )
disabled: Optional[Mapping[UniqueID, List[ResultNode]]] = field( disabled: Optional[Mapping[UniqueID, List[CompileResultNode]]] = field(
metadata=dict(description="A mapping of the disabled nodes in the target") metadata=dict(description="A mapping of the disabled nodes in the target")
) )
parent_map: Optional[NodeEdgeMap] = field( parent_map: Optional[NodeEdgeMap] = field(
@@ -1296,8 +1203,7 @@ class WritableManifest(ArtifactMixin):
@classmethod @classmethod
def compatible_previous_versions(self): def compatible_previous_versions(self):
# return [("manifest", 4), ("manifest", 5), ("manifest", 6), ("manifest", 7)] return [("manifest", 4), ("manifest", 5)]
return []
def __post_serialize__(self, dct): def __post_serialize__(self, dct):
for unique_id, node in dct["nodes"].items(): for unique_id, node in dct["nodes"].items():
@@ -1306,9 +1212,9 @@ class WritableManifest(ArtifactMixin):
return dct return dct
def _check_duplicates(value: BaseNode, src: Mapping[str, BaseNode]): def _check_duplicates(value: HasUniqueID, src: Mapping[str, HasUniqueID]):
if value.unique_id in src: if value.unique_id in src:
raise DuplicateResourceNameError(value, src[value.unique_id]) raise_duplicate_resource_name(value, src[value.unique_id])
K_T = TypeVar("K_T") K_T = TypeVar("K_T")
@@ -1317,7 +1223,7 @@ V_T = TypeVar("V_T")
def _expect_value(key: K_T, src: Mapping[K_T, V_T], old_file: SourceFile, name: str) -> V_T: def _expect_value(key: K_T, src: Mapping[K_T, V_T], old_file: SourceFile, name: str) -> V_T:
if key not in src: if key not in src:
raise CompilationError( raise CompilationException(
'Expected to find "{}" in cached "result.{}" based ' 'Expected to find "{}" in cached "result.{}" based '
"on cached file information: {}!".format(key, name, old_file) "on cached file information: {}!".format(key, name, old_file)
) )

View File

@@ -12,7 +12,7 @@ class MetricReference(object):
class ResolvedMetricReference(MetricReference): class ResolvedMetricReference(MetricReference):
""" """
Simple proxy over a Metric which delegates property Simple proxy over a ParsedMetric which delegates property
lookups to the underlying node. Also adds helper functions lookups to the underlying node. Also adds helper functions
for working with metrics (ie. __str__ and templating functions) for working with metrics (ie. __str__ and templating functions)
""" """
@@ -38,56 +38,33 @@ class ResolvedMetricReference(MetricReference):
if node and node.resource_type == NodeType.Metric: if node and node.resource_type == NodeType.Metric:
yield from cls.parent_metrics(node, manifest) yield from cls.parent_metrics(node, manifest)
@classmethod def parent_models(self):
def parent_metrics_names(cls, metric_node, manifest):
yield metric_node.name
for parent_unique_id in metric_node.depends_on.nodes:
node = manifest.metrics.get(parent_unique_id)
if node and node.resource_type == NodeType.Metric:
yield from cls.parent_metrics_names(node, manifest)
@classmethod
def reverse_dag_parsing(cls, metric_node, manifest, metric_depth_count):
if metric_node.calculation_method == "derived":
yield {metric_node.name: metric_depth_count}
metric_depth_count = metric_depth_count + 1
for parent_unique_id in metric_node.depends_on.nodes:
node = manifest.metrics.get(parent_unique_id)
if (
node
and node.resource_type == NodeType.Metric
and node.calculation_method == "derived"
):
yield from cls.reverse_dag_parsing(node, manifest, metric_depth_count)
def full_metric_dependency(self):
to_return = list(set(self.parent_metrics_names(self.node, self.manifest)))
return to_return
def base_metric_dependency(self):
in_scope_metrics = list(self.parent_metrics(self.node, self.manifest)) in_scope_metrics = list(self.parent_metrics(self.node, self.manifest))
to_return = [] to_return = {
"base": [],
"derived": [],
}
for metric in in_scope_metrics: for metric in in_scope_metrics:
if metric.calculation_method != "derived" and metric.name not in to_return: if metric.type == "expression":
to_return.append(metric.name) to_return["derived"].append(
{"metric_source": None, "metric": metric, "is_derived": True}
return to_return )
else:
def derived_metric_dependency(self): for node_unique_id in metric.depends_on.nodes:
in_scope_metrics = list(self.parent_metrics(self.node, self.manifest)) node = self.manifest.nodes.get(node_unique_id)
if node and node.resource_type in NodeType.refable():
to_return = [] to_return["base"].append(
for metric in in_scope_metrics: {
if metric.calculation_method == "derived" and metric.name not in to_return: "metric_relation_node": node,
to_return.append(metric.name) "metric_relation": self.Relation.create(
database=node.database,
return to_return schema=node.schema,
identifier=node.alias,
def derived_metric_dependency_depth(self): ),
metric_depth_count = 1 "metric": metric,
to_return = list(self.reverse_dag_parsing(self.node, self.manifest, metric_depth_count)) "is_derived": False,
}
)
return to_return return to_return

View File

@@ -7,9 +7,8 @@ from dbt.dataclass_schema import (
ValidationError, ValidationError,
register_pattern, register_pattern,
) )
from dbt.contracts.graph.unparsed import AdditionalPropertiesAllowed, Docs from dbt.contracts.graph.unparsed import AdditionalPropertiesAllowed
from dbt.contracts.graph.utils import validate_color from dbt.exceptions import InternalException, CompilationException
from dbt.exceptions import DbtInternalError, CompilationError
from dbt.contracts.util import Replaceable, list_str from dbt.contracts.util import Replaceable, list_str
from dbt import hooks from dbt import hooks
from dbt.node_types import NodeType from dbt.node_types import NodeType
@@ -30,7 +29,7 @@ def _get_meta_value(cls: Type[M], fld: Field, key: str, default: Any) -> M:
try: try:
return cls(value) return cls(value)
except ValueError as exc: except ValueError as exc:
raise DbtInternalError(f"Invalid {cls} value: {value}") from exc raise InternalException(f"Invalid {cls} value: {value}") from exc
def _set_meta_value(obj: M, key: str, existing: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: def _set_meta_value(obj: M, key: str, existing: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
@@ -140,17 +139,17 @@ def _merge_field_value(
return _listify(self_value) + _listify(other_value) return _listify(self_value) + _listify(other_value)
elif merge_behavior == MergeBehavior.Update: elif merge_behavior == MergeBehavior.Update:
if not isinstance(self_value, dict): if not isinstance(self_value, dict):
raise DbtInternalError(f"expected dict, got {self_value}") raise InternalException(f"expected dict, got {self_value}")
if not isinstance(other_value, dict): if not isinstance(other_value, dict):
raise DbtInternalError(f"expected dict, got {other_value}") raise InternalException(f"expected dict, got {other_value}")
value = self_value.copy() value = self_value.copy()
value.update(other_value) value.update(other_value)
return value return value
elif merge_behavior == MergeBehavior.DictKeyAppend: elif merge_behavior == MergeBehavior.DictKeyAppend:
if not isinstance(self_value, dict): if not isinstance(self_value, dict):
raise DbtInternalError(f"expected dict, got {self_value}") raise InternalException(f"expected dict, got {self_value}")
if not isinstance(other_value, dict): if not isinstance(other_value, dict):
raise DbtInternalError(f"expected dict, got {other_value}") raise InternalException(f"expected dict, got {other_value}")
new_dict = {} new_dict = {}
for key in self_value.keys(): for key in self_value.keys():
new_dict[key] = _listify(self_value[key]) new_dict[key] = _listify(self_value[key])
@@ -172,7 +171,7 @@ def _merge_field_value(
return new_dict return new_dict
else: else:
raise DbtInternalError(f"Got an invalid merge_behavior: {merge_behavior}") raise InternalException(f"Got an invalid merge_behavior: {merge_behavior}")
def insensitive_patterns(*patterns: str): def insensitive_patterns(*patterns: str):
@@ -227,7 +226,7 @@ class BaseConfig(AdditionalPropertiesAllowed, Replaceable):
msg = ( msg = (
'Error, tried to delete config key "{}": Cannot delete ' "built-in keys" 'Error, tried to delete config key "{}": Cannot delete ' "built-in keys"
).format(key) ).format(key)
raise CompilationError(msg) raise CompilationException(msg)
else: else:
del self._extra[key] del self._extra[key]
@@ -286,7 +285,7 @@ class BaseConfig(AdditionalPropertiesAllowed, Replaceable):
# 'meta' moved here from node # 'meta' moved here from node
mergebehavior = { mergebehavior = {
"append": ["pre-hook", "pre_hook", "post-hook", "post_hook", "tags"], "append": ["pre-hook", "pre_hook", "post-hook", "post_hook", "tags"],
"update": ["quoting", "column_types", "meta", "docs"], "update": ["quoting", "column_types", "meta"],
"dict_key_append": ["grants"], "dict_key_append": ["grants"],
} }
@@ -363,24 +362,43 @@ class BaseConfig(AdditionalPropertiesAllowed, Replaceable):
return self.from_dict(dct) return self.from_dict(dct)
@dataclass
class MetricConfig(BaseConfig):
enabled: bool = True
@dataclass
class EntityConfig(BaseConfig):
enabled: bool = True
@dataclass
class ExposureConfig(BaseConfig):
enabled: bool = True
@dataclass @dataclass
class SourceConfig(BaseConfig): class SourceConfig(BaseConfig):
enabled: bool = True enabled: bool = True
# to be implmented to complete CT-201
# quoting: Dict[str, Any] = field(
# default_factory=dict,
# metadata=MergeBehavior.Update.meta(),
# )
# freshness: Optional[Dict[str, Any]] = field(
# default=None,
# metadata=CompareBehavior.Exclude.meta(),
# )
# loader: Optional[str] = field(
# default=None,
# metadata=CompareBehavior.Exclude.meta(),
# )
# # TODO what type is this? docs say: "<column_name_or_expression>"
# loaded_at_field: Optional[str] = field(
# default=None,
# metadata=CompareBehavior.Exclude.meta(),
# )
# database: Optional[str] = field(
# default=None,
# metadata=CompareBehavior.Exclude.meta(),
# )
# schema: Optional[str] = field(
# default=None,
# metadata=CompareBehavior.Exclude.meta(),
# )
# meta: Dict[str, Any] = field(
# default_factory=dict,
# metadata=MergeBehavior.Update.meta(),
# )
# tags: Union[List[str], str] = field(
# default_factory=list_str,
# metadata=metas(ShowBehavior.Hide, MergeBehavior.Append, CompareBehavior.Exclude),
# )
@dataclass @dataclass
@@ -415,7 +433,6 @@ class NodeConfig(NodeAndTestConfig):
# Note: if any new fields are added with MergeBehavior, also update the # Note: if any new fields are added with MergeBehavior, also update the
# 'mergebehavior' dictionary # 'mergebehavior' dictionary
materialized: str = "view" materialized: str = "view"
incremental_strategy: Optional[str] = None
persist_docs: Dict[str, Any] = field(default_factory=dict) persist_docs: Dict[str, Any] = field(default_factory=dict)
post_hook: List[Hook] = field( post_hook: List[Hook] = field(
default_factory=list, default_factory=list,
@@ -443,24 +460,6 @@ class NodeConfig(NodeAndTestConfig):
grants: Dict[str, Any] = field( grants: Dict[str, Any] = field(
default_factory=dict, metadata=MergeBehavior.DictKeyAppend.meta() default_factory=dict, metadata=MergeBehavior.DictKeyAppend.meta()
) )
packages: List[str] = field(
default_factory=list,
metadata=MergeBehavior.Append.meta(),
)
docs: Docs = field(
default_factory=Docs,
metadata=MergeBehavior.Update.meta(),
)
# we validate that node_color has a suitable value to prevent dbt-docs from crashing
def __post_init__(self):
if self.docs.node_color:
node_color = self.docs.node_color
if not validate_color(node_color):
raise ValidationError(
f"Invalid color name for docs.node_color: {node_color}. "
"It is neither a valid HTML color name nor a valid HEX code."
)
@classmethod @classmethod
def __pre_deserialize__(cls, data): def __pre_deserialize__(cls, data):
@@ -500,12 +499,6 @@ class SeedConfig(NodeConfig):
materialized: str = "seed" materialized: str = "seed"
quote_columns: Optional[bool] = None quote_columns: Optional[bool] = None
@classmethod
def validate(cls, data):
super().validate(data)
if data.get("materialized") and data.get("materialized") != "seed":
raise ValidationError("A seed must have a materialized value of 'seed'")
@dataclass @dataclass
class TestConfig(NodeAndTestConfig): class TestConfig(NodeAndTestConfig):
@@ -545,12 +538,6 @@ class TestConfig(NodeAndTestConfig):
return False return False
return True return True
@classmethod
def validate(cls, data):
super().validate(data)
if data.get("materialized") and data.get("materialized") != "test":
raise ValidationError("A test must have a materialized value of 'test'")
@dataclass @dataclass
class EmptySnapshotConfig(NodeConfig): class EmptySnapshotConfig(NodeConfig):
@@ -587,6 +574,7 @@ class SnapshotConfig(EmptySnapshotConfig):
f"Invalid value for 'check_cols': {data['check_cols']}. " f"Invalid value for 'check_cols': {data['check_cols']}. "
"Expected 'all' or a list of strings." "Expected 'all' or a list of strings."
) )
elif data.get("strategy") == "timestamp": elif data.get("strategy") == "timestamp":
if not data.get("updated_at"): if not data.get("updated_at"):
raise ValidationError( raise ValidationError(
@@ -598,9 +586,6 @@ class SnapshotConfig(EmptySnapshotConfig):
# If the strategy is not 'check' or 'timestamp' it's a custom strategy, # If the strategy is not 'check' or 'timestamp' it's a custom strategy,
# formerly supported with GenericSnapshotConfig # formerly supported with GenericSnapshotConfig
if data.get("materialized") and data.get("materialized") != "snapshot":
raise ValidationError("A snapshot must have a materialized value of 'snapshot'")
def finalize_and_validate(self): def finalize_and_validate(self):
data = self.to_dict(omit_none=True) data = self.to_dict(omit_none=True)
self.validate(data) self.validate(data)
@@ -608,9 +593,6 @@ class SnapshotConfig(EmptySnapshotConfig):
RESOURCE_TYPES: Dict[NodeType, Type[BaseConfig]] = { RESOURCE_TYPES: Dict[NodeType, Type[BaseConfig]] = {
NodeType.Metric: MetricConfig,
NodeType.Entity: EntityConfig,
NodeType.Exposure: ExposureConfig,
NodeType.Source: SourceConfig, NodeType.Source: SourceConfig,
NodeType.Seed: SeedConfig, NodeType.Seed: SeedConfig,
NodeType.Test: TestConfig, NodeType.Test: TestConfig,

View File

@@ -1,17 +1,9 @@
import re
from dbt import deprecations
from dbt.node_types import NodeType from dbt.node_types import NodeType
from dbt.contracts.util import ( from dbt.contracts.util import AdditionalPropertiesMixin, Mergeable, Replaceable
AdditionalPropertiesMixin,
Mergeable,
Replaceable,
rename_metric_attr,
)
# trigger the PathEncoder # trigger the PathEncoder
import dbt.helper_types # noqa:F401 import dbt.helper_types # noqa:F401
from dbt.exceptions import CompilationError, ParsingError from dbt.exceptions import CompilationException, ParsingException
from dbt.dataclass_schema import dbtClassMixin, StrEnum, ExtensibleDbtClassMixin, ValidationError from dbt.dataclass_schema import dbtClassMixin, StrEnum, ExtensibleDbtClassMixin, ValidationError
@@ -24,6 +16,7 @@ from typing import Optional, List, Union, Dict, Any, Sequence
@dataclass @dataclass
class UnparsedBaseNode(dbtClassMixin, Replaceable): class UnparsedBaseNode(dbtClassMixin, Replaceable):
package_name: str package_name: str
root_path: str
path: str path: str
original_file_path: str original_file_path: str
@@ -33,27 +26,26 @@ class UnparsedBaseNode(dbtClassMixin, Replaceable):
@dataclass @dataclass
class HasCode(dbtClassMixin): class HasSQL:
raw_code: str raw_sql: str
language: str
@property @property
def empty(self): def empty(self):
return not self.raw_code.strip() return not self.raw_sql.strip()
@dataclass @dataclass
class UnparsedMacro(UnparsedBaseNode, HasCode): class UnparsedMacro(UnparsedBaseNode, HasSQL):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Macro]}) resource_type: NodeType = field(metadata={"restrict": [NodeType.Macro]})
@dataclass @dataclass
class UnparsedGenericTest(UnparsedBaseNode, HasCode): class UnparsedGenericTest(UnparsedBaseNode, HasSQL):
resource_type: NodeType = field(metadata={"restrict": [NodeType.Macro]}) resource_type: NodeType = field(metadata={"restrict": [NodeType.Macro]})
@dataclass @dataclass
class UnparsedNode(UnparsedBaseNode, HasCode): class UnparsedNode(UnparsedBaseNode, HasSQL):
name: str name: str
resource_type: NodeType = field( resource_type: NodeType = field(
metadata={ metadata={
@@ -84,7 +76,6 @@ class UnparsedRunHook(UnparsedNode):
@dataclass @dataclass
class Docs(dbtClassMixin, Replaceable): class Docs(dbtClassMixin, Replaceable):
show: bool = True show: bool = True
node_color: Optional[str] = None
@dataclass @dataclass
@@ -222,7 +213,7 @@ class ExternalPartition(AdditionalPropertiesAllowed, Replaceable):
def __post_init__(self): def __post_init__(self):
if self.name == "" or self.data_type == "": if self.name == "" or self.data_type == "":
raise CompilationError("External partition columns must have names and data types") raise CompilationException("External partition columns must have names and data types")
@dataclass @dataclass
@@ -231,7 +222,7 @@ class ExternalTable(AdditionalPropertiesAllowed, Mergeable):
file_format: Optional[str] = None file_format: Optional[str] = None
row_format: Optional[str] = None row_format: Optional[str] = None
tbl_properties: Optional[str] = None tbl_properties: Optional[str] = None
partitions: Optional[Union[List[str], List[ExternalPartition]]] = None partitions: Optional[List[ExternalPartition]] = None
def __bool__(self): def __bool__(self):
return self.location is not None return self.location is not None
@@ -363,6 +354,7 @@ class SourcePatch(dbtClassMixin, Replaceable):
@dataclass @dataclass
class UnparsedDocumentation(dbtClassMixin, Replaceable): class UnparsedDocumentation(dbtClassMixin, Replaceable):
package_name: str package_name: str
root_path: str
path: str path: str
original_file_path: str original_file_path: str
@@ -435,21 +427,11 @@ class UnparsedExposure(dbtClassMixin, Replaceable):
type: ExposureType type: ExposureType
owner: ExposureOwner owner: ExposureOwner
description: str = "" description: str = ""
label: Optional[str] = None
maturity: Optional[MaturityType] = None maturity: Optional[MaturityType] = None
meta: Dict[str, Any] = field(default_factory=dict) meta: Dict[str, Any] = field(default_factory=dict)
tags: List[str] = field(default_factory=list) tags: List[str] = field(default_factory=list)
url: Optional[str] = None url: Optional[str] = None
depends_on: List[str] = field(default_factory=list) depends_on: List[str] = field(default_factory=list)
config: Dict[str, Any] = field(default_factory=dict)
@classmethod
def validate(cls, data):
super(UnparsedExposure, cls).validate(data)
if "name" in data:
# name can only contain alphanumeric chars and underscores
if not (re.match(r"[\w-]+$", data["name"])):
deprecations.warn("exposure-name", exposure=data["name"])
@dataclass @dataclass
@@ -460,94 +442,35 @@ class MetricFilter(dbtClassMixin, Replaceable):
value: str value: str
class MetricTimePeriod(StrEnum):
day = "day"
week = "week"
month = "month"
year = "year"
def plural(self) -> str:
return str(self) + "s"
@dataclass
class MetricTime(dbtClassMixin, Mergeable):
count: Optional[int] = None
period: Optional[MetricTimePeriod] = None
def __bool__(self):
return self.count is not None and self.period is not None
@dataclass @dataclass
class UnparsedMetric(dbtClassMixin, Replaceable): class UnparsedMetric(dbtClassMixin, Replaceable):
# TODO : verify that this disallows metric names with spaces
# TODO: fix validation that you broke :p
# name: Identifier
name: str name: str
label: str label: str
calculation_method: str type: str
expression: str model: Optional[str] = None
description: str = "" description: str = ""
sql: Union[str, int] = ""
timestamp: Optional[str] = None timestamp: Optional[str] = None
time_grains: List[str] = field(default_factory=list) time_grains: List[str] = field(default_factory=list)
dimensions: List[str] = field(default_factory=list) dimensions: List[str] = field(default_factory=list)
window: Optional[MetricTime] = None
model: Optional[str] = None
filters: List[MetricFilter] = field(default_factory=list) filters: List[MetricFilter] = field(default_factory=list)
meta: Dict[str, Any] = field(default_factory=dict) meta: Dict[str, Any] = field(default_factory=dict)
tags: List[str] = field(default_factory=list) tags: List[str] = field(default_factory=list)
config: Dict[str, Any] = field(default_factory=dict)
@classmethod @classmethod
def validate(cls, data): def validate(cls, data):
data = rename_metric_attr(data, raise_deprecation_warning=True) # super().validate(data)
# TODO: putting this back for now to get tests passing. Do we want to implement name: Identifier?
super(UnparsedMetric, cls).validate(data) super(UnparsedMetric, cls).validate(data)
if "name" in data: if "name" in data and " " in data["name"]:
errors = [] raise ParsingException(f"Metrics name '{data['name']}' cannot contain spaces")
if " " in data["name"]:
errors.append("cannot contain spaces")
# This handles failing queries due to too long metric names.
# It only occurs in BigQuery and Snowflake (Postgres/Redshift truncate)
if len(data["name"]) > 250:
errors.append("cannot contain more than 250 characters")
if not (re.match(r"^[A-Za-z]", data["name"])):
errors.append("must begin with a letter")
if not (re.match(r"[\w-]+$", data["name"])):
errors.append("must contain only letters, numbers and underscores")
if errors: # TODO: Expressions _cannot_ have `model` properties
raise ParsingError( if data.get("model") is None and data.get("type") != "expression":
f"The metric name '{data['name']}' is invalid. It {', '.join(e for e in errors)}" raise ValidationError("Non-expression metrics require a 'model' property")
)
if data.get("timestamp") is None and data.get("time_grains") is not None: if data.get("model") is not None and data.get("type") == "expression":
raise ValidationError( raise ValidationError("Expression metrics cannot have a 'model' property")
f"The metric '{data['name']} has time_grains defined but is missing a timestamp dimension."
)
if data.get("timestamp") is None and data.get("window") is not None:
raise ValidationError(
f"The metric '{data['name']} has a window defined but is missing a timestamp dimension."
)
if data.get("model") is None and data.get("calculation_method") != "derived":
raise ValidationError("Non-derived metrics require a 'model' property")
if data.get("model") is not None and data.get("calculation_method") == "derived":
raise ValidationError("Derived metrics cannot have a 'model' property")
@dataclass
class UnparsedEntity(dbtClassMixin, Replaceable):
"""This class is used for entity information"""
name: str
model: str
description: str = ""
dimensions: List[str] = field(default_factory=list)
meta: Dict[str, Any] = field(default_factory=dict)
tags: List[str] = field(default_factory=list)
config: Dict[str, Any] = field(default_factory=dict)
@classmethod
def validate(cls, data):
super(UnparsedEntity, cls).validate(data)
# TODO: Add validation here around include/exclude and others

View File

@@ -1,153 +0,0 @@
import re
HTML_COLORS = [
"aliceblue",
"antiquewhite",
"aqua",
"aquamarine",
"azure",
"beige",
"bisque",
"black",
"blanchedalmond",
"blue",
"blueviolet",
"brown",
"burlywood",
"cadetblue",
"chartreuse",
"chocolate",
"coral",
"cornflowerblue",
"cornsilk",
"crimson",
"cyan",
"darkblue",
"darkcyan",
"darkgoldenrod",
"darkgray",
"darkgreen",
"darkkhaki",
"darkmagenta",
"darkolivegreen",
"darkorange",
"darkorchid",
"darkred",
"darksalmon",
"darkseagreen",
"darkslateblue",
"darkslategray",
"darkturquoise",
"darkviolet",
"deeppink",
"deepskyblue",
"dimgray",
"dodgerblue",
"firebrick",
"floralwhite",
"forestgreen",
"fuchsia",
"gainsboro",
"ghostwhite",
"gold",
"goldenrod",
"gray",
"green",
"greenyellow",
"honeydew",
"hotpink",
"indianred",
"indigo",
"ivory",
"khaki",
"lavender",
"lavenderblush",
"lawngreen",
"lemonchiffon",
"lightblue",
"lightcoral",
"lightcyan",
"lightgoldenrodyellow",
"lightgray",
"lightgreen",
"lightpink",
"lightsalmon",
"lightsalmon",
"lightseagreen",
"lightskyblue",
"lightslategray",
"lightsteelblue",
"lightyellow",
"lime",
"limegreen",
"linen",
"magenta",
"maroon",
"mediumaquamarine",
"mediumblue",
"mediumorchid",
"mediumpurple",
"mediumseagreen",
"mediumslateblue",
"mediumslateblue",
"mediumspringgreen",
"mediumturquoise",
"mediumvioletred",
"midnightblue",
"mintcream",
"mistyrose",
"moccasin",
"navajowhite",
"navy",
"oldlace",
"olive",
"olivedrab",
"orange",
"orangered",
"orchid",
"palegoldenrod",
"palegreen",
"paleturquoise",
"palevioletred",
"papayawhip",
"peachpuff",
"peru",
"pink",
"plum",
"powderblue",
"purple",
"rebeccapurple",
"red",
"rosybrown",
"royalblue",
"saddlebrown",
"salmon",
"sandybrown",
"seagreen",
"seashell",
"sienna",
"silver",
"skyblue",
"slateblue",
"slategray",
"snow",
"springgreen",
"steelblue",
"tan",
"teal",
"thistle",
"tomato",
"turquoise",
"violet",
"wheat",
"white",
"whitesmoke",
"yellow",
"yellowgreen",
]
def validate_color(color: str) -> bool:
match_hex = re.search(r"^#(?:[0-9a-f]{3}){1,2}$", color.lower())
match_html_color_name = color.lower() in HTML_COLORS
return bool(match_hex or match_html_color_name)

View File

@@ -12,7 +12,9 @@ from dataclasses import dataclass, field
from typing import Optional, List, Dict, Union, Any from typing import Optional, List, Dict, Union, Any
from mashumaro.types import SerializableType from mashumaro.types import SerializableType
PIN_PACKAGE_URL = (
"https://docs.getdbt.com/docs/package-management#section-specifying-package-versions" # noqa
)
DEFAULT_SEND_ANONYMOUS_USAGE_STATS = True DEFAULT_SEND_ANONYMOUS_USAGE_STATS = True
@@ -55,12 +57,6 @@ class LocalPackage(Package):
RawVersion = Union[str, float] RawVersion = Union[str, float]
@dataclass
class TarballPackage(Package):
tarball: str
name: str
@dataclass @dataclass
class GitPackage(Package): class GitPackage(Package):
git: str git: str
@@ -88,30 +84,13 @@ class RegistryPackage(Package):
return [str(self.version)] return [str(self.version)]
PackageSpec = Union[LocalPackage, TarballPackage, GitPackage, RegistryPackage] PackageSpec = Union[LocalPackage, GitPackage, RegistryPackage]
@dataclass @dataclass
class PackageConfig(dbtClassMixin, Replaceable): class PackageConfig(dbtClassMixin, Replaceable):
packages: List[PackageSpec] packages: List[PackageSpec]
@classmethod
def validate(cls, data):
for package in data.get("packages", data):
if isinstance(package, dict) and package.get("package"):
if not package["version"]:
raise ValidationError(
f"{package['package']} is missing the version. When installing from the Hub "
"package index, version is a required property"
)
if "/" not in package["package"]:
raise ValidationError(
f"{package['package']} was not found in the package index. Packages on the index "
"require a namespace, e.g dbt-labs/dbt_utils"
)
super().validate(data)
@dataclass @dataclass
class ProjectPackageMetadata: class ProjectPackageMetadata:
@@ -213,9 +192,6 @@ class Project(HyphenatedDbtClassMixin, Replaceable):
analyses: Dict[str, Any] = field(default_factory=dict) analyses: Dict[str, Any] = field(default_factory=dict)
sources: Dict[str, Any] = field(default_factory=dict) sources: Dict[str, Any] = field(default_factory=dict)
tests: Dict[str, Any] = field(default_factory=dict) tests: Dict[str, Any] = field(default_factory=dict)
metrics: Dict[str, Any] = field(default_factory=dict)
entities: Dict[str, Any] = field(default_factory=dict)
exposures: Dict[str, Any] = field(default_factory=dict)
vars: Optional[Dict[str, Any]] = field( vars: Optional[Dict[str, Any]] = field(
default=None, default=None,
metadata=dict( metadata=dict(
@@ -223,7 +199,7 @@ class Project(HyphenatedDbtClassMixin, Replaceable):
), ),
) )
packages: List[PackageSpec] = field(default_factory=list) packages: List[PackageSpec] = field(default_factory=list)
query_comment: Optional[Union[QueryComment, NoValue, str]] = field(default_factory=NoValue) query_comment: Optional[Union[QueryComment, NoValue, str]] = NoValue()
@classmethod @classmethod
def validate(cls, data): def validate(cls, data):
@@ -250,7 +226,6 @@ class UserConfig(ExtensibleDbtClassMixin, Replaceable, UserConfigContract):
printer_width: Optional[int] = None printer_width: Optional[int] = None
write_json: Optional[bool] = None write_json: Optional[bool] = None
warn_error: Optional[bool] = None warn_error: Optional[bool] = None
warn_error_options: Optional[Dict[str, Union[str, List[str]]]] = None
log_format: Optional[str] = None log_format: Optional[str] = None
debug: Optional[bool] = None debug: Optional[bool] = None
version_check: Optional[bool] = None version_check: Optional[bool] = None

Some files were not shown because too many files have changed in this diff Show More