Compare commits

...

22 Commits

Author SHA1 Message Date
github-actions[bot]
d21f2cd090 Bumping version to 1.2.3 and generate changelog (#6262)
* Bumping version to 1.2.3rc1 and generate CHANGELOG

* Updating version to remove rc1

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-11-16 09:47:22 -05:00
github-actions[bot]
d6b5147566 Updates lib to use new profile name functionality (#6202) (#6220)
* Updates lib to use new profile name functionality

* Adds changie entry

* Fixes formatting

(cherry picked from commit d0543c9242)

Co-authored-by: Rachel <41338402+racheldaniel@users.noreply.github.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-11-15 22:09:55 -05:00
github-actions[bot]
e4cbc9790b s/gitlab/github for flake8 precommit repo (#6252) (#6255)
(cherry picked from commit eae98677b9)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2022-11-15 11:11:52 -05:00
Emily Rockman
b25a09d374 add python version and upgrade action (#6204) (#6210)
(cherry picked from commit c3ccbe3357)
2022-11-03 12:44:12 -05:00
github-actions[bot]
111c639f03 more ergonomic profile name handling (#6157) (#6158) 2022-11-02 13:35:51 -05:00
github-actions[bot]
abee0f0dc1 Bumping version to 1.2.2 and generate changelog (#5996)
* Bumping version to 1.2.2rc1 and generate CHANGELOG

* Bumping version to 1.2.2 and generate CHANGELOG (#5997)

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-10-03 14:16:51 -04:00
leahwicz
f903aed3e0 Update Version Bump to include Homebrew in PATH (#5963) (#5993) 2022-10-03 13:37:56 -04:00
github-actions[bot]
1aee7b3700 Runtime: Prevent introspective queries at compile (SL only) (#5926) (#5943)
* Preliminary changes to keep compile from connecting to the warehouse for runtime calls

* Adds option to lib to skip connecting to warehouse for compile; adds prelim tests

* Removes unused imports

* Simplifies test and renames to SqlCompileRunnerNoIntrospection

* Updates name in tests

* Spacing

* Updates test to check for adapter connection call instead of compile and execute

* Removes commented line

* Fixes test names

* Updates plugin to postgres type as snowflake isn't available

* Fixes docstring

* Fixes formatting

* Moves conditional logic out of class

* Fixes formatting

* Removes commented line

* Moves import

* Unmoves import

* Updates changelog

* Adds further info to method docstring

(cherry picked from commit f1326f526c)

Co-authored-by: Rachel <41338402+racheldaniel@users.noreply.github.com>
2022-09-29 15:47:22 +02:00
github-actions[bot]
6278880fa0 Fix adapter reset race condition in lib.py (#5921) (#5931)
* (#5919) Fix adapter reset race condition in lib.py

* run black

* changie

(cherry picked from commit 4e8aa007cf)

Co-authored-by: Drew Banin <drew@dbtlabs.com>
2022-09-27 14:28:54 -04:00
github-actions[bot]
07415ca3e7 Bumping version to 1.2.1 and generate changelog (#5713)
* Bumping version to 1.2.1 and generate CHANGELOG

* Fix Docker file version

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-08-25 13:03:30 -04:00
github-actions[bot]
1d6eb7dcd5 Bumping version to 1.2.1rc2 and generate changelog (#5682)
* Bumping version to 1.2.1rc2 and generate CHANGELOG

* Remove newline

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-08-18 13:02:13 -04:00
github-actions[bot]
7955fe7c3b Change postgres name truncation logic to be overridable. (#5656) (#5681)
* Change postgres name truncation logic to be overridable. Add exception with debugging instructions.

* Add changelog.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
(cherry picked from commit 0d02446e07)

Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com>
2022-08-18 12:10:58 -04:00
github-actions[bot]
f82cddbefe Bumping version to 1.2.1rc1 and generate CHANGELOG (#5641)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-08-11 13:02:45 +02:00
Jeremy Cohen
39901c5155 Use sys.exit instead of exit (#5627) (#5640)
* Use sys.exit instead of exit

* Add changelog

Co-authored-by: varun-dc <91232924+varun-dc@users.noreply.github.com>
2022-08-10 20:49:17 +02:00
github-actions[bot]
6c21d10469 Do not render metrics description field when doing render_data (#5603) (#5619)
(cherry picked from commit eb72dbf32a)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-08-05 15:54:19 -04:00
github-actions[bot]
9a10de926e Add changelog and whitespace fix to version bump Action (#5563) (#5572)
* Add changelog and whitespace fix to version bump Action

* Fixing whitespace

* Remove tabs

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Updating per comments

* Fix whitespace

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
(cherry picked from commit 2547e4f55e)

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-07-29 14:01:32 -04:00
github-actions[bot]
9c32441d05 Fix handling of top-level exceptions (#5560) (#5570)
(cherry picked from commit b43fc76701)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-07-28 16:15:37 -04:00
github-actions[bot]
c7652f6a33 Bumping version to 1.2.0 (#5557)
* Bumping version to 1.2.0

* Whitespace fix

* Changlog update

* Docker file version update

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-07-26 08:28:05 -04:00
github-actions[bot]
a2a531722e Bumping version to 1.2.0rc2 (#5497)
* Bumping version to 1.2.0rc2

* Removing whitespace

* Update Dockerfile

* Changelog update

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-07-20 09:01:13 -04:00
leahwicz
7656ffb9e6 [CT-472] feat: Retrying method for acquiring connection handles (#5432) (#5494)
Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift).

Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
2022-07-19 21:36:04 -04:00
Jeremy Yeo
3f3792882c Fix: Rename try methods to strict (#5477)
* rename strict methods

* add changelog
2022-07-15 08:17:01 -05:00
github-actions[bot]
0db634d12f Bumping version to 1.2.0rc1 (#5458)
* Bumping version to 1.2.0rc1

* Remove whitespace

* Update Changelog

* Update Dockerfile

* Removing dependabot as contributor

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Removing dependabot as contributor

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-07-11 16:58:51 -04:00
104 changed files with 1134 additions and 683 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.2.0b1
current_version = 1.2.3
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)

View File

@@ -1,4 +1,4 @@
## dbt-core 1.2.0-b1 - June 24, 2022
## dbt-core 1.2.0 - July 26, 2022
### Features
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
@@ -10,6 +10,11 @@
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- Allow customizing `target-path` and `log-path` through environment variables and CLI flags. ([#5399](https://github.com/dbt-labs/dbt-core/issues/5399), [#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- Move type_* macros from dbt-utils into dbt-core, with tests ([#5317](https://github.com/dbt-labs/dbt-core/issues/5317), [#5428](https://github.com/dbt-labs/dbt-core/pull/5428))
- Add support for ratio metrics ([#4884](https://github.com/dbt-labs/dbt-core/issues/4884), [#5027](https://github.com/dbt-labs/dbt-core/pull/5027))
- Allow users to define grants as a reasonable default in the dbt_project.yml or within each model sql or yml file combined. ([#5263](https://github.com/dbt-labs/dbt-core/issues/5263), [#5369](https://github.com/dbt-labs/dbt-core/pull/5369))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
### Fixes
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
@@ -29,7 +34,19 @@
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
- Add inheritance to materialization macro resolution ([#4646](https://github.com/dbt-labs/dbt-core/issues/4646), [#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
- Improve pluralizations for Documentation and SqlOperation NodeTypes ([#5352](https://github.com/dbt-labs/dbt-core/issues/5352), [#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- Properly use quotes for Snowflake snapshots when checking all columns ([#2975](https://github.com/dbt-labs/dbt-core/issues/2975), [#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- fixes handling of RESET color code with USE_COLORS=False ([#5288](https://github.com/dbt-labs/dbt-core/issues/5288), [#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
### Docs
- Fixed sample SQL Code for sources when no database is defined ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Add support for `file:` selector in DAG viz ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [Snyk] Upgrade prismjs from 1.27.0 to 1.28.0 ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Run build and tests in CI checks ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Improve metrics DAG viz and documentation page ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Upgrade cytoscape.js fork ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
### Under the Hood
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
@@ -46,24 +63,41 @@
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
- Added the suggested RegEx to check the SemVer string within a package dependency and improved invalid version error handling. ([#5201](https://github.com/dbt-labs/dbt-core/issues/5201), [#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
- Bump manifest version to v6 ([#5417](https://github.com/dbt-labs/dbt-core/issues/5417), [#5430](https://github.com/dbt-labs/dbt-core/pull/5430))
- Add tests for SQL grants ([#5437](https://github.com/dbt-labs/dbt-core/issues/5437), [#5447](https://github.com/dbt-labs/dbt-core/pull/5447))
### Dependencies
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Bump mypy from 0.942 to 0.961 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5337](https://github.com/dbt-labs/dbt-core/pull/5337))
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5388](https://github.com/dbt-labs/dbt-core/pull/5388))
- Bump black from 22.3.0 to 22.6.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5420](https://github.com/dbt-labs/dbt-core/pull/5420))
### Security
- Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Contributors
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- [@b-per](https://github.com/b-per) ([#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- [@darin-reify](https://github.com/darin-reify) ([#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@dependabot[bot]](https://github.com/dependabot[bot]) ([#5141](https://github.com/dbt-labs/dbt-core/pull/5141), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- [@drewbanin](https://github.com/drewbanin) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) ([#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- [@isidentical](https://github.com/isidentical) ([#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- [@tomasfarias](https://github.com/tomasfarias) ([#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@pdebelak](https://github.com/pdebelak) ([#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- [@pquadri](https://github.com/pquadri) ([#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- [@volkangurel](https://github.com/volkangurel) ([#5348](https://github.com/dbt-labs/dbt-core/pull/5348))

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump ubuntu from 20.04 to 22.04"
time: 2022-04-27T19:51:28.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5141"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bumping hologram version"
time: 2022-05-06T16:09:07.000000-05:00
custom:
Author: leahwicz
Issue: "5219"
PR: "5218"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker"
time: 2022-06-13T00:14:56.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5367"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add selector method when reading selector definitions
time: 2022-04-08T11:26:10.713088+10:00
custom:
Author: danieldiamond
Issue: "4821"
PR: "4827"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add set and zip function to contexts
time: 2022-04-23T23:17:56.851793+12:00
custom:
Author: jeremyyeo
Issue: "2345"
PR: "5107"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adds itertools to modules Jinja namespace
time: 2022-04-24T13:26:55.008246+01:00
custom:
Author: bd3dowling
Issue: "5130"
PR: "5140"

View File

@@ -1,7 +0,0 @@
kind: Features
body: allow target as an option in profile_template.yml
time: 2022-04-28T06:56:44.511519-04:00
custom:
Author: alexrosenfeld10
Issue: "5179"
PR: "5184"

View File

@@ -1,7 +0,0 @@
kind: Features
body: 'seed: Add new macro get_csv_sql'
time: 2022-05-03T14:29:34.847959075Z
custom:
Author: adamantike
Issue: "5206"
PR: "5207"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Grants as Node Configs
time: 2022-05-10T20:49:49.197999-04:00
custom:
Author: gshank
Issue: "5189"
PR: "5230"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adds file selectors and support for file selectors in the default method selector
time: 2022-05-12T21:57:48.289674-07:00
custom:
Author: jwills
Issue: "5240"
PR: "5241"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Move cross-db macros from dbt-utils into dbt-core global project
time: 2022-05-18T11:46:04.557104+02:00
custom:
Author: jtcohen6 dbeatty10
Issue: "4813"
PR: "5265"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Prettify duration message at the end of execution
time: 2022-06-11T16:39:31.725960083+01:00
custom:
Author: jared-rimmer
Issue: "5253"
PR: "5364"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Early return from dbt init if no available adapters
time: 2022-06-14T08:20:51.096872718+02:00
custom:
Author: ulisesojeda
Issue: "5365"
PR: "5366"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Adding new cols to check_cols in snapshots
time: 2022-03-17T21:09:16.977086+01:00
custom:
Author: GtheSheep
Issue: "3146"
PR: "4893"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Truncate relation names when appending a suffix that will result in len > 63
characters using make_temp_relation and make_backup_relation macros
time: 2022-03-22T17:37:53.320082-07:00
custom:
Author: epapineau
Issue: "2869"
PR: "4921"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Restore ability to utilize `updated_at` for check_cols snapshots
time: 2022-04-15T11:29:27.063462-06:00
custom:
Author: dbeatty10
Issue: "5076"
PR: "5077"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix retry logic to return values after initial try
time: 2022-04-22T13:12:27.239055-05:00
custom:
Author: emmyoop
Issue: "5023"
PR: "5137"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Use yaml renderer (with target context) for rendering selectors
time: 2022-04-22T13:56:45.147893-04:00
custom:
Author: gshank
Issue: "5131"
PR: "5136"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Scrub secret env vars from CommandError in exception stacktrace
time: 2022-04-25T20:39:24.365495+02:00
custom:
Author: jtcohen6
Issue: "5151"
PR: "5152"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Ensure the metric name does not contain spaces
time: 2022-04-26T20:21:04.360693-04:00
custom:
Author: gshank
Issue: "4572"
PR: "5173"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: When parsing 'all_sources' should be a list of unique dirs
time: 2022-04-27T10:26:48.648388-04:00
custom:
Author: gshank
Issue: "5120"
PR: "5176"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add warning if yaml contains duplicate keys
time: 2022-04-28T10:01:57.893956+12:00
custom:
Author: jeremyyeo
Issue: "5114"
PR: "5146"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Modifying the drop_test_schema to work better with Redshift issues around locked
tables and current transactions
time: 2022-04-29T16:07:42.750046-05:00
custom:
Author: Mcknight-42
Issue: "5200"
PR: "5198"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Fix column comparison in snapshot_check_all_get_existing_columns for check-strategy
snapshots with explicit check_cols defined
time: 2022-05-09T13:00:21.649028+02:00
custom:
Author: jtcohen6
Issue: "5222"
PR: "5223"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Changed how `--select state:modified` detects changes for macros nodes depend
on
time: 2022-05-09T13:13:12.889074-05:00
custom:
Author: stu-k
Issue: "5202"
PR: "5224"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix column comparison in snapshot_check_all_get_existing_columns to use adapter.get_columns_in_relation
time: 2022-05-11T12:32:38.313321+02:00
custom:
Author: jtcohen6
Issue: "5222"
PR: "5232"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove docs file from manifest when removing doc node
time: 2022-05-18T13:46:10.167143-04:00
custom:
Author: gshank
Issue: "4146"
PR: "5270"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove duplicate dbt script entry
time: 2022-06-01T08:13:14.067001+10:00
custom:
Author: groodt
Issue: "5314"
PR: "5304"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Change node ancestor/descendant algo, fixes issue where downstream models aren't
run when using networkx >= 2.8.1
time: 2022-06-01T13:59:08.886215-05:00
custom:
Author: iknox-fa
Issue: "5286"
PR: "5326"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fixing Windows color regression
time: 2022-06-01T19:42:34.263009-04:00
custom:
Author: leahwicz
Issue: "5191"
PR: "5327"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: 'Define compatibility for older manifest versions when using state: selection
methods'
time: 2022-06-08T08:09:14.321735+02:00
custom:
Author: jtcohen6
Issue: "5213"
PR: "5346"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove duplicate key checking introduced in 1.2.0a1
time: 2022-06-23T08:24:31.900647+12:00
custom:
Author: jeremyyeo
Issue: "5331"
PR: "5403"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Migrating 005_simple_seed to the new test framework.
time: 2022-04-09T04:05:39.20045-07:00
custom:
Author: versusfacit
Issue: "200"
PR: "5013"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert 029_docs_generate tests to new framework
time: 2022-04-13T18:30:14.706391-04:00
custom:
Author: gshank
Issue: "5035"
PR: "5058"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Move package deprecation check outside of package cache
time: 2022-04-14T13:22:06.157579-05:00
custom:
Author: emmyoop
Issue: "5068"
PR: "5069"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: removal of scaffold first attempt and create_adapter_plugin.py as they are deprecated new scaffold can be found https://github.com/dbt-labs/dbt-database-adapter-scaffold
time: 2022-04-20T12:00:25.171923-05:00
custom:
Author: McKnight-42
Issue: "4980"
PR: "5117"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Mypy -> 0.942 + fixed import logic to allow for full mypy coverage
time: 2022-04-27T11:21:27.499359-05:00
custom:
Author: iknox-fa
Issue: "4805"
PR: "5171"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Converted dbt list tests to pytest
time: 2022-04-27T14:06:28.882908-05:00
custom:
Author: stu-k
Issue: "5049"
PR: "5178"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: 'Fix: Call str and repr for UnsetProfileConfig without a RuntimeException'
time: 2022-05-03T19:52:12.793729384+02:00
custom:
Author: tomasfarias
Issue: "5081"
PR: "5209"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Improve tracking error logging message
time: 2022-05-04T01:00:31.60387036+02:00
custom:
Author: NicolasPA
Issue: "5197"
PR: "5211"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: 'Clean up materialization logic: more consistent relation names, loading from
cache'
time: 2022-05-09T09:26:28.551068+02:00
custom:
Author: jtcohen6
Issue: "2869"
PR: "4921"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Use the default Python version for local dev and test instead of requiring Python
3.8
time: 2022-05-18T09:51:44.603193-07:00
custom:
Author: jwills
Issue: "5257"
PR: "5269"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix test for context set function
time: 2022-05-18T14:55:22.554316-04:00
custom:
Author: gshank
Issue: "5266"
PR: "5272"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix pip upgrade step in CI for Windows
time: 2022-06-01T10:52:45.872931-04:00
custom:
Author: gshank
Issue: "5321"
PR: "5320"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix unit test test_graph_selection
time: 2022-06-01T11:26:48.725831-04:00
custom:
Author: gshank
Issue: "5323"
PR: "5324"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Update context readme + clean up context code"
time: 2022-06-06T23:03:53.022568+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5334"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: removed script meant for snowflake to snowflake
time: 2022-06-10T11:10:56.080236-05:00
custom:
Author: McKnight-42
Issue: "5361"
PR: "5362"

9
.changes/1.2.1.md Normal file
View File

@@ -0,0 +1,9 @@
## dbt-core 1.2.1 - August 25, 2022
### Fixes
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
### Contributors
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))

9
.changes/1.2.2.md Normal file
View File

@@ -0,0 +1,9 @@
## dbt-core 1.2.2 - October 03, 2022
### Features
- This conditionally no-ops warehouse connection at compile depending on an env var, disabling introspection/queries during compilation only. This is a temporary solution to more complex permissions requirements for the semantic layer. ([#5936](https://github.com/dbt-labs/dbt-core/issues/5936), [#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
### Fixes
- Fix race condition when invoking dbt via lib.py concurrently ([#5919](https://github.com/dbt-labs/dbt-core/issues/5919), [#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
### Contributors
- [@drewbanin](https://github.com/drewbanin) ([#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
- [@racheldaniel](https://github.com/racheldaniel) ([#5926](https://github.com/dbt-labs/dbt-core/pull/5926))

6
.changes/1.2.3.md Normal file
View File

@@ -0,0 +1,6 @@
## dbt-core 1.2.3 - November 16, 2022
### Features
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201), [#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
### Contributors
- [@racheldaniel](https://github.com/racheldaniel) ([#6202](https://github.com/dbt-labs/dbt-core/pull/6202))

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump mypy from 0.942 to 0.961"
time: 2022-06-07T00:09:25.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5337"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core"
time: 2022-06-17T00:18:49.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5388"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump black from 22.3.0 to 22.6.0"
time: 2022-06-29T00:08:31.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5420"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Fixed sample SQL Code for sources when no database is defined
time: 2022-07-06T14:20:29.7347-05:00
custom:
Author: b-per
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Add support for `file:` selector in DAG viz
time: 2022-07-06T14:21:42.544814-05:00
custom:
Author: drewbanin
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: '[Snyk] Upgrade prismjs from 1.27.0 to 1.28.0'
time: 2022-07-06T14:22:26.039137-05:00
custom:
Author: emmyoop
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Run build and tests in CI checks
time: 2022-07-06T14:23:00.842859-05:00
custom:
Author: drewbanin
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Improve metrics DAG viz and documentation page
time: 2022-07-06T14:24:10.120545-05:00
custom:
Author: drewbanin callum-mcdata
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Upgrade cytoscape.js fork
time: 2022-07-06T14:25:05.575755-05:00
custom:
Author: emmyoop
Issue: "5255"
PR: "5446"

View File

@@ -1,8 +0,0 @@
kind: Features
body: 'Allow customizing `target-path` and `log-path` through
environment variables and CLI flags.'
time: 2022-06-22T19:23:29.748577478+03:00
custom:
Author: isidentical
Issue: "5399"
PR: "5402"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Move type_* macros from dbt-utils into dbt-core, with tests
time: 2022-06-30T13:54:52.165139+02:00
custom:
Author: jtcohen6
Issue: "5317"
PR: "5428"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add support for ratio metrics
time: 2022-07-05T08:30:26.494837-05:00
custom:
Author: drewbanin callum-mcdata
Issue: "4884"
PR: "5027"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Allow users to define grants as a reasonable default in the dbt_project.yml
or within each model sql or yml file combined.
time: 2022-07-11T11:15:14.695386-05:00
custom:
Author: McKnight-42
Issue: "5263"
PR: "5369"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add inheritance to materialization macro resolution
time: 2022-06-08T10:42:51.839945-04:00
custom:
Author: volkangurel
Issue: "4646"
PR: "5348"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Improve pluralizations for Documentation and SqlOperation NodeTypes
time: 2022-06-09T11:52:46.578469-05:00
custom:
Author: pdebelak
Issue: "5352"
PR: "5356"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Properly use quotes for Snowflake snapshots when checking all columns
time: 2022-06-17T11:44:43.978834+02:00
custom:
Author: pquadri
Issue: "2975"
PR: "5389"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: fixes handling of RESET color code with USE_COLORS=False
time: 2022-06-17T16:00:27.038058-04:00
custom:
Author: darin-reify
Issue: "5288"
PR: "5394"

View File

@@ -1,7 +0,0 @@
kind: Security
body: Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README
time: 2022-06-06T23:03:53.022568+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5334"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Added the suggested RegEx to check the SemVer string within a package dependency
and improved invalid version error handling.
time: 2022-06-13T17:07:03.773668-05:00
custom:
Author: fivetran-joemarkiewicz
Issue: "5201"
PR: "5370"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: 'Add annotation to render_value method reimplemented in #5334'
time: 2022-06-16T12:05:16.474078+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5382"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Bump manifest version to v6
time: 2022-07-05T14:21:20.66768-04:00
custom:
Author: leahwicz
Issue: "5417"
PR: "5430"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add tests for SQL grants
time: 2022-07-06T21:50:01.498562-04:00
custom:
Author: gshank
Issue: "5437"
PR: "5447"

View File

@@ -44,7 +44,9 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: '3.8'
- name: Install python dependencies
run: |
@@ -80,7 +82,7 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: ${{ matrix.python-version }}
@@ -134,7 +136,7 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: ${{ matrix.python-version }}
@@ -187,9 +189,9 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: 3.8
python-version: '3.8'
- name: Install python dependencies
run: |

View File

@@ -1,18 +1,15 @@
# **what?**
# This workflow will take a version number and a dry run flag. With that
# This workflow will take the new version number to bump to. With that
# it will run versionbump to update the version number everywhere in the
# code base and then generate an update Docker requirements file. If this
# is a dry run, a draft PR will open with the changes. If this isn't a dry
# run, the changes will be committed to the branch this is run on.
# code base and then run changie to create the corresponding changelog.
# A PR will be created with the changes that can be reviewed before committing.
# **why?**
# This is to aid in releasing dbt and making sure we have updated
# the versions and Docker requirements in all places.
# the version in all places and generated the changelog.
# **when?**
# This is triggered either manually OR
# from the repository_dispatch event "version-bump" which is sent from
# the dbt-release repo Action
# This is triggered manually
name: Version Bump
@@ -20,35 +17,21 @@ on:
workflow_dispatch:
inputs:
version_number:
description: 'The version number to bump to'
description: 'The version number to bump to (ex. 1.2.0, 1.3.0b1)'
required: true
is_dry_run:
description: 'Creates a draft PR to allow testing instead of committing to a branch'
required: true
default: 'true'
repository_dispatch:
types: [version-bump]
jobs:
bump:
runs-on: ubuntu-latest
steps:
- name: "[DEBUG] Print Variables"
run: |
echo "all variables defined as inputs"
echo The version_number: ${{ github.event.inputs.version_number }}
- name: Check out the repository
uses: actions/checkout@v2
- name: Set version and dry run values
id: variables
env:
VERSION_NUMBER: "${{ github.event.client_payload.version_number == '' && github.event.inputs.version_number || github.event.client_payload.version_number }}"
IS_DRY_RUN: "${{ github.event.client_payload.is_dry_run == '' && github.event.inputs.is_dry_run || github.event.client_payload.is_dry_run }}"
run: |
echo Repository dispatch event version: ${{ github.event.client_payload.version_number }}
echo Repository dispatch event dry run: ${{ github.event.client_payload.is_dry_run }}
echo Workflow dispatch event version: ${{ github.event.inputs.version_number }}
echo Workflow dispatch event dry run: ${{ github.event.inputs.is_dry_run }}
echo ::set-output name=VERSION_NUMBER::$VERSION_NUMBER
echo ::set-output name=IS_DRY_RUN::$IS_DRY_RUN
- uses: actions/setup-python@v2
with:
python-version: "3.8"
@@ -59,53 +42,80 @@ jobs:
source env/bin/activate
pip install --upgrade pip
- name: Create PR branch
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
- name: Add Homebrew to PATH
run: |
git checkout -b bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
git push origin bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
git branch --set-upstream-to=origin/bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
echo "/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin" >> $GITHUB_PATH
# - name: Generate Docker requirements
# run: |
# source env/bin/activate
# pip install -r requirements.txt
# pip freeze -l > docker/requirements/requirements.txt
# git status
- name: Install Homebrew packages
run: |
brew install pre-commit
brew tap miniscruff/changie https://github.com/miniscruff/changie
brew install changie
- name: Audit Version and Parse Into Parts
id: semver
uses: dbt-labs/actions/parse-semver@v1
with:
version: ${{ github.event.inputs.version_number }}
- name: Set branch value
id: variables
run: |
echo "::set-output name=BRANCH_NAME::prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID"
- name: Create PR branch
run: |
git checkout -b ${{ steps.variables.outputs.BRANCH_NAME }}
git push origin ${{ steps.variables.outputs.BRANCH_NAME }}
git branch --set-upstream-to=origin/${{ steps.variables.outputs.BRANCH_NAME }} ${{ steps.variables.outputs.BRANCH_NAME }}
- name: Bump version
run: |
source env/bin/activate
pip install -r dev-requirements.txt
env/bin/bumpversion --allow-dirty --new-version ${{steps.variables.outputs.VERSION_NUMBER}} major
env/bin/bumpversion --allow-dirty --new-version ${{ github.event.inputs.version_number }} major
git status
- name: Commit version bump directly
uses: EndBug/add-and-commit@v7
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'false' }}
with:
author_name: 'Github Build Bot'
author_email: 'buildbot@fishtownanalytics.com'
message: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
- name: Run changie
run: |
if [[ ${{ steps.semver.outputs.is-pre-release }} -eq 1 ]]
then
changie batch ${{ steps.semver.outputs.base-version }} --move-dir '${{ steps.semver.outputs.base-version }}' --prerelease '${{ steps.semver.outputs.pre-release }}'
else
changie batch ${{ steps.semver.outputs.base-version }} --include '${{ steps.semver.outputs.base-version }}' --remove-prereleases
fi
changie merge
git status
# this step will fail on whitespace errors but also correct them
- name: Remove trailing whitespace
continue-on-error: true
run: |
pre-commit run trailing-whitespace --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
# this step will fail on newline errors but also correct them
- name: Removing extra newlines
continue-on-error: true
run: |
pre-commit run end-of-file-fixer --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
- name: Commit version bump to branch
uses: EndBug/add-and-commit@v7
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
with:
author_name: 'Github Build Bot'
author_email: 'buildbot@fishtownanalytics.com'
message: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
push: 'origin origin/bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
message: 'Bumping version to ${{ github.event.inputs.version_number }} and generate CHANGELOG'
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
push: 'origin origin/${{ steps.variables.outputs.BRANCH_NAME }}'
- name: Create Pull Request
uses: peter-evans/create-pull-request@v3
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
with:
author: 'Github Build Bot <buildbot@fishtownanalytics.com>'
draft: true
base: ${{github.ref}}
title: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
title: 'Bumping version to ${{ github.event.inputs.version_number }} and generate changelog'
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
labels: |
Skip Changelog

View File

@@ -35,7 +35,7 @@ repos:
- "--target-version=py38"
- "--check"
- "--diff"
- repo: https://gitlab.com/pycqa/flake8
- repo: https://github.com/pycqa/flake8
rev: 4.0.1
hooks:
- id: flake8

View File

@@ -6,7 +6,37 @@
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.2.0-b1 - June 24, 2022
## dbt-core 1.2.3 - November 16, 2022
### Features
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201), [#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
### Contributors
- [@racheldaniel](https://github.com/racheldaniel) ([#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
## dbt-core 1.2.2 - October 03, 2022
### Features
- This conditionally no-ops warehouse connection at compile depending on an env var, disabling introspection/queries during compilation only. This is a temporary solution to more complex permissions requirements for the semantic layer. ([#5936](https://github.com/dbt-labs/dbt-core/issues/5936), [#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
### Fixes
- Fix race condition when invoking dbt via lib.py concurrently ([#5919](https://github.com/dbt-labs/dbt-core/issues/5919), [#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
### Contributors
- [@drewbanin](https://github.com/drewbanin) ([#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
- [@racheldaniel](https://github.com/racheldaniel) ([#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
## dbt-core 1.2.1 - August 25, 2022
### Fixes
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
### Contributors
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
## dbt-core 1.2.0 - July 26, 2022
### Features
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
@@ -18,6 +48,11 @@
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- Allow customizing `target-path` and `log-path` through environment variables and CLI flags. ([#5399](https://github.com/dbt-labs/dbt-core/issues/5399), [#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- Move type_* macros from dbt-utils into dbt-core, with tests ([#5317](https://github.com/dbt-labs/dbt-core/issues/5317), [#5428](https://github.com/dbt-labs/dbt-core/pull/5428))
- Add support for ratio metrics ([#4884](https://github.com/dbt-labs/dbt-core/issues/4884), [#5027](https://github.com/dbt-labs/dbt-core/pull/5027))
- Allow users to define grants as a reasonable default in the dbt_project.yml or within each model sql or yml file combined. ([#5263](https://github.com/dbt-labs/dbt-core/issues/5263), [#5369](https://github.com/dbt-labs/dbt-core/pull/5369))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
### Fixes
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
@@ -37,7 +72,19 @@
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
- Add inheritance to materialization macro resolution ([#4646](https://github.com/dbt-labs/dbt-core/issues/4646), [#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
- Improve pluralizations for Documentation and SqlOperation NodeTypes ([#5352](https://github.com/dbt-labs/dbt-core/issues/5352), [#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- Properly use quotes for Snowflake snapshots when checking all columns ([#2975](https://github.com/dbt-labs/dbt-core/issues/2975), [#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- fixes handling of RESET color code with USE_COLORS=False ([#5288](https://github.com/dbt-labs/dbt-core/issues/5288), [#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
### Docs
- Fixed sample SQL Code for sources when no database is defined ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Add support for `file:` selector in DAG viz ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [Snyk] Upgrade prismjs from 1.27.0 to 1.28.0 ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Run build and tests in CI checks ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Improve metrics DAG viz and documentation page ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Upgrade cytoscape.js fork ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
### Under the Hood
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
@@ -54,27 +101,44 @@
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
- Added the suggested RegEx to check the SemVer string within a package dependency and improved invalid version error handling. ([#5201](https://github.com/dbt-labs/dbt-core/issues/5201), [#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
- Bump manifest version to v6 ([#5417](https://github.com/dbt-labs/dbt-core/issues/5417), [#5430](https://github.com/dbt-labs/dbt-core/pull/5430))
- Add tests for SQL grants ([#5437](https://github.com/dbt-labs/dbt-core/issues/5437), [#5447](https://github.com/dbt-labs/dbt-core/pull/5447))
### Dependencies
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Bump mypy from 0.942 to 0.961 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5337](https://github.com/dbt-labs/dbt-core/pull/5337))
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5388](https://github.com/dbt-labs/dbt-core/pull/5388))
- Bump black from 22.3.0 to 22.6.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5420](https://github.com/dbt-labs/dbt-core/pull/5420))
### Security
- Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Contributors
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- [@b-per](https://github.com/b-per) ([#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- [@darin-reify](https://github.com/darin-reify) ([#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@dependabot[bot]](https://github.com/dependabot[bot]) ([#5141](https://github.com/dbt-labs/dbt-core/pull/5141), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- [@drewbanin](https://github.com/drewbanin) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) ([#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- [@isidentical](https://github.com/isidentical) ([#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- [@tomasfarias](https://github.com/tomasfarias) ([#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@pdebelak](https://github.com/pdebelak) ([#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- [@pquadri](https://github.com/pquadri) ([#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- [@volkangurel](https://github.com/volkangurel) ([#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
## Previous Releases

View File

@@ -1,10 +1,24 @@
import abc
import os
from time import sleep
import sys
# multiprocessing.RLock is a function returning this type
from multiprocessing.synchronize import RLock
from threading import get_ident
from typing import Dict, Tuple, Hashable, Optional, ContextManager, List
from typing import (
Any,
Dict,
Tuple,
Hashable,
Optional,
ContextManager,
List,
Type,
Union,
Iterable,
Callable,
)
import agate
@@ -21,6 +35,7 @@ from dbt.contracts.graph.manifest import Manifest
from dbt.adapters.base.query_headers import (
MacroQueryStringSetter,
)
from dbt.events import AdapterLogger
from dbt.events.functions import fire_event
from dbt.events.types import (
NewConnection,
@@ -34,6 +49,9 @@ from dbt.events.types import (
)
from dbt import flags
SleepTime = Union[int, float] # As taken by time.sleep.
AdapterHandle = Any # Adapter connection handle objects can be any class.
class BaseConnectionManager(metaclass=abc.ABCMeta):
"""Methods to implement:
@@ -159,6 +177,94 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
conn.name = conn_name
return conn
@classmethod
def retry_connection(
cls,
connection: Connection,
connect: Callable[[], AdapterHandle],
logger: AdapterLogger,
retryable_exceptions: Iterable[Type[Exception]],
retry_limit: int = 1,
retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1,
_attempts: int = 0,
) -> Connection:
"""Given a Connection, set its handle by calling connect.
The calls to connect will be retried up to retry_limit times to deal with transient
connection errors. By default, one retry will be attempted if retryable_exceptions is set.
:param Connection connection: An instance of a Connection that needs a handle to be set,
usually when attempting to open it.
:param connect: A callable that returns the appropiate connection handle for a
given adapter. This callable will be retried retry_limit times if a subclass of any
Exception in retryable_exceptions is raised by connect.
:type connect: Callable[[], AdapterHandle]
:param AdapterLogger logger: A logger to emit messages on retry attempts or errors. When
handling expected errors, we call debug, and call warning on unexpected errors or when
all retry attempts have been exhausted.
:param retryable_exceptions: An iterable of exception classes that if raised by
connect should trigger a retry.
:type retryable_exceptions: Iterable[Type[Exception]]
:param int retry_limit: How many times to retry the call to connect. If this limit
is exceeded before a successful call, a FailedToConnectException will be raised.
Must be non-negative.
:param retry_timeout: Time to wait between attempts to connect. Can also take a
Callable that takes the number of attempts so far, beginning at 0, and returns an int
or float to be passed to time.sleep.
:type retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1
:param int _attempts: Parameter used to keep track of the number of attempts in calling the
connect function across recursive calls. Passed as an argument to retry_timeout if it
is a Callable. This parameter should not be set by the initial caller.
:raises dbt.exceptions.FailedToConnectException: Upon exhausting all retry attempts without
successfully acquiring a handle.
:return: The given connection with its appropriate state and handle attributes set
depending on whether we successfully acquired a handle or not.
"""
timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout
if timeout < 0:
raise dbt.exceptions.FailedToConnectException(
"retry_timeout cannot be negative or return a negative time."
)
if retry_limit < 0 or retry_limit > sys.getrecursionlimit():
# This guard is not perfect others may add to the recursion limit (e.g. built-ins).
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException("retry_limit cannot be negative")
try:
connection.handle = connect()
connection.state = ConnectionState.OPEN
return connection
except tuple(retryable_exceptions) as e:
if retry_limit <= 0:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException(str(e))
logger.debug(
f"Got a retryable error when attempting to open a {cls.TYPE} connection.\n"
f"{retry_limit} attempts remaining. Retrying in {timeout} seconds.\n"
f"Error:\n{e}"
)
sleep(timeout)
return cls.retry_connection(
connection=connection,
connect=connect,
logger=logger,
retry_limit=retry_limit - 1,
retry_timeout=retry_timeout,
retryable_exceptions=retryable_exceptions,
_attempts=_attempts + 1,
)
except Exception as e:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException(str(e))
@abc.abstractmethod
def cancel_open(self) -> Optional[List[str]]:
"""Cancel all open connections on the adapter. (passable)"""

View File

@@ -1,3 +1,4 @@
import re
import threading
from copy import deepcopy
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
@@ -413,9 +414,24 @@ class RelationsCache:
:raises InternalError: If the new key is already present.
"""
if new_key in self.relations:
# Tell user when collision caused by model names truncated during
# materialization.
match = re.search("__dbt_backup|__dbt_tmp$", new_key.identifier)
if match:
truncated_model_name_prefix = new_key.identifier[: match.start()]
message_addendum = (
"\n\nName collisions can occur when the length of two "
"models' names approach your database's builtin limit. "
"Try restructuring your project such that no two models "
"share the prefix '{}'.".format(truncated_model_name_prefix)
+ " Then, clean your warehouse of any removed models."
)
else:
message_addendum = ""
dbt.exceptions.raise_cache_inconsistent(
"in rename, new key {} already in cache: {}".format(
new_key, list(self.relations.keys())
"in rename, new key {} already in cache: {}{}".format(
new_key, list(self.relations.keys()), message_addendum
)
)

View File

@@ -3,31 +3,42 @@ import os
from copy import deepcopy
from dataclasses import dataclass, field
from pathlib import Path
from typing import Dict, Any, Optional, Mapping, Iterator, Iterable, Tuple, List, MutableSet, Type
from typing import (
Any,
Dict,
Iterable,
Iterator,
List,
Mapping,
MutableSet,
Optional,
Tuple,
Type,
Union,
)
from .profile import Profile
from .project import Project
from .renderer import DbtProjectYamlRenderer, ProfileRenderer
from .utils import parse_cli_vars
from dbt import flags
from dbt.adapters.factory import get_relation_class_by_name, get_include_paths
from dbt.helper_types import FQNPath, PathSet, DictDefaultEmptyStr
from dbt.adapters.factory import get_include_paths, get_relation_class_by_name
from dbt.config.profile import read_user_config
from dbt.contracts.connection import AdapterRequiredConfig, Credentials
from dbt.contracts.graph.manifest import ManifestMetadata
from dbt.contracts.relation import ComponentName
from dbt.ui import warning_tag
from dbt.contracts.project import Configuration, UserConfig
from dbt.contracts.relation import ComponentName
from dbt.dataclass_schema import ValidationError
from dbt.exceptions import (
RuntimeException,
DbtProjectError,
RuntimeException,
raise_compiler_error,
validator_error_message,
warn_or_error,
raise_compiler_error,
)
from dbt.helper_types import DictDefaultEmptyStr, FQNPath, PathSet
from dbt.ui import warning_tag
from dbt.dataclass_schema import ValidationError
from .profile import Profile
from .project import Project, PartialProject
from .renderer import DbtProjectYamlRenderer, ProfileRenderer
from .utils import parse_cli_vars
def _project_quoting_dict(proj: Project, profile: Profile) -> Dict[ComponentName, bool]:
@@ -188,28 +199,52 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
@classmethod
def collect_parts(cls: Type["RuntimeConfig"], args: Any) -> Tuple[Project, Profile]:
# profile_name from the project
project_root = args.project_dir if args.project_dir else os.getcwd()
version_check = bool(flags.VERSION_CHECK)
partial = Project.partial_load(project_root, verify_version=version_check)
# build the profile using the base renderer and the one fact we know
# Note: only the named profile section is rendered. The rest of the
# profile is ignored.
cli_vars: Dict[str, Any] = parse_cli_vars(getattr(args, "vars", "{}"))
profile = cls.collect_profile(args=args)
project_renderer = DbtProjectYamlRenderer(profile, cli_vars)
project = cls.collect_project(args=args, project_renderer=project_renderer)
assert type(project) is Project
return (project, profile)
@classmethod
def collect_profile(
cls: Type["RuntimeConfig"], args: Any, profile_name: Optional[str] = None
) -> Profile:
cli_vars: Dict[str, Any] = parse_cli_vars(getattr(args, "vars", "{}"))
profile_renderer = ProfileRenderer(cli_vars)
profile_name = partial.render_profile_name(profile_renderer)
# build the profile using the base renderer and the one fact we know
if profile_name is None:
# Note: only the named profile section is rendered here. The rest of the
# profile is ignored.
partial = cls.collect_project(args)
assert type(partial) is PartialProject
profile_name = partial.render_profile_name(profile_renderer)
profile = cls._get_rendered_profile(args, profile_renderer, profile_name)
# Save env_vars encountered in rendering for partial parsing
profile.profile_env_vars = profile_renderer.ctx_obj.env_vars
return profile
# get a new renderer using our target information and render the
# project
project_renderer = DbtProjectYamlRenderer(profile, cli_vars)
project = partial.render(project_renderer)
# Save env_vars encountered in rendering for partial parsing
project.project_env_vars = project_renderer.ctx_obj.env_vars
return (project, profile)
@classmethod
def collect_project(
cls: Type["RuntimeConfig"],
args: Any,
project_renderer: Optional[DbtProjectYamlRenderer] = None,
) -> Union[Project, PartialProject]:
project_root = args.project_dir if args.project_dir else os.getcwd()
version_check = bool(flags.VERSION_CHECK)
partial = Project.partial_load(project_root, verify_version=version_check)
if project_renderer is None:
return partial
else:
project = partial.render(project_renderer)
project.project_env_vars = project_renderer.ctx_obj.env_vars
return project
# Called in main.py, lib.py, task/base.py
@classmethod

View File

@@ -474,19 +474,17 @@ class BaseContext(metaclass=ContextMeta):
@contextmember
@staticmethod
def try_set(value: Iterable[Any]) -> Set[Any]:
"""The `try_set` context method can be used to convert any iterable
def set_strict(value: Iterable[Any]) -> Set[Any]:
"""The `set_strict` context method can be used to convert any iterable
to a sequence of iterable elements that are unique (a set). The
difference to the `set` context method is that the `try_set` method
difference to the `set` context method is that the `set_strict` method
will raise an exception on a TypeError.
:param value: The iterable
:param default: A default value to return if the `value` argument
is not an iterable
Usage:
{% set my_list = [1, 2, 2, 3] %}
{% set my_set = try_set(my_list) %}
{% set my_set = set_strict(my_list) %}
{% do log(my_set) %} {# {1, 2, 3} #}
"""
try:
@@ -497,7 +495,7 @@ class BaseContext(metaclass=ContextMeta):
@contextmember("zip")
@staticmethod
def _zip(*args: Iterable[Any], default: Any = None) -> Optional[Iterable[Any]]:
"""The `try_zip` context method can be used to used to return
"""The `zip` context method can be used to used to return
an iterator of tuples, where the i-th tuple contains the i-th
element from each of the argument iterables.
@@ -518,21 +516,19 @@ class BaseContext(metaclass=ContextMeta):
@contextmember
@staticmethod
def try_zip(*args: Iterable[Any]) -> Iterable[Any]:
"""The `try_zip` context method can be used to used to return
def zip_strict(*args: Iterable[Any]) -> Iterable[Any]:
"""The `zip_strict` context method can be used to used to return
an iterator of tuples, where the i-th tuple contains the i-th
element from each of the argument iterables. The difference to the
`zip` context method is that the `try_zip` method will raise an
`zip` context method is that the `zip_strict` method will raise an
exception on a TypeError.
:param *args: Any number of iterables
:param default: A default value to return if `*args` is not
iterable
Usage:
{% set my_list_a = [1, 2] %}
{% set my_list_b = ['alice', 'bob'] %}
{% set my_zip = try_zip(my_list_a, my_list_b) | list %}
{% set my_zip = zip_strict(my_list_a, my_list_b) | list %}
{% do log(my_set) %} {# [(1, 'alice'), (2, 'bob')] #}
"""
try:

View File

@@ -105,6 +105,8 @@ class ErrorLevel(EventSerialization, Event):
# prevents an event from going to the file
# This should rarely be used in core code. It is currently
# only used in integration tests and for the 'clean' command.
class NoFile:
pass

View File

@@ -218,7 +218,7 @@ def create_log_line(e: T_Event, file_output=False) -> Optional[str]:
return create_info_text_log_line(e) # console output
# allows for resuse of this obnoxious if else tree.
# allows for reuse of this obnoxious if else tree.
# do not use for exceptions, it doesn't pass along exc_info, stack_info, or extra
def send_to_logger(l: Union[Logger, logbook.Logger], level_tag: str, log_line: str):
if not log_line:

View File

@@ -1,5 +1,6 @@
from dataclasses import dataclass
from .types import InfoLevel, DebugLevel, WarnLevel, ErrorLevel, ShowException, NoFile
from dbt.events.types import InfoLevel, DebugLevel, WarnLevel, ErrorLevel, ShowException
from dbt.events.base_types import NoFile
# Keeping log messages for testing separate since they are used for debugging.

View File

@@ -94,7 +94,7 @@ class AdapterEventError(ErrorLevel, AdapterEventBase, ShowException):
@dataclass
class MainKeyboardInterrupt(InfoLevel, NoFile):
class MainKeyboardInterrupt(InfoLevel):
code: str = "Z001"
def message(self) -> str:
@@ -102,7 +102,7 @@ class MainKeyboardInterrupt(InfoLevel, NoFile):
@dataclass
class MainEncounteredError(ErrorLevel, NoFile):
class MainEncounteredError(ErrorLevel):
e: BaseException
code: str = "Z002"
@@ -111,7 +111,7 @@ class MainEncounteredError(ErrorLevel, NoFile):
@dataclass
class MainStackTrace(DebugLevel, NoFile):
class MainStackTrace(ErrorLevel):
stack_trace: str
code: str = "Z003"
@@ -1353,6 +1353,8 @@ class NodeConnectionReleaseError(ShowException, DebugLevel):
return "Error releasing connection for node {}: {!s}".format(self.node_name, self.exc)
# We don't write "clean" events to the log, because the clean command
# may have removed the log directory.
@dataclass
class CheckCleanPath(InfoLevel, NoFile):
path: str
@@ -2382,7 +2384,7 @@ class SendEventFailure(DebugLevel):
@dataclass
class FlushEvents(DebugLevel, NoFile):
class FlushEvents(DebugLevel):
code: str = "Z042"
def message(self) -> str:
@@ -2390,7 +2392,7 @@ class FlushEvents(DebugLevel, NoFile):
@dataclass
class FlushEventsFailure(DebugLevel, NoFile):
class FlushEventsFailure(DebugLevel):
code: str = "Z043"
def message(self) -> str:

View File

@@ -1,10 +1,66 @@
# TODO: this file is one big TODO
import os
from dbt.config.project import Project
from dbt.config.renderer import DbtProjectYamlRenderer
from dbt.contracts.results import RunningStatus, collect_timing_info
from dbt.events.functions import fire_event
from dbt.events.types import NodeCompiling, NodeExecuting
from dbt.exceptions import RuntimeException
from dbt import flags
from collections import namedtuple
from dbt.task.sql import SqlCompileRunner
from dataclasses import dataclass
RuntimeArgs = namedtuple("RuntimeArgs", "project_dir profiles_dir single_threaded profile target")
@dataclass
class RuntimeArgs:
project_dir: str
profiles_dir: str
single_threaded: bool
profile: str
target: str
class SqlCompileRunnerNoIntrospection(SqlCompileRunner):
def compile_and_execute(self, manifest, ctx):
"""
This version of this method does not connect to the data warehouse.
As a result, introspective queries at compilation will not be supported
and will throw an error.
TODO: This is a temporary solution to more complex permissions requirements
for the semantic layer, and thus largely duplicates the code in the parent class
method. Once conditional credential usage is enabled, this should be removed.
"""
result = None
ctx.node._event_status["node_status"] = RunningStatus.Compiling
fire_event(
NodeCompiling(
node_info=ctx.node.node_info,
unique_id=ctx.node.unique_id,
)
)
with collect_timing_info("compile") as timing_info:
# if we fail here, we still have a compiled node to return
# this has the benefit of showing a build path for the errant
# model
ctx.node = self.compile(manifest)
ctx.timing.append(timing_info)
# for ephemeral nodes, we only want to compile, not run
if not ctx.node.is_ephemeral_model:
ctx.node._event_status["node_status"] = RunningStatus.Executing
fire_event(
NodeExecuting(
node_info=ctx.node.node_info,
unique_id=ctx.node.unique_id,
)
)
with collect_timing_info("execute") as timing_info:
result = self.run(ctx.node, manifest)
ctx.node = result.node
ctx.timing.append(timing_info)
return result
def get_dbt_config(project_dir, args=None, single_threaded=False):
@@ -17,27 +73,36 @@ def get_dbt_config(project_dir, args=None, single_threaded=False):
else:
profiles_dir = os.path.expanduser("~/.dbt")
profile = args.profile if hasattr(args, "profile") else None
target = args.target if hasattr(args, "target") else None
profile_name = getattr(args, "profile", None)
# Construct a phony config
config = RuntimeConfig.from_args(
RuntimeArgs(project_dir, profiles_dir, single_threaded, profile, target)
runtime_args = RuntimeArgs(
project_dir=project_dir,
profiles_dir=profiles_dir,
single_threaded=single_threaded,
profile=profile_name,
target=getattr(args, "target", None),
)
# Clear previously registered adapters--
# this fixes cacheing behavior on the dbt-server
profile = RuntimeConfig.collect_profile(args=runtime_args, profile_name=profile_name)
project_renderer = DbtProjectYamlRenderer(profile, None)
project = RuntimeConfig.collect_project(args=runtime_args, project_renderer=project_renderer)
assert type(project) is Project
config = RuntimeConfig.from_parts(project, profile, runtime_args)
# Set global flags from arguments
flags.set_from_args(args, config)
dbt.adapters.factory.reset_adapters()
# Load the relevant adapter
# This is idempotent, so we can call it repeatedly
dbt.adapters.factory.register_adapter(config)
# Set invocation id
# Make sure we have a valid invocation_id
dbt.events.functions.set_invocation_id()
return config
def get_task_by_type(type):
# TODO: we need to tell dbt-server what tasks are available
from dbt.task.run import RunTask
from dbt.task.list import ListTask
from dbt.task.seed import SeedTask
@@ -70,16 +135,13 @@ def create_task(type, args, manifest, config):
def no_op(*args, **kwargs):
pass
# TODO: yuck, let's rethink tasks a little
task = task(args, config)
# Wow! We can monkeypatch taskCls.load_manifest to return _our_ manifest
task.load_manifest = no_op
task.manifest = manifest
return task
def _get_operation_node(manifest, project_path, sql):
def _get_operation_node(manifest, project_path, sql, node_name):
from dbt.parser.manifest import process_node
from dbt.parser.sql import SqlBlockParser
import dbt.adapters.factory
@@ -92,26 +154,33 @@ def _get_operation_node(manifest, project_path, sql):
)
adapter = dbt.adapters.factory.get_adapter(config)
# TODO : This needs a real name?
sql_node = block_parser.parse_remote(sql, "name")
sql_node = block_parser.parse_remote(sql, node_name)
process_node(config, manifest, sql_node)
return config, sql_node, adapter
def compile_sql(manifest, project_path, sql):
from dbt.task.sql import SqlCompileRunner
def compile_sql(manifest, project_path, sql, node_name="query"):
config, node, adapter = _get_operation_node(manifest, project_path, sql, node_name)
allow_introspection = str(os.environ.get("__DBT_ALLOW_INTROSPECTION", "1")).lower() in (
"true",
"1",
"on",
)
config, node, adapter = _get_operation_node(manifest, project_path, sql)
runner = SqlCompileRunner(config, adapter, node, 1, 1)
if allow_introspection:
runner = SqlCompileRunner(config, adapter, node, 1, 1)
else:
runner = SqlCompileRunnerNoIntrospection(config, adapter, node, 1, 1)
return runner.safe_run(manifest)
def execute_sql(manifest, project_path, sql):
def execute_sql(manifest, project_path, sql, node_name="query"):
from dbt.task.sql import SqlExecuteRunner
config, node, adapter = _get_operation_node(manifest, project_path, sql)
config, node, adapter = _get_operation_node(manifest, project_path, sql, node_name)
runner = SqlExecuteRunner(config, adapter, node, 1, 1)
# TODO: use same interface for runner
return runner.safe_run(manifest)
@@ -128,5 +197,4 @@ def deserialize_manifest(manifest_msgpack):
def serialize_manifest(manifest):
# TODO: what should this take as an arg?
return manifest.to_msgpack()

View File

@@ -142,7 +142,6 @@ def main(args=None):
exit_code = e.code
except BaseException as e:
traceback.print_exc()
fire_event(MainEncounteredError(e=str(e)))
fire_event(MainStackTrace(stack_trace=traceback.format_exc()))
exit_code = ExitCodes.UnhandledError.value

View File

@@ -69,6 +69,8 @@ class SchemaYamlRenderer(BaseRenderer):
elif self.key == "metrics":
if keypath[0] == "sql":
return False
elif self._is_norender_key(keypath[0:]):
return False
else: # models, seeds, snapshots, analyses
if self._is_norender_key(keypath[0:]):
return False

View File

@@ -3,6 +3,7 @@ import os
from pathlib import Path
import re
import shutil
import sys
from typing import Optional
import yaml
@@ -302,7 +303,7 @@ class InitTask(BaseTask):
available_adapters = list(_get_adapter_plugin_names())
if not len(available_adapters):
print("No adapters available. Go to https://docs.getdbt.com/docs/available-adapters")
exit(1)
sys.exit(1)
project_name = self.get_valid_project_name()
project_path = Path(project_name)
if project_path.exists():

View File

@@ -235,5 +235,5 @@ def _get_adapter_plugin_names() -> Iterator[str]:
yield plugin_name
__version__ = "1.2.0b1"
__version__ = "1.2.3"
installed = get_installed_version()

View File

@@ -25,7 +25,7 @@ with open(os.path.join(this_directory, "README.md")) as f:
package_name = "dbt-core"
package_version = "1.2.0b1"
package_version = "1.2.3"
description = """With dbt, data analysts and engineers can build analytics \
the way engineers build applications."""

View File

@@ -14,12 +14,12 @@ FROM --platform=$build_for python:3.10.5-slim-bullseye as base
# N.B. The refs updated automagically every release via bumpversion
# N.B. dbt-postgres is currently found in the core codebase so a value of dbt-core@<some_version> is correct
ARG dbt_core_ref=dbt-core@v1.2.0b1
ARG dbt_postgres_ref=dbt-core@v1.2.0b1
ARG dbt_redshift_ref=dbt-redshift@v1.0.0
ARG dbt_bigquery_ref=dbt-bigquery@v1.0.0
ARG dbt_snowflake_ref=dbt-snowflake@v1.0.0
ARG dbt_spark_ref=dbt-spark@v1.0.0
ARG dbt_core_ref=dbt-core@v1.2.3
ARG dbt_postgres_ref=dbt-core@v1.2.3
ARG dbt_redshift_ref=dbt-redshift@v1.2.2
ARG dbt_bigquery_ref=dbt-bigquery@v1.2.0
ARG dbt_snowflake_ref=dbt-snowflake@v1.2.0
ARG dbt_spark_ref=dbt-spark@v1.2.0
# special case args
ARG dbt_spark_version=all
ARG dbt_third_party

View File

@@ -1 +1 @@
version = "1.2.0b1"
version = "1.2.3"

View File

@@ -31,6 +31,7 @@ class PostgresCredentials(Credentials):
sslkey: Optional[str] = None
sslrootcert: Optional[str] = None
application_name: Optional[str] = "dbt"
retries: int = 1
_ALIASES = {"dbname": "database", "pass": "password"}
@@ -121,7 +122,7 @@ class PostgresConnectionManager(SQLConnectionManager):
if credentials.application_name:
kwargs["application_name"] = credentials.application_name
try:
def connect():
handle = psycopg2.connect(
dbname=credentials.database,
user=credentials.user,
@@ -131,23 +132,26 @@ class PostgresConnectionManager(SQLConnectionManager):
connect_timeout=credentials.connect_timeout,
**kwargs,
)
if credentials.role:
handle.cursor().execute("set role {}".format(credentials.role))
return handle
connection.handle = handle
connection.state = "open"
except psycopg2.Error as e:
logger.debug(
"Got an error when attempting to open a postgres " "connection: '{}'".format(e)
)
retryable_exceptions = [
# OperationalError is subclassed by all psycopg2 Connection Exceptions and it's raised
# by generic connection timeouts without an error code. This is a limitation of
# psycopg2 which doesn't provide subclasses for errors without a SQLSTATE error code.
# The limitation has been known for a while and there are no efforts to tackle it.
# See: https://github.com/psycopg/psycopg2/issues/682
psycopg2.errors.OperationalError,
]
connection.handle = None
connection.state = "fail"
raise dbt.exceptions.FailedToConnectException(str(e))
return connection
return cls.retry_connection(
connection,
connect=connect,
logger=logger,
retry_limit=credentials.retries,
retryable_exceptions=retryable_exceptions,
)
def cancel(self, connection):
connection_name = connection.name

View File

@@ -146,7 +146,7 @@
{% set suffix = suffix ~ dtstring %}
{% endif %}
{% set suffix_length = suffix|length %}
{% set relation_max_name_length = 63 %}
{% set relation_max_name_length = base_relation.relation_max_name_length() %}
{% if suffix_length > relation_max_name_length %}
{% do exceptions.raise_compiler_error('Relation suffix is too long (' ~ suffix_length ~ ' characters). Maximum length is ' ~ relation_max_name_length ~ ' characters.') %}
{% endif %}

View File

@@ -41,7 +41,7 @@ def _dbt_psycopg2_name():
package_name = "dbt-postgres"
package_version = "1.2.0b1"
package_version = "1.2.3"
description = """The postgres adapter plugin for dbt (data build tool)"""
this_directory = os.path.abspath(os.path.dirname(__file__))

View File

@@ -0,0 +1,494 @@
import unittest
from unittest import mock
import sys
import dbt.exceptions
import psycopg2
from dbt.contracts.connection import Connection
from dbt.adapters.base import BaseConnectionManager
from dbt.adapters.postgres import PostgresCredentials, PostgresConnectionManager
from dbt.events import AdapterLogger
class BaseConnectionManagerTest(unittest.TestCase):
def setUp(self):
self.postgres_credentials = PostgresCredentials(
host="localhost",
user="test-user",
port=1111,
password="test-password",
database="test-db",
schema="test-schema",
)
self.logger = AdapterLogger("test")
self.postgres_connection = Connection("postgres", None, self.postgres_credentials)
def test_retry_connection(self):
"""Test a dummy handle is set on a connection on the first attempt.
This test uses a Connection populated with test PostgresCredentials values, and
expects the Connection.handle attribute to be set to True and it's state to
"open", after calling retry_connection.
Moreover, the attribute should be set in the first attempt as no exception would
be raised for retrying. A mock connect function is used to simulate a real connection
passing on the first attempt.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
return True
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retryable_exceptions=[],
)
assert conn.state == "open"
assert conn.handle is True
assert attempts == 1
def test_retry_connection_fails_unhandled(self):
"""Test setting a handle fails upon raising a non-handled exception.
This test uses a Connection populated with test PostgresCredentials values, and
expects a ValueError to be raised by a mock connect function. As a
result:
* The Connection state should be "fail" and the handle None.
* The resulting attempt count should be 1 as we are not explicitly configured to handle a
ValueError.
* retry_connection should raise a FailedToConnectException with the Exception message.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
raise ValueError("Something went horribly wrong")
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_limit=1,
retry_timeout=lambda attempt: 0,
retryable_exceptions=(TypeError,),
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 1
def test_retry_connection_fails_handled(self):
"""Test setting a handle fails upon raising a handled exception.
This test uses a Connection populated with test PostgresCredentials values, and
expects a ValueError to be raised by a mock connect function.
As a result:
* The Connection state should be "fail" and the handle None.
* The resulting attempt count should be 2 as we are configured to handle a ValueError.
* retry_connection should raise a FailedToConnectException with the Exception message.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
raise ValueError("Something went horribly wrong")
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError,),
retry_limit=1,
)
assert conn.state == "fail"
assert conn.handle is None
def test_retry_connection_passes_handled(self):
"""Test setting a handle fails upon raising a handled exception.
This test uses a Connection populated with test PostgresCredentials values, and
expects a ValueError to be raised by a mock connect function only the first
time is called. Upon handling the exception once, connect should return.
As a result:
* The Connection state should be "open" and the handle True.
* The resulting attempt count should be 2 as we are configured to handle a ValueError.
"""
conn = self.postgres_connection
is_handled = False
attempts = 0
def connect():
nonlocal is_handled
nonlocal attempts
attempts += 1
if is_handled:
return True
is_handled = True
raise ValueError("Something went horribly wrong")
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError,),
retry_limit=1,
)
assert conn.state == "open"
assert conn.handle is True
assert is_handled is True
assert attempts == 2
def test_retry_connection_attempts(self):
"""Test setting a handle fails upon raising a handled exception multiple times.
This test uses a Connection populated with test PostgresCredentials values, and
expects a ValueError to be raised by a mock connect function. As a result:
* The Connection state should be "fail" and the handle None, as connect
never returns.
* The resulting attempt count should be 11 as we are configured to handle a ValueError.
* retry_connection should raise a FailedToConnectException with the Exception message.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
raise ValueError("Something went horribly wrong")
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError,),
retry_limit=10,
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 11
def test_retry_connection_fails_handling_all_exceptions(self):
"""Test setting a handle fails after exhausting all attempts.
This test uses a Connection populated with test PostgresCredentials values, and
expects a TypeError to be raised by a mock connect function. As a result:
* The Connection state should be "fail" and the handle None, as connect
never returns.
* The resulting attempt count should be 11 as we are configured to handle all Exceptions.
* retry_connection should raise a FailedToConnectException with the Exception message.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
raise TypeError("An unhandled thing went horribly wrong")
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "An unhandled thing went horribly wrong"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=[Exception],
retry_limit=15,
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 16
def test_retry_connection_passes_multiple_handled(self):
"""Test setting a handle passes upon handling multiple exceptions.
This test uses a Connection populated with test PostgresCredentials values, and
expects a mock connect to raise a ValueError in the first invocation and a
TypeError in the second invocation. As a result:
* The Connection state should be "open" and the handle True, as connect
returns after both exceptions have been handled.
* The resulting attempt count should be 3.
"""
conn = self.postgres_connection
is_value_err_handled = False
is_type_err_handled = False
attempts = 0
def connect():
nonlocal is_value_err_handled
nonlocal is_type_err_handled
nonlocal attempts
attempts += 1
if is_value_err_handled and is_type_err_handled:
return True
elif is_type_err_handled:
is_value_err_handled = True
raise ValueError("Something went horribly wrong")
else:
is_type_err_handled = True
raise TypeError("An unhandled thing went horribly wrong")
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError, TypeError),
retry_limit=2,
)
assert conn.state == "open"
assert conn.handle is True
assert is_type_err_handled is True
assert is_value_err_handled is True
assert attempts == 3
def test_retry_connection_passes_none_excluded(self):
"""Test setting a handle passes upon handling multiple exceptions.
This test uses a Connection populated with test PostgresCredentials values, and
expects a mock connect to raise a ValueError in the first invocation and a
TypeError in the second invocation. As a result:
* The Connection state should be "open" and the handle True, as connect
returns after both exceptions have been handled.
* The resulting attempt count should be 3.
"""
conn = self.postgres_connection
is_value_err_handled = False
is_type_err_handled = False
attempts = 0
def connect():
nonlocal is_value_err_handled
nonlocal is_type_err_handled
nonlocal attempts
attempts += 1
if is_value_err_handled and is_type_err_handled:
return True
elif is_type_err_handled:
is_value_err_handled = True
raise ValueError("Something went horribly wrong")
else:
is_type_err_handled = True
raise TypeError("An unhandled thing went horribly wrong")
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError, TypeError),
retry_limit=2,
)
assert conn.state == "open"
assert conn.handle is True
assert is_type_err_handled is True
assert is_value_err_handled is True
assert attempts == 3
def test_retry_connection_retry_limit(self):
"""Test retry_connection raises an exception with a negative retry limit."""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
return True
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "retry_limit cannot be negative"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError,),
retry_limit=-2,
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 0
def test_retry_connection_retry_timeout(self):
"""Test retry_connection raises an exception with a negative timeout."""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
return True
for retry_timeout in [-10, -2.5, lambda _: -100, lambda _: -10.1]:
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException,
"retry_timeout cannot be negative or return a negative time",
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=-10,
retryable_exceptions=(ValueError,),
retry_limit=2,
)
assert conn.state == "init"
assert conn.handle is None
assert attempts == 0
def test_retry_connection_exceeds_recursion_limit(self):
"""Test retry_connection raises an exception with retries that exceed recursion limit."""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
return True
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException,
"retry_limit cannot be negative",
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=2,
retryable_exceptions=(ValueError,),
retry_limit=sys.getrecursionlimit() + 1,
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 0
def test_retry_connection_with_exponential_backoff_timeout(self):
"""Test retry_connection with an exponential backoff timeout.
We assert the provided exponential backoff function gets passed the right attempt number
and produces the expected timeouts.
"""
conn = self.postgres_connection
attempts = 0
timeouts = []
def connect():
nonlocal attempts
attempts += 1
if attempts < 12:
raise ValueError("Keep trying!")
return True
def exp_backoff(n):
nonlocal timeouts
computed = 2**n
# We store the computed values to ensure they match the expected backoff...
timeouts.append((n, computed))
# but we return 0 as we don't want the test to go on forever.
return 0
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=exp_backoff,
retryable_exceptions=(ValueError,),
retry_limit=12,
)
assert conn.state == "open"
assert conn.handle is True
assert attempts == 12
assert timeouts == [(n, 2**n) for n in range(12)]
class PostgresConnectionManagerTest(unittest.TestCase):
def setUp(self):
self.credentials = PostgresCredentials(
host="localhost",
user="test-user",
port=1111,
password="test-password",
database="test-db",
schema="test-schema",
retries=2,
)
self.connection = Connection("postgres", None, self.credentials)
def test_open(self):
"""Test opening a Postgres Connection with failures in the first 3 attempts.
This test uses a Connection populated with test PostgresCredentials values, and
expects a mock connect to raise a psycopg2.errors.ConnectionFailuer
in the first 3 invocations, after which the mock should return True. As a result:
* The Connection state should be "open" and the handle True, as connect
returns in the 4th attempt.
* The resulting attempt count should be 4.
"""
conn = self.connection
attempt = 0
def connect(*args, **kwargs):
nonlocal attempt
attempt += 1
if attempt <= 2:
raise psycopg2.errors.ConnectionFailure("Connection has failed")
return True
with mock.patch("psycopg2.connect", wraps=connect) as mock_connect:
PostgresConnectionManager.open(conn)
assert mock_connect.call_count == 3
assert attempt == 3
assert conn.state == "open"
assert conn.handle is True

View File

@@ -190,9 +190,9 @@ REQUIRED_BASE_KEYS = frozenset(
"fromyaml",
"toyaml",
"set",
"try_set",
"set_strict",
"zip",
"try_zip",
"zip_strict",
"log",
"run_started_at",
"invocation_id",

63
test/unit/test_lib.py Normal file
View File

@@ -0,0 +1,63 @@
import os
import unittest
from unittest import mock
from dbt.contracts.results import RunningStatus
from dbt.lib import compile_sql
from dbt.adapters.postgres import Plugin
from test.unit.utils import clear_plugin, inject_adapter
class MockContext:
def __init__(self, node):
self.timing = []
self.node = mock.MagicMock()
self.node._event_status = {
"node_status": RunningStatus.Started
}
self.node.is_ephemeral_model = True
def noop_ephemeral_result(*args):
return None
class TestSqlCompileRunnerNoIntrospection(unittest.TestCase):
def setUp(self):
self.manifest = {'mock':'manifest'}
self.adapter = Plugin.adapter({})
self.adapter.connection_for = mock.MagicMock()
self.ephemeral_result = lambda: None
inject_adapter(self.adapter, Plugin)
def tearDown(self):
clear_plugin(Plugin)
@mock.patch('dbt.lib._get_operation_node')
@mock.patch('dbt.task.sql.GenericSqlRunner.compile')
@mock.patch('dbt.task.sql.GenericSqlRunner.ephemeral_result', noop_ephemeral_result)
@mock.patch('dbt.task.base.ExecutionContext', MockContext)
def test__compile_and_execute__with_connection(self, mock_compile, mock_get_node):
"""
By default, env var for allowing introspection is true, and calling this
method should defer to the parent method.
"""
mock_get_node.return_value = ({}, None, self.adapter)
compile_sql(self.manifest, 'some/path', None)
mock_compile.assert_called_once_with(self.manifest)
self.adapter.connection_for.assert_called_once()
@mock.patch('dbt.lib._get_operation_node')
@mock.patch('dbt.task.sql.GenericSqlRunner.compile')
@mock.patch('dbt.task.sql.GenericSqlRunner.ephemeral_result', noop_ephemeral_result)
@mock.patch('dbt.task.base.ExecutionContext', MockContext)
def test__compile_and_execute__without_connection(self, mock_compile, mock_get_node):
"""
Ensure that compile is called but does not attempt warehouse connection
"""
with mock.patch.dict(os.environ, {"__DBT_ALLOW_INTROSPECTION": "0"}):
mock_get_node.return_value = ({}, None, self.adapter)
compile_sql(self.manifest, 'some/path', None)
mock_compile.assert_called_once_with(self.manifest)
self.adapter.connection_for.assert_not_called()

Some files were not shown because too many files have changed in this diff Show More