Compare commits

...

5 Commits

Author SHA1 Message Date
github-actions[bot]
c7652f6a33 Bumping version to 1.2.0 (#5557)
* Bumping version to 1.2.0

* Whitespace fix

* Changlog update

* Docker file version update

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-07-26 08:28:05 -04:00
github-actions[bot]
a2a531722e Bumping version to 1.2.0rc2 (#5497)
* Bumping version to 1.2.0rc2

* Removing whitespace

* Update Dockerfile

* Changelog update

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-07-20 09:01:13 -04:00
leahwicz
7656ffb9e6 [CT-472] feat: Retrying method for acquiring connection handles (#5432) (#5494)
Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift).

Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
2022-07-19 21:36:04 -04:00
Jeremy Yeo
3f3792882c Fix: Rename try methods to strict (#5477)
* rename strict methods

* add changelog
2022-07-15 08:17:01 -05:00
github-actions[bot]
0db634d12f Bumping version to 1.2.0rc1 (#5458)
* Bumping version to 1.2.0rc1

* Remove whitespace

* Update Changelog

* Update Dockerfile

* Removing dependabot as contributor

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Removing dependabot as contributor

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-07-11 16:58:51 -04:00
85 changed files with 728 additions and 554 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.2.0b1
current_version = 1.2.0
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)

View File

@@ -1,4 +1,4 @@
## dbt-core 1.2.0-b1 - June 24, 2022
## dbt-core 1.2.0 - July 26, 2022
### Features
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
@@ -10,6 +10,11 @@
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- Allow customizing `target-path` and `log-path` through environment variables and CLI flags. ([#5399](https://github.com/dbt-labs/dbt-core/issues/5399), [#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- Move type_* macros from dbt-utils into dbt-core, with tests ([#5317](https://github.com/dbt-labs/dbt-core/issues/5317), [#5428](https://github.com/dbt-labs/dbt-core/pull/5428))
- Add support for ratio metrics ([#4884](https://github.com/dbt-labs/dbt-core/issues/4884), [#5027](https://github.com/dbt-labs/dbt-core/pull/5027))
- Allow users to define grants as a reasonable default in the dbt_project.yml or within each model sql or yml file combined. ([#5263](https://github.com/dbt-labs/dbt-core/issues/5263), [#5369](https://github.com/dbt-labs/dbt-core/pull/5369))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
### Fixes
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
@@ -29,7 +34,19 @@
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
- Add inheritance to materialization macro resolution ([#4646](https://github.com/dbt-labs/dbt-core/issues/4646), [#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
- Improve pluralizations for Documentation and SqlOperation NodeTypes ([#5352](https://github.com/dbt-labs/dbt-core/issues/5352), [#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- Properly use quotes for Snowflake snapshots when checking all columns ([#2975](https://github.com/dbt-labs/dbt-core/issues/2975), [#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- fixes handling of RESET color code with USE_COLORS=False ([#5288](https://github.com/dbt-labs/dbt-core/issues/5288), [#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
### Docs
- Fixed sample SQL Code for sources when no database is defined ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Add support for `file:` selector in DAG viz ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [Snyk] Upgrade prismjs from 1.27.0 to 1.28.0 ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Run build and tests in CI checks ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Improve metrics DAG viz and documentation page ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Upgrade cytoscape.js fork ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
### Under the Hood
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
@@ -46,24 +63,41 @@
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
- Added the suggested RegEx to check the SemVer string within a package dependency and improved invalid version error handling. ([#5201](https://github.com/dbt-labs/dbt-core/issues/5201), [#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
- Bump manifest version to v6 ([#5417](https://github.com/dbt-labs/dbt-core/issues/5417), [#5430](https://github.com/dbt-labs/dbt-core/pull/5430))
- Add tests for SQL grants ([#5437](https://github.com/dbt-labs/dbt-core/issues/5437), [#5447](https://github.com/dbt-labs/dbt-core/pull/5447))
### Dependencies
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Bump mypy from 0.942 to 0.961 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5337](https://github.com/dbt-labs/dbt-core/pull/5337))
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5388](https://github.com/dbt-labs/dbt-core/pull/5388))
- Bump black from 22.3.0 to 22.6.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5420](https://github.com/dbt-labs/dbt-core/pull/5420))
### Security
- Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Contributors
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- [@b-per](https://github.com/b-per) ([#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- [@darin-reify](https://github.com/darin-reify) ([#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@dependabot[bot]](https://github.com/dependabot[bot]) ([#5141](https://github.com/dbt-labs/dbt-core/pull/5141), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- [@drewbanin](https://github.com/drewbanin) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) ([#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- [@isidentical](https://github.com/isidentical) ([#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- [@tomasfarias](https://github.com/tomasfarias) ([#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@pdebelak](https://github.com/pdebelak) ([#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- [@pquadri](https://github.com/pquadri) ([#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- [@volkangurel](https://github.com/volkangurel) ([#5348](https://github.com/dbt-labs/dbt-core/pull/5348))

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump ubuntu from 20.04 to 22.04"
time: 2022-04-27T19:51:28.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5141"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bumping hologram version"
time: 2022-05-06T16:09:07.000000-05:00
custom:
Author: leahwicz
Issue: "5219"
PR: "5218"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker"
time: 2022-06-13T00:14:56.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5367"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add selector method when reading selector definitions
time: 2022-04-08T11:26:10.713088+10:00
custom:
Author: danieldiamond
Issue: "4821"
PR: "4827"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add set and zip function to contexts
time: 2022-04-23T23:17:56.851793+12:00
custom:
Author: jeremyyeo
Issue: "2345"
PR: "5107"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adds itertools to modules Jinja namespace
time: 2022-04-24T13:26:55.008246+01:00
custom:
Author: bd3dowling
Issue: "5130"
PR: "5140"

View File

@@ -1,7 +0,0 @@
kind: Features
body: allow target as an option in profile_template.yml
time: 2022-04-28T06:56:44.511519-04:00
custom:
Author: alexrosenfeld10
Issue: "5179"
PR: "5184"

View File

@@ -1,7 +0,0 @@
kind: Features
body: 'seed: Add new macro get_csv_sql'
time: 2022-05-03T14:29:34.847959075Z
custom:
Author: adamantike
Issue: "5206"
PR: "5207"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Grants as Node Configs
time: 2022-05-10T20:49:49.197999-04:00
custom:
Author: gshank
Issue: "5189"
PR: "5230"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adds file selectors and support for file selectors in the default method selector
time: 2022-05-12T21:57:48.289674-07:00
custom:
Author: jwills
Issue: "5240"
PR: "5241"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Move cross-db macros from dbt-utils into dbt-core global project
time: 2022-05-18T11:46:04.557104+02:00
custom:
Author: jtcohen6 dbeatty10
Issue: "4813"
PR: "5265"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Prettify duration message at the end of execution
time: 2022-06-11T16:39:31.725960083+01:00
custom:
Author: jared-rimmer
Issue: "5253"
PR: "5364"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Early return from dbt init if no available adapters
time: 2022-06-14T08:20:51.096872718+02:00
custom:
Author: ulisesojeda
Issue: "5365"
PR: "5366"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Adding new cols to check_cols in snapshots
time: 2022-03-17T21:09:16.977086+01:00
custom:
Author: GtheSheep
Issue: "3146"
PR: "4893"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Truncate relation names when appending a suffix that will result in len > 63
characters using make_temp_relation and make_backup_relation macros
time: 2022-03-22T17:37:53.320082-07:00
custom:
Author: epapineau
Issue: "2869"
PR: "4921"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Restore ability to utilize `updated_at` for check_cols snapshots
time: 2022-04-15T11:29:27.063462-06:00
custom:
Author: dbeatty10
Issue: "5076"
PR: "5077"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix retry logic to return values after initial try
time: 2022-04-22T13:12:27.239055-05:00
custom:
Author: emmyoop
Issue: "5023"
PR: "5137"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Use yaml renderer (with target context) for rendering selectors
time: 2022-04-22T13:56:45.147893-04:00
custom:
Author: gshank
Issue: "5131"
PR: "5136"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Scrub secret env vars from CommandError in exception stacktrace
time: 2022-04-25T20:39:24.365495+02:00
custom:
Author: jtcohen6
Issue: "5151"
PR: "5152"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Ensure the metric name does not contain spaces
time: 2022-04-26T20:21:04.360693-04:00
custom:
Author: gshank
Issue: "4572"
PR: "5173"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: When parsing 'all_sources' should be a list of unique dirs
time: 2022-04-27T10:26:48.648388-04:00
custom:
Author: gshank
Issue: "5120"
PR: "5176"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add warning if yaml contains duplicate keys
time: 2022-04-28T10:01:57.893956+12:00
custom:
Author: jeremyyeo
Issue: "5114"
PR: "5146"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Modifying the drop_test_schema to work better with Redshift issues around locked
tables and current transactions
time: 2022-04-29T16:07:42.750046-05:00
custom:
Author: Mcknight-42
Issue: "5200"
PR: "5198"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Fix column comparison in snapshot_check_all_get_existing_columns for check-strategy
snapshots with explicit check_cols defined
time: 2022-05-09T13:00:21.649028+02:00
custom:
Author: jtcohen6
Issue: "5222"
PR: "5223"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Changed how `--select state:modified` detects changes for macros nodes depend
on
time: 2022-05-09T13:13:12.889074-05:00
custom:
Author: stu-k
Issue: "5202"
PR: "5224"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix column comparison in snapshot_check_all_get_existing_columns to use adapter.get_columns_in_relation
time: 2022-05-11T12:32:38.313321+02:00
custom:
Author: jtcohen6
Issue: "5222"
PR: "5232"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove docs file from manifest when removing doc node
time: 2022-05-18T13:46:10.167143-04:00
custom:
Author: gshank
Issue: "4146"
PR: "5270"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove duplicate dbt script entry
time: 2022-06-01T08:13:14.067001+10:00
custom:
Author: groodt
Issue: "5314"
PR: "5304"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Change node ancestor/descendant algo, fixes issue where downstream models aren't
run when using networkx >= 2.8.1
time: 2022-06-01T13:59:08.886215-05:00
custom:
Author: iknox-fa
Issue: "5286"
PR: "5326"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fixing Windows color regression
time: 2022-06-01T19:42:34.263009-04:00
custom:
Author: leahwicz
Issue: "5191"
PR: "5327"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: 'Define compatibility for older manifest versions when using state: selection
methods'
time: 2022-06-08T08:09:14.321735+02:00
custom:
Author: jtcohen6
Issue: "5213"
PR: "5346"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove duplicate key checking introduced in 1.2.0a1
time: 2022-06-23T08:24:31.900647+12:00
custom:
Author: jeremyyeo
Issue: "5331"
PR: "5403"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Migrating 005_simple_seed to the new test framework.
time: 2022-04-09T04:05:39.20045-07:00
custom:
Author: versusfacit
Issue: "200"
PR: "5013"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert 029_docs_generate tests to new framework
time: 2022-04-13T18:30:14.706391-04:00
custom:
Author: gshank
Issue: "5035"
PR: "5058"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Move package deprecation check outside of package cache
time: 2022-04-14T13:22:06.157579-05:00
custom:
Author: emmyoop
Issue: "5068"
PR: "5069"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: removal of scaffold first attempt and create_adapter_plugin.py as they are deprecated new scaffold can be found https://github.com/dbt-labs/dbt-database-adapter-scaffold
time: 2022-04-20T12:00:25.171923-05:00
custom:
Author: McKnight-42
Issue: "4980"
PR: "5117"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Mypy -> 0.942 + fixed import logic to allow for full mypy coverage
time: 2022-04-27T11:21:27.499359-05:00
custom:
Author: iknox-fa
Issue: "4805"
PR: "5171"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Converted dbt list tests to pytest
time: 2022-04-27T14:06:28.882908-05:00
custom:
Author: stu-k
Issue: "5049"
PR: "5178"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: 'Fix: Call str and repr for UnsetProfileConfig without a RuntimeException'
time: 2022-05-03T19:52:12.793729384+02:00
custom:
Author: tomasfarias
Issue: "5081"
PR: "5209"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Improve tracking error logging message
time: 2022-05-04T01:00:31.60387036+02:00
custom:
Author: NicolasPA
Issue: "5197"
PR: "5211"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: 'Clean up materialization logic: more consistent relation names, loading from
cache'
time: 2022-05-09T09:26:28.551068+02:00
custom:
Author: jtcohen6
Issue: "2869"
PR: "4921"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Use the default Python version for local dev and test instead of requiring Python
3.8
time: 2022-05-18T09:51:44.603193-07:00
custom:
Author: jwills
Issue: "5257"
PR: "5269"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix test for context set function
time: 2022-05-18T14:55:22.554316-04:00
custom:
Author: gshank
Issue: "5266"
PR: "5272"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix pip upgrade step in CI for Windows
time: 2022-06-01T10:52:45.872931-04:00
custom:
Author: gshank
Issue: "5321"
PR: "5320"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix unit test test_graph_selection
time: 2022-06-01T11:26:48.725831-04:00
custom:
Author: gshank
Issue: "5323"
PR: "5324"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Update context readme + clean up context code"
time: 2022-06-06T23:03:53.022568+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5334"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: removed script meant for snowflake to snowflake
time: 2022-06-10T11:10:56.080236-05:00
custom:
Author: McKnight-42
Issue: "5361"
PR: "5362"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump mypy from 0.942 to 0.961"
time: 2022-06-07T00:09:25.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5337"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core"
time: 2022-06-17T00:18:49.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5388"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump black from 22.3.0 to 22.6.0"
time: 2022-06-29T00:08:31.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5420"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Fixed sample SQL Code for sources when no database is defined
time: 2022-07-06T14:20:29.7347-05:00
custom:
Author: b-per
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Add support for `file:` selector in DAG viz
time: 2022-07-06T14:21:42.544814-05:00
custom:
Author: drewbanin
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: '[Snyk] Upgrade prismjs from 1.27.0 to 1.28.0'
time: 2022-07-06T14:22:26.039137-05:00
custom:
Author: emmyoop
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Run build and tests in CI checks
time: 2022-07-06T14:23:00.842859-05:00
custom:
Author: drewbanin
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Improve metrics DAG viz and documentation page
time: 2022-07-06T14:24:10.120545-05:00
custom:
Author: drewbanin callum-mcdata
Issue: "5255"
PR: "5446"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Upgrade cytoscape.js fork
time: 2022-07-06T14:25:05.575755-05:00
custom:
Author: emmyoop
Issue: "5255"
PR: "5446"

View File

@@ -1,8 +0,0 @@
kind: Features
body: 'Allow customizing `target-path` and `log-path` through
environment variables and CLI flags.'
time: 2022-06-22T19:23:29.748577478+03:00
custom:
Author: isidentical
Issue: "5399"
PR: "5402"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Move type_* macros from dbt-utils into dbt-core, with tests
time: 2022-06-30T13:54:52.165139+02:00
custom:
Author: jtcohen6
Issue: "5317"
PR: "5428"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add support for ratio metrics
time: 2022-07-05T08:30:26.494837-05:00
custom:
Author: drewbanin callum-mcdata
Issue: "4884"
PR: "5027"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Allow users to define grants as a reasonable default in the dbt_project.yml
or within each model sql or yml file combined.
time: 2022-07-11T11:15:14.695386-05:00
custom:
Author: McKnight-42
Issue: "5263"
PR: "5369"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add inheritance to materialization macro resolution
time: 2022-06-08T10:42:51.839945-04:00
custom:
Author: volkangurel
Issue: "4646"
PR: "5348"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Improve pluralizations for Documentation and SqlOperation NodeTypes
time: 2022-06-09T11:52:46.578469-05:00
custom:
Author: pdebelak
Issue: "5352"
PR: "5356"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Properly use quotes for Snowflake snapshots when checking all columns
time: 2022-06-17T11:44:43.978834+02:00
custom:
Author: pquadri
Issue: "2975"
PR: "5389"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: fixes handling of RESET color code with USE_COLORS=False
time: 2022-06-17T16:00:27.038058-04:00
custom:
Author: darin-reify
Issue: "5288"
PR: "5394"

View File

@@ -1,7 +0,0 @@
kind: Security
body: Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README
time: 2022-06-06T23:03:53.022568+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5334"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Added the suggested RegEx to check the SemVer string within a package dependency
and improved invalid version error handling.
time: 2022-06-13T17:07:03.773668-05:00
custom:
Author: fivetran-joemarkiewicz
Issue: "5201"
PR: "5370"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: 'Add annotation to render_value method reimplemented in #5334'
time: 2022-06-16T12:05:16.474078+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5382"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Bump manifest version to v6
time: 2022-07-05T14:21:20.66768-04:00
custom:
Author: leahwicz
Issue: "5417"
PR: "5430"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add tests for SQL grants
time: 2022-07-06T21:50:01.498562-04:00
custom:
Author: gshank
Issue: "5437"
PR: "5447"

View File

@@ -6,7 +6,7 @@
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.2.0-b1 - June 24, 2022
## dbt-core 1.2.0 - July 26, 2022
### Features
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
@@ -18,6 +18,11 @@
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- Allow customizing `target-path` and `log-path` through environment variables and CLI flags. ([#5399](https://github.com/dbt-labs/dbt-core/issues/5399), [#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- Move type_* macros from dbt-utils into dbt-core, with tests ([#5317](https://github.com/dbt-labs/dbt-core/issues/5317), [#5428](https://github.com/dbt-labs/dbt-core/pull/5428))
- Add support for ratio metrics ([#4884](https://github.com/dbt-labs/dbt-core/issues/4884), [#5027](https://github.com/dbt-labs/dbt-core/pull/5027))
- Allow users to define grants as a reasonable default in the dbt_project.yml or within each model sql or yml file combined. ([#5263](https://github.com/dbt-labs/dbt-core/issues/5263), [#5369](https://github.com/dbt-labs/dbt-core/pull/5369))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
### Fixes
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
@@ -37,7 +42,19 @@
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
- Add inheritance to materialization macro resolution ([#4646](https://github.com/dbt-labs/dbt-core/issues/4646), [#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
- Improve pluralizations for Documentation and SqlOperation NodeTypes ([#5352](https://github.com/dbt-labs/dbt-core/issues/5352), [#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- Properly use quotes for Snowflake snapshots when checking all columns ([#2975](https://github.com/dbt-labs/dbt-core/issues/2975), [#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- fixes handling of RESET color code with USE_COLORS=False ([#5288](https://github.com/dbt-labs/dbt-core/issues/5288), [#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
### Docs
- Fixed sample SQL Code for sources when no database is defined ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Add support for `file:` selector in DAG viz ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [Snyk] Upgrade prismjs from 1.27.0 to 1.28.0 ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Run build and tests in CI checks ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Improve metrics DAG viz and documentation page ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- Upgrade cytoscape.js fork ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
### Under the Hood
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
@@ -54,27 +71,44 @@
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
- Added the suggested RegEx to check the SemVer string within a package dependency and improved invalid version error handling. ([#5201](https://github.com/dbt-labs/dbt-core/issues/5201), [#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
- Bump manifest version to v6 ([#5417](https://github.com/dbt-labs/dbt-core/issues/5417), [#5430](https://github.com/dbt-labs/dbt-core/pull/5430))
- Add tests for SQL grants ([#5437](https://github.com/dbt-labs/dbt-core/issues/5437), [#5447](https://github.com/dbt-labs/dbt-core/pull/5447))
### Dependencies
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Bump mypy from 0.942 to 0.961 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5337](https://github.com/dbt-labs/dbt-core/pull/5337))
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5388](https://github.com/dbt-labs/dbt-core/pull/5388))
- Bump black from 22.3.0 to 22.6.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5420](https://github.com/dbt-labs/dbt-core/pull/5420))
### Security
- Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Contributors
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
- [@b-per](https://github.com/b-per) ([#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
- [@darin-reify](https://github.com/darin-reify) ([#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@dependabot[bot]](https://github.com/dependabot[bot]) ([#5141](https://github.com/dbt-labs/dbt-core/pull/5141), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
- [@drewbanin](https://github.com/drewbanin) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
- [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) ([#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
- [@isidentical](https://github.com/isidentical) ([#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
- [@tomasfarias](https://github.com/tomasfarias) ([#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@pdebelak](https://github.com/pdebelak) ([#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
- [@pquadri](https://github.com/pquadri) ([#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
- [@volkangurel](https://github.com/volkangurel) ([#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
## Previous Releases

View File

@@ -1,10 +1,24 @@
import abc
import os
from time import sleep
import sys
# multiprocessing.RLock is a function returning this type
from multiprocessing.synchronize import RLock
from threading import get_ident
from typing import Dict, Tuple, Hashable, Optional, ContextManager, List
from typing import (
Any,
Dict,
Tuple,
Hashable,
Optional,
ContextManager,
List,
Type,
Union,
Iterable,
Callable,
)
import agate
@@ -21,6 +35,7 @@ from dbt.contracts.graph.manifest import Manifest
from dbt.adapters.base.query_headers import (
MacroQueryStringSetter,
)
from dbt.events import AdapterLogger
from dbt.events.functions import fire_event
from dbt.events.types import (
NewConnection,
@@ -34,6 +49,9 @@ from dbt.events.types import (
)
from dbt import flags
SleepTime = Union[int, float] # As taken by time.sleep.
AdapterHandle = Any # Adapter connection handle objects can be any class.
class BaseConnectionManager(metaclass=abc.ABCMeta):
"""Methods to implement:
@@ -159,6 +177,94 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
conn.name = conn_name
return conn
@classmethod
def retry_connection(
cls,
connection: Connection,
connect: Callable[[], AdapterHandle],
logger: AdapterLogger,
retryable_exceptions: Iterable[Type[Exception]],
retry_limit: int = 1,
retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1,
_attempts: int = 0,
) -> Connection:
"""Given a Connection, set its handle by calling connect.
The calls to connect will be retried up to retry_limit times to deal with transient
connection errors. By default, one retry will be attempted if retryable_exceptions is set.
:param Connection connection: An instance of a Connection that needs a handle to be set,
usually when attempting to open it.
:param connect: A callable that returns the appropiate connection handle for a
given adapter. This callable will be retried retry_limit times if a subclass of any
Exception in retryable_exceptions is raised by connect.
:type connect: Callable[[], AdapterHandle]
:param AdapterLogger logger: A logger to emit messages on retry attempts or errors. When
handling expected errors, we call debug, and call warning on unexpected errors or when
all retry attempts have been exhausted.
:param retryable_exceptions: An iterable of exception classes that if raised by
connect should trigger a retry.
:type retryable_exceptions: Iterable[Type[Exception]]
:param int retry_limit: How many times to retry the call to connect. If this limit
is exceeded before a successful call, a FailedToConnectException will be raised.
Must be non-negative.
:param retry_timeout: Time to wait between attempts to connect. Can also take a
Callable that takes the number of attempts so far, beginning at 0, and returns an int
or float to be passed to time.sleep.
:type retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1
:param int _attempts: Parameter used to keep track of the number of attempts in calling the
connect function across recursive calls. Passed as an argument to retry_timeout if it
is a Callable. This parameter should not be set by the initial caller.
:raises dbt.exceptions.FailedToConnectException: Upon exhausting all retry attempts without
successfully acquiring a handle.
:return: The given connection with its appropriate state and handle attributes set
depending on whether we successfully acquired a handle or not.
"""
timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout
if timeout < 0:
raise dbt.exceptions.FailedToConnectException(
"retry_timeout cannot be negative or return a negative time."
)
if retry_limit < 0 or retry_limit > sys.getrecursionlimit():
# This guard is not perfect others may add to the recursion limit (e.g. built-ins).
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException("retry_limit cannot be negative")
try:
connection.handle = connect()
connection.state = ConnectionState.OPEN
return connection
except tuple(retryable_exceptions) as e:
if retry_limit <= 0:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException(str(e))
logger.debug(
f"Got a retryable error when attempting to open a {cls.TYPE} connection.\n"
f"{retry_limit} attempts remaining. Retrying in {timeout} seconds.\n"
f"Error:\n{e}"
)
sleep(timeout)
return cls.retry_connection(
connection=connection,
connect=connect,
logger=logger,
retry_limit=retry_limit - 1,
retry_timeout=retry_timeout,
retryable_exceptions=retryable_exceptions,
_attempts=_attempts + 1,
)
except Exception as e:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectException(str(e))
@abc.abstractmethod
def cancel_open(self) -> Optional[List[str]]:
"""Cancel all open connections on the adapter. (passable)"""

View File

@@ -474,19 +474,17 @@ class BaseContext(metaclass=ContextMeta):
@contextmember
@staticmethod
def try_set(value: Iterable[Any]) -> Set[Any]:
"""The `try_set` context method can be used to convert any iterable
def set_strict(value: Iterable[Any]) -> Set[Any]:
"""The `set_strict` context method can be used to convert any iterable
to a sequence of iterable elements that are unique (a set). The
difference to the `set` context method is that the `try_set` method
difference to the `set` context method is that the `set_strict` method
will raise an exception on a TypeError.
:param value: The iterable
:param default: A default value to return if the `value` argument
is not an iterable
Usage:
{% set my_list = [1, 2, 2, 3] %}
{% set my_set = try_set(my_list) %}
{% set my_set = set_strict(my_list) %}
{% do log(my_set) %} {# {1, 2, 3} #}
"""
try:
@@ -497,7 +495,7 @@ class BaseContext(metaclass=ContextMeta):
@contextmember("zip")
@staticmethod
def _zip(*args: Iterable[Any], default: Any = None) -> Optional[Iterable[Any]]:
"""The `try_zip` context method can be used to used to return
"""The `zip` context method can be used to used to return
an iterator of tuples, where the i-th tuple contains the i-th
element from each of the argument iterables.
@@ -518,21 +516,19 @@ class BaseContext(metaclass=ContextMeta):
@contextmember
@staticmethod
def try_zip(*args: Iterable[Any]) -> Iterable[Any]:
"""The `try_zip` context method can be used to used to return
def zip_strict(*args: Iterable[Any]) -> Iterable[Any]:
"""The `zip_strict` context method can be used to used to return
an iterator of tuples, where the i-th tuple contains the i-th
element from each of the argument iterables. The difference to the
`zip` context method is that the `try_zip` method will raise an
`zip` context method is that the `zip_strict` method will raise an
exception on a TypeError.
:param *args: Any number of iterables
:param default: A default value to return if `*args` is not
iterable
Usage:
{% set my_list_a = [1, 2] %}
{% set my_list_b = ['alice', 'bob'] %}
{% set my_zip = try_zip(my_list_a, my_list_b) | list %}
{% set my_zip = zip_strict(my_list_a, my_list_b) | list %}
{% do log(my_set) %} {# [(1, 'alice'), (2, 'bob')] #}
"""
try:

View File

@@ -235,5 +235,5 @@ def _get_adapter_plugin_names() -> Iterator[str]:
yield plugin_name
__version__ = "1.2.0b1"
__version__ = "1.2.0"
installed = get_installed_version()

View File

@@ -25,7 +25,7 @@ with open(os.path.join(this_directory, "README.md")) as f:
package_name = "dbt-core"
package_version = "1.2.0b1"
package_version = "1.2.0"
description = """With dbt, data analysts and engineers can build analytics \
the way engineers build applications."""

View File

@@ -14,12 +14,12 @@ FROM --platform=$build_for python:3.10.5-slim-bullseye as base
# N.B. The refs updated automagically every release via bumpversion
# N.B. dbt-postgres is currently found in the core codebase so a value of dbt-core@<some_version> is correct
ARG dbt_core_ref=dbt-core@v1.2.0b1
ARG dbt_postgres_ref=dbt-core@v1.2.0b1
ARG dbt_redshift_ref=dbt-redshift@v1.0.0
ARG dbt_bigquery_ref=dbt-bigquery@v1.0.0
ARG dbt_snowflake_ref=dbt-snowflake@v1.0.0
ARG dbt_spark_ref=dbt-spark@v1.0.0
ARG dbt_core_ref=dbt-core@v1.2.0
ARG dbt_postgres_ref=dbt-core@v1.2.0
ARG dbt_redshift_ref=dbt-redshift@v1.2.0
ARG dbt_bigquery_ref=dbt-bigquery@v1.2.0
ARG dbt_snowflake_ref=dbt-snowflake@v1.2.0
ARG dbt_spark_ref=dbt-spark@v1.2.0
# special case args
ARG dbt_spark_version=all
ARG dbt_third_party

View File

@@ -1 +1 @@
version = "1.2.0b1"
version = "1.2.0"

View File

@@ -31,6 +31,7 @@ class PostgresCredentials(Credentials):
sslkey: Optional[str] = None
sslrootcert: Optional[str] = None
application_name: Optional[str] = "dbt"
retries: int = 1
_ALIASES = {"dbname": "database", "pass": "password"}
@@ -121,7 +122,7 @@ class PostgresConnectionManager(SQLConnectionManager):
if credentials.application_name:
kwargs["application_name"] = credentials.application_name
try:
def connect():
handle = psycopg2.connect(
dbname=credentials.database,
user=credentials.user,
@@ -131,23 +132,26 @@ class PostgresConnectionManager(SQLConnectionManager):
connect_timeout=credentials.connect_timeout,
**kwargs,
)
if credentials.role:
handle.cursor().execute("set role {}".format(credentials.role))
return handle
connection.handle = handle
connection.state = "open"
except psycopg2.Error as e:
logger.debug(
"Got an error when attempting to open a postgres " "connection: '{}'".format(e)
)
retryable_exceptions = [
# OperationalError is subclassed by all psycopg2 Connection Exceptions and it's raised
# by generic connection timeouts without an error code. This is a limitation of
# psycopg2 which doesn't provide subclasses for errors without a SQLSTATE error code.
# The limitation has been known for a while and there are no efforts to tackle it.
# See: https://github.com/psycopg/psycopg2/issues/682
psycopg2.errors.OperationalError,
]
connection.handle = None
connection.state = "fail"
raise dbt.exceptions.FailedToConnectException(str(e))
return connection
return cls.retry_connection(
connection,
connect=connect,
logger=logger,
retry_limit=credentials.retries,
retryable_exceptions=retryable_exceptions,
)
def cancel(self, connection):
connection_name = connection.name

View File

@@ -41,7 +41,7 @@ def _dbt_psycopg2_name():
package_name = "dbt-postgres"
package_version = "1.2.0b1"
package_version = "1.2.0"
description = """The postgres adapter plugin for dbt (data build tool)"""
this_directory = os.path.abspath(os.path.dirname(__file__))

View File

@@ -0,0 +1,494 @@
import unittest
from unittest import mock
import sys
import dbt.exceptions
import psycopg2
from dbt.contracts.connection import Connection
from dbt.adapters.base import BaseConnectionManager
from dbt.adapters.postgres import PostgresCredentials, PostgresConnectionManager
from dbt.events import AdapterLogger
class BaseConnectionManagerTest(unittest.TestCase):
def setUp(self):
self.postgres_credentials = PostgresCredentials(
host="localhost",
user="test-user",
port=1111,
password="test-password",
database="test-db",
schema="test-schema",
)
self.logger = AdapterLogger("test")
self.postgres_connection = Connection("postgres", None, self.postgres_credentials)
def test_retry_connection(self):
"""Test a dummy handle is set on a connection on the first attempt.
This test uses a Connection populated with test PostgresCredentials values, and
expects the Connection.handle attribute to be set to True and it's state to
"open", after calling retry_connection.
Moreover, the attribute should be set in the first attempt as no exception would
be raised for retrying. A mock connect function is used to simulate a real connection
passing on the first attempt.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
return True
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retryable_exceptions=[],
)
assert conn.state == "open"
assert conn.handle is True
assert attempts == 1
def test_retry_connection_fails_unhandled(self):
"""Test setting a handle fails upon raising a non-handled exception.
This test uses a Connection populated with test PostgresCredentials values, and
expects a ValueError to be raised by a mock connect function. As a
result:
* The Connection state should be "fail" and the handle None.
* The resulting attempt count should be 1 as we are not explicitly configured to handle a
ValueError.
* retry_connection should raise a FailedToConnectException with the Exception message.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
raise ValueError("Something went horribly wrong")
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_limit=1,
retry_timeout=lambda attempt: 0,
retryable_exceptions=(TypeError,),
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 1
def test_retry_connection_fails_handled(self):
"""Test setting a handle fails upon raising a handled exception.
This test uses a Connection populated with test PostgresCredentials values, and
expects a ValueError to be raised by a mock connect function.
As a result:
* The Connection state should be "fail" and the handle None.
* The resulting attempt count should be 2 as we are configured to handle a ValueError.
* retry_connection should raise a FailedToConnectException with the Exception message.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
raise ValueError("Something went horribly wrong")
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError,),
retry_limit=1,
)
assert conn.state == "fail"
assert conn.handle is None
def test_retry_connection_passes_handled(self):
"""Test setting a handle fails upon raising a handled exception.
This test uses a Connection populated with test PostgresCredentials values, and
expects a ValueError to be raised by a mock connect function only the first
time is called. Upon handling the exception once, connect should return.
As a result:
* The Connection state should be "open" and the handle True.
* The resulting attempt count should be 2 as we are configured to handle a ValueError.
"""
conn = self.postgres_connection
is_handled = False
attempts = 0
def connect():
nonlocal is_handled
nonlocal attempts
attempts += 1
if is_handled:
return True
is_handled = True
raise ValueError("Something went horribly wrong")
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError,),
retry_limit=1,
)
assert conn.state == "open"
assert conn.handle is True
assert is_handled is True
assert attempts == 2
def test_retry_connection_attempts(self):
"""Test setting a handle fails upon raising a handled exception multiple times.
This test uses a Connection populated with test PostgresCredentials values, and
expects a ValueError to be raised by a mock connect function. As a result:
* The Connection state should be "fail" and the handle None, as connect
never returns.
* The resulting attempt count should be 11 as we are configured to handle a ValueError.
* retry_connection should raise a FailedToConnectException with the Exception message.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
raise ValueError("Something went horribly wrong")
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError,),
retry_limit=10,
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 11
def test_retry_connection_fails_handling_all_exceptions(self):
"""Test setting a handle fails after exhausting all attempts.
This test uses a Connection populated with test PostgresCredentials values, and
expects a TypeError to be raised by a mock connect function. As a result:
* The Connection state should be "fail" and the handle None, as connect
never returns.
* The resulting attempt count should be 11 as we are configured to handle all Exceptions.
* retry_connection should raise a FailedToConnectException with the Exception message.
"""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
raise TypeError("An unhandled thing went horribly wrong")
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "An unhandled thing went horribly wrong"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=[Exception],
retry_limit=15,
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 16
def test_retry_connection_passes_multiple_handled(self):
"""Test setting a handle passes upon handling multiple exceptions.
This test uses a Connection populated with test PostgresCredentials values, and
expects a mock connect to raise a ValueError in the first invocation and a
TypeError in the second invocation. As a result:
* The Connection state should be "open" and the handle True, as connect
returns after both exceptions have been handled.
* The resulting attempt count should be 3.
"""
conn = self.postgres_connection
is_value_err_handled = False
is_type_err_handled = False
attempts = 0
def connect():
nonlocal is_value_err_handled
nonlocal is_type_err_handled
nonlocal attempts
attempts += 1
if is_value_err_handled and is_type_err_handled:
return True
elif is_type_err_handled:
is_value_err_handled = True
raise ValueError("Something went horribly wrong")
else:
is_type_err_handled = True
raise TypeError("An unhandled thing went horribly wrong")
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError, TypeError),
retry_limit=2,
)
assert conn.state == "open"
assert conn.handle is True
assert is_type_err_handled is True
assert is_value_err_handled is True
assert attempts == 3
def test_retry_connection_passes_none_excluded(self):
"""Test setting a handle passes upon handling multiple exceptions.
This test uses a Connection populated with test PostgresCredentials values, and
expects a mock connect to raise a ValueError in the first invocation and a
TypeError in the second invocation. As a result:
* The Connection state should be "open" and the handle True, as connect
returns after both exceptions have been handled.
* The resulting attempt count should be 3.
"""
conn = self.postgres_connection
is_value_err_handled = False
is_type_err_handled = False
attempts = 0
def connect():
nonlocal is_value_err_handled
nonlocal is_type_err_handled
nonlocal attempts
attempts += 1
if is_value_err_handled and is_type_err_handled:
return True
elif is_type_err_handled:
is_value_err_handled = True
raise ValueError("Something went horribly wrong")
else:
is_type_err_handled = True
raise TypeError("An unhandled thing went horribly wrong")
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError, TypeError),
retry_limit=2,
)
assert conn.state == "open"
assert conn.handle is True
assert is_type_err_handled is True
assert is_value_err_handled is True
assert attempts == 3
def test_retry_connection_retry_limit(self):
"""Test retry_connection raises an exception with a negative retry limit."""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
return True
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException, "retry_limit cannot be negative"
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=0,
retryable_exceptions=(ValueError,),
retry_limit=-2,
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 0
def test_retry_connection_retry_timeout(self):
"""Test retry_connection raises an exception with a negative timeout."""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
return True
for retry_timeout in [-10, -2.5, lambda _: -100, lambda _: -10.1]:
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException,
"retry_timeout cannot be negative or return a negative time",
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=-10,
retryable_exceptions=(ValueError,),
retry_limit=2,
)
assert conn.state == "init"
assert conn.handle is None
assert attempts == 0
def test_retry_connection_exceeds_recursion_limit(self):
"""Test retry_connection raises an exception with retries that exceed recursion limit."""
conn = self.postgres_connection
attempts = 0
def connect():
nonlocal attempts
attempts += 1
return True
with self.assertRaisesRegex(
dbt.exceptions.FailedToConnectException,
"retry_limit cannot be negative",
):
BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=2,
retryable_exceptions=(ValueError,),
retry_limit=sys.getrecursionlimit() + 1,
)
assert conn.state == "fail"
assert conn.handle is None
assert attempts == 0
def test_retry_connection_with_exponential_backoff_timeout(self):
"""Test retry_connection with an exponential backoff timeout.
We assert the provided exponential backoff function gets passed the right attempt number
and produces the expected timeouts.
"""
conn = self.postgres_connection
attempts = 0
timeouts = []
def connect():
nonlocal attempts
attempts += 1
if attempts < 12:
raise ValueError("Keep trying!")
return True
def exp_backoff(n):
nonlocal timeouts
computed = 2**n
# We store the computed values to ensure they match the expected backoff...
timeouts.append((n, computed))
# but we return 0 as we don't want the test to go on forever.
return 0
conn = BaseConnectionManager.retry_connection(
conn,
connect,
self.logger,
retry_timeout=exp_backoff,
retryable_exceptions=(ValueError,),
retry_limit=12,
)
assert conn.state == "open"
assert conn.handle is True
assert attempts == 12
assert timeouts == [(n, 2**n) for n in range(12)]
class PostgresConnectionManagerTest(unittest.TestCase):
def setUp(self):
self.credentials = PostgresCredentials(
host="localhost",
user="test-user",
port=1111,
password="test-password",
database="test-db",
schema="test-schema",
retries=2,
)
self.connection = Connection("postgres", None, self.credentials)
def test_open(self):
"""Test opening a Postgres Connection with failures in the first 3 attempts.
This test uses a Connection populated with test PostgresCredentials values, and
expects a mock connect to raise a psycopg2.errors.ConnectionFailuer
in the first 3 invocations, after which the mock should return True. As a result:
* The Connection state should be "open" and the handle True, as connect
returns in the 4th attempt.
* The resulting attempt count should be 4.
"""
conn = self.connection
attempt = 0
def connect(*args, **kwargs):
nonlocal attempt
attempt += 1
if attempt <= 2:
raise psycopg2.errors.ConnectionFailure("Connection has failed")
return True
with mock.patch("psycopg2.connect", wraps=connect) as mock_connect:
PostgresConnectionManager.open(conn)
assert mock_connect.call_count == 3
assert attempt == 3
assert conn.state == "open"
assert conn.handle is True

View File

@@ -190,9 +190,9 @@ REQUIRED_BASE_KEYS = frozenset(
"fromyaml",
"toyaml",
"set",
"try_set",
"set_strict",
"zip",
"try_zip",
"zip_strict",
"log",
"run_started_at",
"invocation_id",

View File

@@ -1 +1 @@
version = "1.2.0b1"
version = "1.2.0"

View File

@@ -20,7 +20,7 @@ except ImportError:
package_name = "dbt-tests-adapter"
package_version = "1.2.0b1"
package_version = "1.2.0"
description = """The dbt adapter tests for adapter plugins"""
this_directory = os.path.abspath(os.path.dirname(__file__))

View File

@@ -7,8 +7,8 @@ macros__validate_set_sql = """
{% macro validate_set() %}
{% set set_result = set([1, 2, 2, 3, 'foo', False]) %}
{{ log("set_result: " ~ set_result) }}
{% set try_set_result = try_set([1, 2, 2, 3, 'foo', False]) %}
{{ log("try_set_result: " ~ try_set_result) }}
{% set set_strict_result = set_strict([1, 2, 2, 3, 'foo', False]) %}
{{ log("set_strict_result: " ~ set_strict_result) }}
{% endmacro %}
"""
@@ -18,17 +18,17 @@ macros__validate_zip_sql = """
{% set list_b = ['foo', 'bar'] %}
{% set zip_result = zip(list_a, list_b) | list %}
{{ log("zip_result: " ~ zip_result) }}
{% set try_zip_result = try_zip(list_a, list_b) | list %}
{{ log("try_zip_result: " ~ try_zip_result) }}
{% set zip_strict_result = zip_strict(list_a, list_b) | list %}
{{ log("zip_strict_result: " ~ zip_strict_result) }}
{% endmacro %}
"""
models__set_exception_sql = """
{% set try_set_result = try_set(1) %}
{% set set_strict_result = set_strict(1) %}
"""
models__zip_exception_sql = """
{% set try_set_result = try_zip(1) %}
{% set zip_strict_result = zip_strict(1) %}
"""
@@ -46,18 +46,18 @@ class TestContextBuiltins:
# The order of the set isn't guaranteed so we can't check for the actual set in the logs
assert "set_result: " in log_output
assert "False" in log_output
assert "try_set_result: " in log_output
assert "set_strict_result: " in log_output
def test_builtin_zip_function(self, project):
_, log_output = run_dbt_and_capture(["--debug", "run-operation", "validate_zip"])
expected_zip = [(1, "foo"), (2, "bar")]
assert f"zip_result: {expected_zip}" in log_output
assert f"try_zip_result: {expected_zip}" in log_output
assert f"zip_strict_result: {expected_zip}" in log_output
class TestContextBuiltinExceptions:
# Assert compilation errors are raised with try_ equivalents
# Assert compilation errors are raised with _strict equivalents
def test_builtin_function_exception(self, project):
write_file(models__set_exception_sql, project.project_root, "models", "raise.sql")
with pytest.raises(CompilationException):