forked from repo-mirrors/dbt-core
Compare commits
13 Commits
main
...
performanc
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
07415ca3e7 | ||
|
|
1d6eb7dcd5 | ||
|
|
7955fe7c3b | ||
|
|
f82cddbefe | ||
|
|
39901c5155 | ||
|
|
6c21d10469 | ||
|
|
9a10de926e | ||
|
|
9c32441d05 | ||
|
|
c7652f6a33 | ||
|
|
a2a531722e | ||
|
|
7656ffb9e6 | ||
|
|
3f3792882c | ||
|
|
0db634d12f |
@@ -1,5 +1,5 @@
|
||||
[bumpversion]
|
||||
current_version = 1.2.0b1
|
||||
current_version = 1.2.1
|
||||
parse = (?P<major>\d+)
|
||||
\.(?P<minor>\d+)
|
||||
\.(?P<patch>\d+)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
## dbt-core 1.2.0-b1 - June 24, 2022
|
||||
## dbt-core 1.2.0 - July 26, 2022
|
||||
### Features
|
||||
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
|
||||
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
|
||||
@@ -10,6 +10,11 @@
|
||||
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
|
||||
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
|
||||
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
|
||||
- Allow customizing `target-path` and `log-path` through environment variables and CLI flags. ([#5399](https://github.com/dbt-labs/dbt-core/issues/5399), [#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
|
||||
- Move type_* macros from dbt-utils into dbt-core, with tests ([#5317](https://github.com/dbt-labs/dbt-core/issues/5317), [#5428](https://github.com/dbt-labs/dbt-core/pull/5428))
|
||||
- Add support for ratio metrics ([#4884](https://github.com/dbt-labs/dbt-core/issues/4884), [#5027](https://github.com/dbt-labs/dbt-core/pull/5027))
|
||||
- Allow users to define grants as a reasonable default in the dbt_project.yml or within each model sql or yml file combined. ([#5263](https://github.com/dbt-labs/dbt-core/issues/5263), [#5369](https://github.com/dbt-labs/dbt-core/pull/5369))
|
||||
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
|
||||
### Fixes
|
||||
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
|
||||
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
|
||||
@@ -29,7 +34,19 @@
|
||||
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
|
||||
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
|
||||
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
|
||||
- Add inheritance to materialization macro resolution ([#4646](https://github.com/dbt-labs/dbt-core/issues/4646), [#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
|
||||
- Improve pluralizations for Documentation and SqlOperation NodeTypes ([#5352](https://github.com/dbt-labs/dbt-core/issues/5352), [#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
|
||||
- Properly use quotes for Snowflake snapshots when checking all columns ([#2975](https://github.com/dbt-labs/dbt-core/issues/2975), [#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
|
||||
- fixes handling of RESET color code with USE_COLORS=False ([#5288](https://github.com/dbt-labs/dbt-core/issues/5288), [#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
|
||||
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
|
||||
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
|
||||
### Docs
|
||||
- Fixed sample SQL Code for sources when no database is defined ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- Add support for `file:` selector in DAG viz ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- [Snyk] Upgrade prismjs from 1.27.0 to 1.28.0 ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- Run build and tests in CI checks ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- Improve metrics DAG viz and documentation page ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- Upgrade cytoscape.js fork ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
### Under the Hood
|
||||
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
|
||||
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
|
||||
@@ -46,24 +63,41 @@
|
||||
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
|
||||
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
|
||||
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
|
||||
- Added the suggested RegEx to check the SemVer string within a package dependency and improved invalid version error handling. ([#5201](https://github.com/dbt-labs/dbt-core/issues/5201), [#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
|
||||
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
|
||||
- Bump manifest version to v6 ([#5417](https://github.com/dbt-labs/dbt-core/issues/5417), [#5430](https://github.com/dbt-labs/dbt-core/pull/5430))
|
||||
- Add tests for SQL grants ([#5437](https://github.com/dbt-labs/dbt-core/issues/5437), [#5447](https://github.com/dbt-labs/dbt-core/pull/5447))
|
||||
### Dependencies
|
||||
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
|
||||
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
|
||||
- Bump mypy from 0.942 to 0.961 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5337](https://github.com/dbt-labs/dbt-core/pull/5337))
|
||||
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
|
||||
- Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5388](https://github.com/dbt-labs/dbt-core/pull/5388))
|
||||
- Bump black from 22.3.0 to 22.6.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5420](https://github.com/dbt-labs/dbt-core/pull/5420))
|
||||
### Security
|
||||
- Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
|
||||
|
||||
### Contributors
|
||||
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
|
||||
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
|
||||
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
|
||||
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
|
||||
- [@b-per](https://github.com/b-per) ([#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
|
||||
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
|
||||
- [@darin-reify](https://github.com/darin-reify) ([#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
|
||||
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
|
||||
- [@dependabot[bot]](https://github.com/dependabot[bot]) ([#5141](https://github.com/dbt-labs/dbt-core/pull/5141), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
|
||||
- [@drewbanin](https://github.com/drewbanin) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
|
||||
- [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) ([#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
|
||||
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
|
||||
- [@isidentical](https://github.com/isidentical) ([#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
|
||||
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
|
||||
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
|
||||
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
|
||||
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
|
||||
- [@tomasfarias](https://github.com/tomasfarias) ([#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
|
||||
- [@pdebelak](https://github.com/pdebelak) ([#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
|
||||
- [@pquadri](https://github.com/pquadri) ([#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
|
||||
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
|
||||
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
|
||||
- [@volkangurel](https://github.com/volkangurel) ([#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Dependencies
|
||||
body: "Bump ubuntu from 20.04 to 22.04"
|
||||
time: 2022-04-27T19:51:28.000000-05:00
|
||||
custom:
|
||||
Author: dependabot[bot]
|
||||
Issue: "4904"
|
||||
PR: "5141"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Dependencies
|
||||
body: "Bumping hologram version"
|
||||
time: 2022-05-06T16:09:07.000000-05:00
|
||||
custom:
|
||||
Author: leahwicz
|
||||
Issue: "5219"
|
||||
PR: "5218"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Dependencies
|
||||
body: "Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker"
|
||||
time: 2022-06-13T00:14:56.000000-05:00
|
||||
custom:
|
||||
Author: dependabot[bot]
|
||||
Issue: "4904"
|
||||
PR: "5367"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Add selector method when reading selector definitions
|
||||
time: 2022-04-08T11:26:10.713088+10:00
|
||||
custom:
|
||||
Author: danieldiamond
|
||||
Issue: "4821"
|
||||
PR: "4827"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Add set and zip function to contexts
|
||||
time: 2022-04-23T23:17:56.851793+12:00
|
||||
custom:
|
||||
Author: jeremyyeo
|
||||
Issue: "2345"
|
||||
PR: "5107"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Adds itertools to modules Jinja namespace
|
||||
time: 2022-04-24T13:26:55.008246+01:00
|
||||
custom:
|
||||
Author: bd3dowling
|
||||
Issue: "5130"
|
||||
PR: "5140"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: allow target as an option in profile_template.yml
|
||||
time: 2022-04-28T06:56:44.511519-04:00
|
||||
custom:
|
||||
Author: alexrosenfeld10
|
||||
Issue: "5179"
|
||||
PR: "5184"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: 'seed: Add new macro get_csv_sql'
|
||||
time: 2022-05-03T14:29:34.847959075Z
|
||||
custom:
|
||||
Author: adamantike
|
||||
Issue: "5206"
|
||||
PR: "5207"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Grants as Node Configs
|
||||
time: 2022-05-10T20:49:49.197999-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "5189"
|
||||
PR: "5230"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Adds file selectors and support for file selectors in the default method selector
|
||||
time: 2022-05-12T21:57:48.289674-07:00
|
||||
custom:
|
||||
Author: jwills
|
||||
Issue: "5240"
|
||||
PR: "5241"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Move cross-db macros from dbt-utils into dbt-core global project
|
||||
time: 2022-05-18T11:46:04.557104+02:00
|
||||
custom:
|
||||
Author: jtcohen6 dbeatty10
|
||||
Issue: "4813"
|
||||
PR: "5265"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Prettify duration message at the end of execution
|
||||
time: 2022-06-11T16:39:31.725960083+01:00
|
||||
custom:
|
||||
Author: jared-rimmer
|
||||
Issue: "5253"
|
||||
PR: "5364"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Early return from dbt init if no available adapters
|
||||
time: 2022-06-14T08:20:51.096872718+02:00
|
||||
custom:
|
||||
Author: ulisesojeda
|
||||
Issue: "5365"
|
||||
PR: "5366"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Adding new cols to check_cols in snapshots
|
||||
time: 2022-03-17T21:09:16.977086+01:00
|
||||
custom:
|
||||
Author: GtheSheep
|
||||
Issue: "3146"
|
||||
PR: "4893"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Truncate relation names when appending a suffix that will result in len > 63
|
||||
characters using make_temp_relation and make_backup_relation macros
|
||||
time: 2022-03-22T17:37:53.320082-07:00
|
||||
custom:
|
||||
Author: epapineau
|
||||
Issue: "2869"
|
||||
PR: "4921"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Restore ability to utilize `updated_at` for check_cols snapshots
|
||||
time: 2022-04-15T11:29:27.063462-06:00
|
||||
custom:
|
||||
Author: dbeatty10
|
||||
Issue: "5076"
|
||||
PR: "5077"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Fix retry logic to return values after initial try
|
||||
time: 2022-04-22T13:12:27.239055-05:00
|
||||
custom:
|
||||
Author: emmyoop
|
||||
Issue: "5023"
|
||||
PR: "5137"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Use yaml renderer (with target context) for rendering selectors
|
||||
time: 2022-04-22T13:56:45.147893-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "5131"
|
||||
PR: "5136"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Scrub secret env vars from CommandError in exception stacktrace
|
||||
time: 2022-04-25T20:39:24.365495+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "5151"
|
||||
PR: "5152"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Ensure the metric name does not contain spaces
|
||||
time: 2022-04-26T20:21:04.360693-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "4572"
|
||||
PR: "5173"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: When parsing 'all_sources' should be a list of unique dirs
|
||||
time: 2022-04-27T10:26:48.648388-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "5120"
|
||||
PR: "5176"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Add warning if yaml contains duplicate keys
|
||||
time: 2022-04-28T10:01:57.893956+12:00
|
||||
custom:
|
||||
Author: jeremyyeo
|
||||
Issue: "5114"
|
||||
PR: "5146"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Modifying the drop_test_schema to work better with Redshift issues around locked
|
||||
tables and current transactions
|
||||
time: 2022-04-29T16:07:42.750046-05:00
|
||||
custom:
|
||||
Author: Mcknight-42
|
||||
Issue: "5200"
|
||||
PR: "5198"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Fix column comparison in snapshot_check_all_get_existing_columns for check-strategy
|
||||
snapshots with explicit check_cols defined
|
||||
time: 2022-05-09T13:00:21.649028+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "5222"
|
||||
PR: "5223"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Changed how `--select state:modified` detects changes for macros nodes depend
|
||||
on
|
||||
time: 2022-05-09T13:13:12.889074-05:00
|
||||
custom:
|
||||
Author: stu-k
|
||||
Issue: "5202"
|
||||
PR: "5224"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Fix column comparison in snapshot_check_all_get_existing_columns to use adapter.get_columns_in_relation
|
||||
time: 2022-05-11T12:32:38.313321+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "5222"
|
||||
PR: "5232"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Remove docs file from manifest when removing doc node
|
||||
time: 2022-05-18T13:46:10.167143-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "4146"
|
||||
PR: "5270"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Remove duplicate dbt script entry
|
||||
time: 2022-06-01T08:13:14.067001+10:00
|
||||
custom:
|
||||
Author: groodt
|
||||
Issue: "5314"
|
||||
PR: "5304"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Change node ancestor/descendant algo, fixes issue where downstream models aren't
|
||||
run when using networkx >= 2.8.1
|
||||
time: 2022-06-01T13:59:08.886215-05:00
|
||||
custom:
|
||||
Author: iknox-fa
|
||||
Issue: "5286"
|
||||
PR: "5326"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Fixing Windows color regression
|
||||
time: 2022-06-01T19:42:34.263009-04:00
|
||||
custom:
|
||||
Author: leahwicz
|
||||
Issue: "5191"
|
||||
PR: "5327"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Fixes
|
||||
body: 'Define compatibility for older manifest versions when using state: selection
|
||||
methods'
|
||||
time: 2022-06-08T08:09:14.321735+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "5213"
|
||||
PR: "5346"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Remove duplicate key checking introduced in 1.2.0a1
|
||||
time: 2022-06-23T08:24:31.900647+12:00
|
||||
custom:
|
||||
Author: jeremyyeo
|
||||
Issue: "5331"
|
||||
PR: "5403"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Migrating 005_simple_seed to the new test framework.
|
||||
time: 2022-04-09T04:05:39.20045-07:00
|
||||
custom:
|
||||
Author: versusfacit
|
||||
Issue: "200"
|
||||
PR: "5013"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Convert 029_docs_generate tests to new framework
|
||||
time: 2022-04-13T18:30:14.706391-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "5035"
|
||||
PR: "5058"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Move package deprecation check outside of package cache
|
||||
time: 2022-04-14T13:22:06.157579-05:00
|
||||
custom:
|
||||
Author: emmyoop
|
||||
Issue: "5068"
|
||||
PR: "5069"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: removal of scaffold first attempt and create_adapter_plugin.py as they are deprecated new scaffold can be found https://github.com/dbt-labs/dbt-database-adapter-scaffold
|
||||
time: 2022-04-20T12:00:25.171923-05:00
|
||||
custom:
|
||||
Author: McKnight-42
|
||||
Issue: "4980"
|
||||
PR: "5117"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Mypy -> 0.942 + fixed import logic to allow for full mypy coverage
|
||||
time: 2022-04-27T11:21:27.499359-05:00
|
||||
custom:
|
||||
Author: iknox-fa
|
||||
Issue: "4805"
|
||||
PR: "5171"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Converted dbt list tests to pytest
|
||||
time: 2022-04-27T14:06:28.882908-05:00
|
||||
custom:
|
||||
Author: stu-k
|
||||
Issue: "5049"
|
||||
PR: "5178"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: 'Fix: Call str and repr for UnsetProfileConfig without a RuntimeException'
|
||||
time: 2022-05-03T19:52:12.793729384+02:00
|
||||
custom:
|
||||
Author: tomasfarias
|
||||
Issue: "5081"
|
||||
PR: "5209"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Improve tracking error logging message
|
||||
time: 2022-05-04T01:00:31.60387036+02:00
|
||||
custom:
|
||||
Author: NicolasPA
|
||||
Issue: "5197"
|
||||
PR: "5211"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: 'Clean up materialization logic: more consistent relation names, loading from
|
||||
cache'
|
||||
time: 2022-05-09T09:26:28.551068+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "2869"
|
||||
PR: "4921"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Use the default Python version for local dev and test instead of requiring Python
|
||||
3.8
|
||||
time: 2022-05-18T09:51:44.603193-07:00
|
||||
custom:
|
||||
Author: jwills
|
||||
Issue: "5257"
|
||||
PR: "5269"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Fix test for context set function
|
||||
time: 2022-05-18T14:55:22.554316-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "5266"
|
||||
PR: "5272"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Fix pip upgrade step in CI for Windows
|
||||
time: 2022-06-01T10:52:45.872931-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "5321"
|
||||
PR: "5320"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Fix unit test test_graph_selection
|
||||
time: 2022-06-01T11:26:48.725831-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "5323"
|
||||
PR: "5324"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Update context readme + clean up context code"
|
||||
time: 2022-06-06T23:03:53.022568+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "4796"
|
||||
PR: "5334"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: removed script meant for snowflake to snowflake
|
||||
time: 2022-06-10T11:10:56.080236-05:00
|
||||
custom:
|
||||
Author: McKnight-42
|
||||
Issue: "5361"
|
||||
PR: "5362"
|
||||
9
.changes/1.2.1.md
Normal file
9
.changes/1.2.1.md
Normal file
@@ -0,0 +1,9 @@
|
||||
## dbt-core 1.2.1 - August 25, 2022
|
||||
### Fixes
|
||||
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
|
||||
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
|
||||
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
|
||||
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
|
||||
|
||||
### Contributors
|
||||
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Dependencies
|
||||
body: "Bump mypy from 0.942 to 0.961"
|
||||
time: 2022-06-07T00:09:25.000000-05:00
|
||||
custom:
|
||||
Author: dependabot[bot]
|
||||
Issue: "4904"
|
||||
PR: "5337"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Dependencies
|
||||
body: "Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core"
|
||||
time: 2022-06-17T00:18:49.000000-05:00
|
||||
custom:
|
||||
Author: dependabot[bot]
|
||||
Issue: "4904"
|
||||
PR: "5388"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Dependencies
|
||||
body: "Bump black from 22.3.0 to 22.6.0"
|
||||
time: 2022-06-29T00:08:31.000000-05:00
|
||||
custom:
|
||||
Author: dependabot[bot]
|
||||
Issue: "4904"
|
||||
PR: "5420"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Docs
|
||||
body: Fixed sample SQL Code for sources when no database is defined
|
||||
time: 2022-07-06T14:20:29.7347-05:00
|
||||
custom:
|
||||
Author: b-per
|
||||
Issue: "5255"
|
||||
PR: "5446"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Docs
|
||||
body: Add support for `file:` selector in DAG viz
|
||||
time: 2022-07-06T14:21:42.544814-05:00
|
||||
custom:
|
||||
Author: drewbanin
|
||||
Issue: "5255"
|
||||
PR: "5446"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Docs
|
||||
body: '[Snyk] Upgrade prismjs from 1.27.0 to 1.28.0'
|
||||
time: 2022-07-06T14:22:26.039137-05:00
|
||||
custom:
|
||||
Author: emmyoop
|
||||
Issue: "5255"
|
||||
PR: "5446"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Docs
|
||||
body: Run build and tests in CI checks
|
||||
time: 2022-07-06T14:23:00.842859-05:00
|
||||
custom:
|
||||
Author: drewbanin
|
||||
Issue: "5255"
|
||||
PR: "5446"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Docs
|
||||
body: Improve metrics DAG viz and documentation page
|
||||
time: 2022-07-06T14:24:10.120545-05:00
|
||||
custom:
|
||||
Author: drewbanin callum-mcdata
|
||||
Issue: "5255"
|
||||
PR: "5446"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Docs
|
||||
body: Upgrade cytoscape.js fork
|
||||
time: 2022-07-06T14:25:05.575755-05:00
|
||||
custom:
|
||||
Author: emmyoop
|
||||
Issue: "5255"
|
||||
PR: "5446"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Features
|
||||
body: 'Allow customizing `target-path` and `log-path` through
|
||||
environment variables and CLI flags.'
|
||||
time: 2022-06-22T19:23:29.748577478+03:00
|
||||
custom:
|
||||
Author: isidentical
|
||||
Issue: "5399"
|
||||
PR: "5402"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Move type_* macros from dbt-utils into dbt-core, with tests
|
||||
time: 2022-06-30T13:54:52.165139+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "5317"
|
||||
PR: "5428"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Features
|
||||
body: Add support for ratio metrics
|
||||
time: 2022-07-05T08:30:26.494837-05:00
|
||||
custom:
|
||||
Author: drewbanin callum-mcdata
|
||||
Issue: "4884"
|
||||
PR: "5027"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Features
|
||||
body: Allow users to define grants as a reasonable default in the dbt_project.yml
|
||||
or within each model sql or yml file combined.
|
||||
time: 2022-07-11T11:15:14.695386-05:00
|
||||
custom:
|
||||
Author: McKnight-42
|
||||
Issue: "5263"
|
||||
PR: "5369"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Add inheritance to materialization macro resolution
|
||||
time: 2022-06-08T10:42:51.839945-04:00
|
||||
custom:
|
||||
Author: volkangurel
|
||||
Issue: "4646"
|
||||
PR: "5348"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Improve pluralizations for Documentation and SqlOperation NodeTypes
|
||||
time: 2022-06-09T11:52:46.578469-05:00
|
||||
custom:
|
||||
Author: pdebelak
|
||||
Issue: "5352"
|
||||
PR: "5356"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: Properly use quotes for Snowflake snapshots when checking all columns
|
||||
time: 2022-06-17T11:44:43.978834+02:00
|
||||
custom:
|
||||
Author: pquadri
|
||||
Issue: "2975"
|
||||
PR: "5389"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Fixes
|
||||
body: fixes handling of RESET color code with USE_COLORS=False
|
||||
time: 2022-06-17T16:00:27.038058-04:00
|
||||
custom:
|
||||
Author: darin-reify
|
||||
Issue: "5288"
|
||||
PR: "5394"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Security
|
||||
body: Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README
|
||||
time: 2022-06-06T23:03:53.022568+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "4796"
|
||||
PR: "5334"
|
||||
@@ -1,8 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Added the suggested RegEx to check the SemVer string within a package dependency
|
||||
and improved invalid version error handling.
|
||||
time: 2022-06-13T17:07:03.773668-05:00
|
||||
custom:
|
||||
Author: fivetran-joemarkiewicz
|
||||
Issue: "5201"
|
||||
PR: "5370"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: 'Add annotation to render_value method reimplemented in #5334'
|
||||
time: 2022-06-16T12:05:16.474078+02:00
|
||||
custom:
|
||||
Author: jtcohen6
|
||||
Issue: "4796"
|
||||
PR: "5382"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Bump manifest version to v6
|
||||
time: 2022-07-05T14:21:20.66768-04:00
|
||||
custom:
|
||||
Author: leahwicz
|
||||
Issue: "5417"
|
||||
PR: "5430"
|
||||
@@ -1,7 +0,0 @@
|
||||
kind: Under the Hood
|
||||
body: Add tests for SQL grants
|
||||
time: 2022-07-06T21:50:01.498562-04:00
|
||||
custom:
|
||||
Author: gshank
|
||||
Issue: "5437"
|
||||
PR: "5447"
|
||||
108
.github/workflows/version-bump.yml
vendored
108
.github/workflows/version-bump.yml
vendored
@@ -1,18 +1,15 @@
|
||||
# **what?**
|
||||
# This workflow will take a version number and a dry run flag. With that
|
||||
# This workflow will take the new version number to bump to. With that
|
||||
# it will run versionbump to update the version number everywhere in the
|
||||
# code base and then generate an update Docker requirements file. If this
|
||||
# is a dry run, a draft PR will open with the changes. If this isn't a dry
|
||||
# run, the changes will be committed to the branch this is run on.
|
||||
# code base and then run changie to create the corresponding changelog.
|
||||
# A PR will be created with the changes that can be reviewed before committing.
|
||||
|
||||
# **why?**
|
||||
# This is to aid in releasing dbt and making sure we have updated
|
||||
# the versions and Docker requirements in all places.
|
||||
# the version in all places and generated the changelog.
|
||||
|
||||
# **when?**
|
||||
# This is triggered either manually OR
|
||||
# from the repository_dispatch event "version-bump" which is sent from
|
||||
# the dbt-release repo Action
|
||||
# This is triggered manually
|
||||
|
||||
name: Version Bump
|
||||
|
||||
@@ -20,35 +17,21 @@ on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version_number:
|
||||
description: 'The version number to bump to'
|
||||
description: 'The version number to bump to (ex. 1.2.0, 1.3.0b1)'
|
||||
required: true
|
||||
is_dry_run:
|
||||
description: 'Creates a draft PR to allow testing instead of committing to a branch'
|
||||
required: true
|
||||
default: 'true'
|
||||
repository_dispatch:
|
||||
types: [version-bump]
|
||||
|
||||
jobs:
|
||||
bump:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: "[DEBUG] Print Variables"
|
||||
run: |
|
||||
echo "all variables defined as inputs"
|
||||
echo The version_number: ${{ github.event.inputs.version_number }}
|
||||
|
||||
- name: Check out the repository
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Set version and dry run values
|
||||
id: variables
|
||||
env:
|
||||
VERSION_NUMBER: "${{ github.event.client_payload.version_number == '' && github.event.inputs.version_number || github.event.client_payload.version_number }}"
|
||||
IS_DRY_RUN: "${{ github.event.client_payload.is_dry_run == '' && github.event.inputs.is_dry_run || github.event.client_payload.is_dry_run }}"
|
||||
run: |
|
||||
echo Repository dispatch event version: ${{ github.event.client_payload.version_number }}
|
||||
echo Repository dispatch event dry run: ${{ github.event.client_payload.is_dry_run }}
|
||||
echo Workflow dispatch event version: ${{ github.event.inputs.version_number }}
|
||||
echo Workflow dispatch event dry run: ${{ github.event.inputs.is_dry_run }}
|
||||
echo ::set-output name=VERSION_NUMBER::$VERSION_NUMBER
|
||||
echo ::set-output name=IS_DRY_RUN::$IS_DRY_RUN
|
||||
|
||||
- uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: "3.8"
|
||||
@@ -59,53 +42,66 @@ jobs:
|
||||
source env/bin/activate
|
||||
pip install --upgrade pip
|
||||
|
||||
- name: Create PR branch
|
||||
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
|
||||
run: |
|
||||
git checkout -b bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
|
||||
git push origin bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
|
||||
git branch --set-upstream-to=origin/bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
|
||||
- name: Audit Version and Parse Into Parts
|
||||
id: semver
|
||||
uses: dbt-labs/actions/parse-semver@v1
|
||||
with:
|
||||
version: ${{ github.event.inputs.version_number }}
|
||||
|
||||
# - name: Generate Docker requirements
|
||||
# run: |
|
||||
# source env/bin/activate
|
||||
# pip install -r requirements.txt
|
||||
# pip freeze -l > docker/requirements/requirements.txt
|
||||
# git status
|
||||
- name: Set branch value
|
||||
id: variables
|
||||
run: |
|
||||
echo "::set-output name=BRANCH_NAME::prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID"
|
||||
|
||||
- name: Create PR branch
|
||||
run: |
|
||||
git checkout -b ${{ steps.variables.outputs.BRANCH_NAME }}
|
||||
git push origin ${{ steps.variables.outputs.BRANCH_NAME }}
|
||||
git branch --set-upstream-to=origin/${{ steps.variables.outputs.BRANCH_NAME }} ${{ steps.variables.outputs.BRANCH_NAME }}
|
||||
|
||||
- name: Bump version
|
||||
run: |
|
||||
source env/bin/activate
|
||||
pip install -r dev-requirements.txt
|
||||
env/bin/bumpversion --allow-dirty --new-version ${{steps.variables.outputs.VERSION_NUMBER}} major
|
||||
env/bin/bumpversion --allow-dirty --new-version ${{ github.event.inputs.version_number }} major
|
||||
git status
|
||||
|
||||
- name: Commit version bump directly
|
||||
uses: EndBug/add-and-commit@v7
|
||||
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'false' }}
|
||||
with:
|
||||
author_name: 'Github Build Bot'
|
||||
author_email: 'buildbot@fishtownanalytics.com'
|
||||
message: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
|
||||
# this step will fail on whitespace errors but also correct them
|
||||
- name: Format bumpversion file
|
||||
continue-on-error: true
|
||||
run: |
|
||||
brew install pre-commit
|
||||
pre-commit run trailing-whitespace --files .bumpversion.cfg
|
||||
git status
|
||||
|
||||
- name: Run changie
|
||||
run: |
|
||||
brew tap miniscruff/changie https://github.com/miniscruff/changie
|
||||
brew install changie
|
||||
if [[ ${{ steps.semver.outputs.is-pre-release }} -eq 1 ]]
|
||||
then
|
||||
changie batch ${{ steps.semver.outputs.base-version }} --move-dir '${{ steps.semver.outputs.base-version }}' --prerelease '${{ steps.semver.outputs.pre-release }}'
|
||||
else
|
||||
changie batch ${{ steps.semver.outputs.base-version }} --include '${{ steps.semver.outputs.base-version }}' --remove-prereleases
|
||||
fi
|
||||
changie merge
|
||||
git status
|
||||
|
||||
- name: Commit version bump to branch
|
||||
uses: EndBug/add-and-commit@v7
|
||||
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
|
||||
with:
|
||||
author_name: 'Github Build Bot'
|
||||
author_email: 'buildbot@fishtownanalytics.com'
|
||||
message: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
|
||||
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
|
||||
push: 'origin origin/bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
|
||||
message: 'Bumping version to ${{ github.event.inputs.version_number }} and generate CHANGELOG'
|
||||
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
|
||||
push: 'origin origin/${{ steps.variables.outputs.BRANCH_NAME }}'
|
||||
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@v3
|
||||
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
|
||||
with:
|
||||
author: 'Github Build Bot <buildbot@fishtownanalytics.com>'
|
||||
draft: true
|
||||
base: ${{github.ref}}
|
||||
title: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
|
||||
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
|
||||
title: 'Bumping version to ${{ github.event.inputs.version_number }} and generate changelog'
|
||||
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
|
||||
labels: |
|
||||
Skip Changelog
|
||||
|
||||
53
CHANGELOG.md
53
CHANGELOG.md
@@ -4,9 +4,16 @@
|
||||
- Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
|
||||
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
|
||||
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
|
||||
## dbt-core 1.2.1 - August 25, 2022
|
||||
### Fixes
|
||||
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
|
||||
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
|
||||
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
|
||||
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
|
||||
|
||||
|
||||
## dbt-core 1.2.0-b1 - June 24, 2022
|
||||
### Contributors
|
||||
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
|
||||
## dbt-core 1.2.0 - July 26, 2022
|
||||
### Features
|
||||
- Add selector method when reading selector definitions ([#4821](https://github.com/dbt-labs/dbt-core/issues/4821), [#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
|
||||
- Add set and zip function to contexts ([#2345](https://github.com/dbt-labs/dbt-core/issues/2345), [#5107](https://github.com/dbt-labs/dbt-core/pull/5107))
|
||||
@@ -18,6 +25,11 @@
|
||||
- Move cross-db macros from dbt-utils into dbt-core global project ([#4813](https://github.com/dbt-labs/dbt-core/issues/4813), [#5265](https://github.com/dbt-labs/dbt-core/pull/5265))
|
||||
- Prettify duration message at the end of execution ([#5253](https://github.com/dbt-labs/dbt-core/issues/5253), [#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
|
||||
- Early return from dbt init if no available adapters ([#5365](https://github.com/dbt-labs/dbt-core/issues/5365), [#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
|
||||
- Allow customizing `target-path` and `log-path` through environment variables and CLI flags. ([#5399](https://github.com/dbt-labs/dbt-core/issues/5399), [#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
|
||||
- Move type_* macros from dbt-utils into dbt-core, with tests ([#5317](https://github.com/dbt-labs/dbt-core/issues/5317), [#5428](https://github.com/dbt-labs/dbt-core/pull/5428))
|
||||
- Add support for ratio metrics ([#4884](https://github.com/dbt-labs/dbt-core/issues/4884), [#5027](https://github.com/dbt-labs/dbt-core/pull/5027))
|
||||
- Allow users to define grants as a reasonable default in the dbt_project.yml or within each model sql or yml file combined. ([#5263](https://github.com/dbt-labs/dbt-core/issues/5263), [#5369](https://github.com/dbt-labs/dbt-core/pull/5369))
|
||||
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
|
||||
### Fixes
|
||||
- Adding new cols to check_cols in snapshots ([#3146](https://github.com/dbt-labs/dbt-core/issues/3146), [#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
|
||||
- Truncate relation names when appending a suffix that will result in len > 63 characters using make_temp_relation and make_backup_relation macros ([#2869](https://github.com/dbt-labs/dbt-core/issues/2869), [#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
|
||||
@@ -37,7 +49,19 @@
|
||||
- Change node ancestor/descendant algo, fixes issue where downstream models aren't run when using networkx >= 2.8.1 ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5326](https://github.com/dbt-labs/dbt-core/pull/5326))
|
||||
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
|
||||
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
|
||||
- Add inheritance to materialization macro resolution ([#4646](https://github.com/dbt-labs/dbt-core/issues/4646), [#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
|
||||
- Improve pluralizations for Documentation and SqlOperation NodeTypes ([#5352](https://github.com/dbt-labs/dbt-core/issues/5352), [#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
|
||||
- Properly use quotes for Snowflake snapshots when checking all columns ([#2975](https://github.com/dbt-labs/dbt-core/issues/2975), [#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
|
||||
- fixes handling of RESET color code with USE_COLORS=False ([#5288](https://github.com/dbt-labs/dbt-core/issues/5288), [#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
|
||||
- Remove duplicate key checking introduced in 1.2.0a1 ([#5331](https://github.com/dbt-labs/dbt-core/issues/5331), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
|
||||
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
|
||||
### Docs
|
||||
- Fixed sample SQL Code for sources when no database is defined ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- Add support for `file:` selector in DAG viz ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- [Snyk] Upgrade prismjs from 1.27.0 to 1.28.0 ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- Run build and tests in CI checks ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- Improve metrics DAG viz and documentation page ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- Upgrade cytoscape.js fork ([#5255](https://github.com/dbt-labs/dbt-core/issues/5255), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
### Under the Hood
|
||||
- Migrating 005_simple_seed to the new test framework. ([#200](https://github.com/dbt-labs/dbt-core/issues/200), [#5013](https://github.com/dbt-labs/dbt-core/pull/5013))
|
||||
- Convert 029_docs_generate tests to new framework ([#5035](https://github.com/dbt-labs/dbt-core/issues/5035), [#5058](https://github.com/dbt-labs/dbt-core/pull/5058))
|
||||
@@ -54,29 +78,44 @@
|
||||
- Fix unit test test_graph_selection ([#5323](https://github.com/dbt-labs/dbt-core/issues/5323), [#5324](https://github.com/dbt-labs/dbt-core/pull/5324))
|
||||
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
|
||||
- removed script meant for snowflake to snowflake ([#5361](https://github.com/dbt-labs/dbt-core/issues/5361), [#5362](https://github.com/dbt-labs/dbt-core/pull/5362))
|
||||
- Added the suggested RegEx to check the SemVer string within a package dependency and improved invalid version error handling. ([#5201](https://github.com/dbt-labs/dbt-core/issues/5201), [#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
|
||||
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
|
||||
- Bump manifest version to v6 ([#5417](https://github.com/dbt-labs/dbt-core/issues/5417), [#5430](https://github.com/dbt-labs/dbt-core/pull/5430))
|
||||
- Add tests for SQL grants ([#5437](https://github.com/dbt-labs/dbt-core/issues/5437), [#5447](https://github.com/dbt-labs/dbt-core/pull/5447))
|
||||
### Dependencies
|
||||
- Bump ubuntu from 20.04 to 22.04 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5141](https://github.com/dbt-labs/dbt-core/pull/5141))
|
||||
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
|
||||
- Bump mypy from 0.942 to 0.961 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5337](https://github.com/dbt-labs/dbt-core/pull/5337))
|
||||
- Bump python from 3.10.3-slim-bullseye to 3.10.5-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
|
||||
- Update colorama requirement from <0.4.5,>=0.3.9 to >=0.3.9,<0.4.6 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5388](https://github.com/dbt-labs/dbt-core/pull/5388))
|
||||
- Bump black from 22.3.0 to 22.6.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5420](https://github.com/dbt-labs/dbt-core/pull/5420))
|
||||
### Security
|
||||
- Move string interpolation of "secret" env vars outside of Jinja context. Update "contexts" README ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
|
||||
|
||||
### Contributors
|
||||
- [@GtheSheep](https://github.com/GtheSheep) ([#4893](https://github.com/dbt-labs/dbt-core/pull/4893))
|
||||
- [@NicolasPA](https://github.com/NicolasPA) ([#5211](https://github.com/dbt-labs/dbt-core/pull/5211))
|
||||
- [@adamantike](https://github.com/adamantike) ([#5207](https://github.com/dbt-labs/dbt-core/pull/5207))
|
||||
- [@alexrosenfeld10](https://github.com/alexrosenfeld10) ([#5184](https://github.com/dbt-labs/dbt-core/pull/5184))
|
||||
- [@b-per](https://github.com/b-per) ([#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- [@bd3dowling](https://github.com/bd3dowling) ([#5140](https://github.com/dbt-labs/dbt-core/pull/5140))
|
||||
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- [@danieldiamond](https://github.com/danieldiamond) ([#4827](https://github.com/dbt-labs/dbt-core/pull/4827))
|
||||
- [@darin-reify](https://github.com/darin-reify) ([#5394](https://github.com/dbt-labs/dbt-core/pull/5394))
|
||||
- [@dbeatty10](https://github.com/dbeatty10) ([#5265](https://github.com/dbt-labs/dbt-core/pull/5265), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
|
||||
- [@dependabot[bot]](https://github.com/dependabot[bot]) ([#5141](https://github.com/dbt-labs/dbt-core/pull/5141), [#5367](https://github.com/dbt-labs/dbt-core/pull/5367))
|
||||
- [@drewbanin](https://github.com/drewbanin) ([#5027](https://github.com/dbt-labs/dbt-core/pull/5027), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446), [#5446](https://github.com/dbt-labs/dbt-core/pull/5446))
|
||||
- [@epapineau](https://github.com/epapineau) ([#4921](https://github.com/dbt-labs/dbt-core/pull/4921))
|
||||
- [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) ([#5370](https://github.com/dbt-labs/dbt-core/pull/5370))
|
||||
- [@groodt](https://github.com/groodt) ([#5304](https://github.com/dbt-labs/dbt-core/pull/5304))
|
||||
- [@isidentical](https://github.com/isidentical) ([#5402](https://github.com/dbt-labs/dbt-core/pull/5402))
|
||||
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5364](https://github.com/dbt-labs/dbt-core/pull/5364))
|
||||
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403))
|
||||
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5107](https://github.com/dbt-labs/dbt-core/pull/5107), [#5146](https://github.com/dbt-labs/dbt-core/pull/5146), [#5403](https://github.com/dbt-labs/dbt-core/pull/5403), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
|
||||
- [@jwills](https://github.com/jwills) ([#5241](https://github.com/dbt-labs/dbt-core/pull/5241), [#5269](https://github.com/dbt-labs/dbt-core/pull/5269))
|
||||
- [@tomasfarias](https://github.com/tomasfarias) ([#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
|
||||
- [@pdebelak](https://github.com/pdebelak) ([#5356](https://github.com/dbt-labs/dbt-core/pull/5356))
|
||||
- [@pquadri](https://github.com/pquadri) ([#5389](https://github.com/dbt-labs/dbt-core/pull/5389))
|
||||
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432), [#5209](https://github.com/dbt-labs/dbt-core/pull/5209))
|
||||
- [@ulisesojeda](https://github.com/ulisesojeda) ([#5366](https://github.com/dbt-labs/dbt-core/pull/5366))
|
||||
|
||||
|
||||
- [@volkangurel](https://github.com/volkangurel) ([#5348](https://github.com/dbt-labs/dbt-core/pull/5348))
|
||||
## Previous Releases
|
||||
|
||||
For information on prior major and minor releases, see their changelogs:
|
||||
|
||||
@@ -1,10 +1,24 @@
|
||||
import abc
|
||||
import os
|
||||
from time import sleep
|
||||
import sys
|
||||
|
||||
# multiprocessing.RLock is a function returning this type
|
||||
from multiprocessing.synchronize import RLock
|
||||
from threading import get_ident
|
||||
from typing import Dict, Tuple, Hashable, Optional, ContextManager, List
|
||||
from typing import (
|
||||
Any,
|
||||
Dict,
|
||||
Tuple,
|
||||
Hashable,
|
||||
Optional,
|
||||
ContextManager,
|
||||
List,
|
||||
Type,
|
||||
Union,
|
||||
Iterable,
|
||||
Callable,
|
||||
)
|
||||
|
||||
import agate
|
||||
|
||||
@@ -21,6 +35,7 @@ from dbt.contracts.graph.manifest import Manifest
|
||||
from dbt.adapters.base.query_headers import (
|
||||
MacroQueryStringSetter,
|
||||
)
|
||||
from dbt.events import AdapterLogger
|
||||
from dbt.events.functions import fire_event
|
||||
from dbt.events.types import (
|
||||
NewConnection,
|
||||
@@ -34,6 +49,9 @@ from dbt.events.types import (
|
||||
)
|
||||
from dbt import flags
|
||||
|
||||
SleepTime = Union[int, float] # As taken by time.sleep.
|
||||
AdapterHandle = Any # Adapter connection handle objects can be any class.
|
||||
|
||||
|
||||
class BaseConnectionManager(metaclass=abc.ABCMeta):
|
||||
"""Methods to implement:
|
||||
@@ -159,6 +177,94 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
|
||||
conn.name = conn_name
|
||||
return conn
|
||||
|
||||
@classmethod
|
||||
def retry_connection(
|
||||
cls,
|
||||
connection: Connection,
|
||||
connect: Callable[[], AdapterHandle],
|
||||
logger: AdapterLogger,
|
||||
retryable_exceptions: Iterable[Type[Exception]],
|
||||
retry_limit: int = 1,
|
||||
retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1,
|
||||
_attempts: int = 0,
|
||||
) -> Connection:
|
||||
"""Given a Connection, set its handle by calling connect.
|
||||
|
||||
The calls to connect will be retried up to retry_limit times to deal with transient
|
||||
connection errors. By default, one retry will be attempted if retryable_exceptions is set.
|
||||
|
||||
:param Connection connection: An instance of a Connection that needs a handle to be set,
|
||||
usually when attempting to open it.
|
||||
:param connect: A callable that returns the appropiate connection handle for a
|
||||
given adapter. This callable will be retried retry_limit times if a subclass of any
|
||||
Exception in retryable_exceptions is raised by connect.
|
||||
:type connect: Callable[[], AdapterHandle]
|
||||
:param AdapterLogger logger: A logger to emit messages on retry attempts or errors. When
|
||||
handling expected errors, we call debug, and call warning on unexpected errors or when
|
||||
all retry attempts have been exhausted.
|
||||
:param retryable_exceptions: An iterable of exception classes that if raised by
|
||||
connect should trigger a retry.
|
||||
:type retryable_exceptions: Iterable[Type[Exception]]
|
||||
:param int retry_limit: How many times to retry the call to connect. If this limit
|
||||
is exceeded before a successful call, a FailedToConnectException will be raised.
|
||||
Must be non-negative.
|
||||
:param retry_timeout: Time to wait between attempts to connect. Can also take a
|
||||
Callable that takes the number of attempts so far, beginning at 0, and returns an int
|
||||
or float to be passed to time.sleep.
|
||||
:type retry_timeout: Union[Callable[[int], SleepTime], SleepTime] = 1
|
||||
:param int _attempts: Parameter used to keep track of the number of attempts in calling the
|
||||
connect function across recursive calls. Passed as an argument to retry_timeout if it
|
||||
is a Callable. This parameter should not be set by the initial caller.
|
||||
:raises dbt.exceptions.FailedToConnectException: Upon exhausting all retry attempts without
|
||||
successfully acquiring a handle.
|
||||
:return: The given connection with its appropriate state and handle attributes set
|
||||
depending on whether we successfully acquired a handle or not.
|
||||
"""
|
||||
timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout
|
||||
if timeout < 0:
|
||||
raise dbt.exceptions.FailedToConnectException(
|
||||
"retry_timeout cannot be negative or return a negative time."
|
||||
)
|
||||
|
||||
if retry_limit < 0 or retry_limit > sys.getrecursionlimit():
|
||||
# This guard is not perfect others may add to the recursion limit (e.g. built-ins).
|
||||
connection.handle = None
|
||||
connection.state = ConnectionState.FAIL
|
||||
raise dbt.exceptions.FailedToConnectException("retry_limit cannot be negative")
|
||||
|
||||
try:
|
||||
connection.handle = connect()
|
||||
connection.state = ConnectionState.OPEN
|
||||
return connection
|
||||
|
||||
except tuple(retryable_exceptions) as e:
|
||||
if retry_limit <= 0:
|
||||
connection.handle = None
|
||||
connection.state = ConnectionState.FAIL
|
||||
raise dbt.exceptions.FailedToConnectException(str(e))
|
||||
|
||||
logger.debug(
|
||||
f"Got a retryable error when attempting to open a {cls.TYPE} connection.\n"
|
||||
f"{retry_limit} attempts remaining. Retrying in {timeout} seconds.\n"
|
||||
f"Error:\n{e}"
|
||||
)
|
||||
|
||||
sleep(timeout)
|
||||
return cls.retry_connection(
|
||||
connection=connection,
|
||||
connect=connect,
|
||||
logger=logger,
|
||||
retry_limit=retry_limit - 1,
|
||||
retry_timeout=retry_timeout,
|
||||
retryable_exceptions=retryable_exceptions,
|
||||
_attempts=_attempts + 1,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
connection.handle = None
|
||||
connection.state = ConnectionState.FAIL
|
||||
raise dbt.exceptions.FailedToConnectException(str(e))
|
||||
|
||||
@abc.abstractmethod
|
||||
def cancel_open(self) -> Optional[List[str]]:
|
||||
"""Cancel all open connections on the adapter. (passable)"""
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import re
|
||||
import threading
|
||||
from copy import deepcopy
|
||||
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
|
||||
@@ -413,9 +414,24 @@ class RelationsCache:
|
||||
:raises InternalError: If the new key is already present.
|
||||
"""
|
||||
if new_key in self.relations:
|
||||
# Tell user when collision caused by model names truncated during
|
||||
# materialization.
|
||||
match = re.search("__dbt_backup|__dbt_tmp$", new_key.identifier)
|
||||
if match:
|
||||
truncated_model_name_prefix = new_key.identifier[: match.start()]
|
||||
message_addendum = (
|
||||
"\n\nName collisions can occur when the length of two "
|
||||
"models' names approach your database's builtin limit. "
|
||||
"Try restructuring your project such that no two models "
|
||||
"share the prefix '{}'.".format(truncated_model_name_prefix)
|
||||
+ " Then, clean your warehouse of any removed models."
|
||||
)
|
||||
else:
|
||||
message_addendum = ""
|
||||
|
||||
dbt.exceptions.raise_cache_inconsistent(
|
||||
"in rename, new key {} already in cache: {}".format(
|
||||
new_key, list(self.relations.keys())
|
||||
"in rename, new key {} already in cache: {}{}".format(
|
||||
new_key, list(self.relations.keys()), message_addendum
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -474,19 +474,17 @@ class BaseContext(metaclass=ContextMeta):
|
||||
|
||||
@contextmember
|
||||
@staticmethod
|
||||
def try_set(value: Iterable[Any]) -> Set[Any]:
|
||||
"""The `try_set` context method can be used to convert any iterable
|
||||
def set_strict(value: Iterable[Any]) -> Set[Any]:
|
||||
"""The `set_strict` context method can be used to convert any iterable
|
||||
to a sequence of iterable elements that are unique (a set). The
|
||||
difference to the `set` context method is that the `try_set` method
|
||||
difference to the `set` context method is that the `set_strict` method
|
||||
will raise an exception on a TypeError.
|
||||
|
||||
:param value: The iterable
|
||||
:param default: A default value to return if the `value` argument
|
||||
is not an iterable
|
||||
|
||||
Usage:
|
||||
{% set my_list = [1, 2, 2, 3] %}
|
||||
{% set my_set = try_set(my_list) %}
|
||||
{% set my_set = set_strict(my_list) %}
|
||||
{% do log(my_set) %} {# {1, 2, 3} #}
|
||||
"""
|
||||
try:
|
||||
@@ -497,7 +495,7 @@ class BaseContext(metaclass=ContextMeta):
|
||||
@contextmember("zip")
|
||||
@staticmethod
|
||||
def _zip(*args: Iterable[Any], default: Any = None) -> Optional[Iterable[Any]]:
|
||||
"""The `try_zip` context method can be used to used to return
|
||||
"""The `zip` context method can be used to used to return
|
||||
an iterator of tuples, where the i-th tuple contains the i-th
|
||||
element from each of the argument iterables.
|
||||
|
||||
@@ -518,21 +516,19 @@ class BaseContext(metaclass=ContextMeta):
|
||||
|
||||
@contextmember
|
||||
@staticmethod
|
||||
def try_zip(*args: Iterable[Any]) -> Iterable[Any]:
|
||||
"""The `try_zip` context method can be used to used to return
|
||||
def zip_strict(*args: Iterable[Any]) -> Iterable[Any]:
|
||||
"""The `zip_strict` context method can be used to used to return
|
||||
an iterator of tuples, where the i-th tuple contains the i-th
|
||||
element from each of the argument iterables. The difference to the
|
||||
`zip` context method is that the `try_zip` method will raise an
|
||||
`zip` context method is that the `zip_strict` method will raise an
|
||||
exception on a TypeError.
|
||||
|
||||
:param *args: Any number of iterables
|
||||
:param default: A default value to return if `*args` is not
|
||||
iterable
|
||||
|
||||
Usage:
|
||||
{% set my_list_a = [1, 2] %}
|
||||
{% set my_list_b = ['alice', 'bob'] %}
|
||||
{% set my_zip = try_zip(my_list_a, my_list_b) | list %}
|
||||
{% set my_zip = zip_strict(my_list_a, my_list_b) | list %}
|
||||
{% do log(my_set) %} {# [(1, 'alice'), (2, 'bob')] #}
|
||||
"""
|
||||
try:
|
||||
|
||||
@@ -105,6 +105,8 @@ class ErrorLevel(EventSerialization, Event):
|
||||
|
||||
|
||||
# prevents an event from going to the file
|
||||
# This should rarely be used in core code. It is currently
|
||||
# only used in integration tests and for the 'clean' command.
|
||||
class NoFile:
|
||||
pass
|
||||
|
||||
|
||||
@@ -218,7 +218,7 @@ def create_log_line(e: T_Event, file_output=False) -> Optional[str]:
|
||||
return create_info_text_log_line(e) # console output
|
||||
|
||||
|
||||
# allows for resuse of this obnoxious if else tree.
|
||||
# allows for reuse of this obnoxious if else tree.
|
||||
# do not use for exceptions, it doesn't pass along exc_info, stack_info, or extra
|
||||
def send_to_logger(l: Union[Logger, logbook.Logger], level_tag: str, log_line: str):
|
||||
if not log_line:
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
from dataclasses import dataclass
|
||||
from .types import InfoLevel, DebugLevel, WarnLevel, ErrorLevel, ShowException, NoFile
|
||||
from dbt.events.types import InfoLevel, DebugLevel, WarnLevel, ErrorLevel, ShowException
|
||||
from dbt.events.base_types import NoFile
|
||||
|
||||
|
||||
# Keeping log messages for testing separate since they are used for debugging.
|
||||
|
||||
@@ -94,7 +94,7 @@ class AdapterEventError(ErrorLevel, AdapterEventBase, ShowException):
|
||||
|
||||
|
||||
@dataclass
|
||||
class MainKeyboardInterrupt(InfoLevel, NoFile):
|
||||
class MainKeyboardInterrupt(InfoLevel):
|
||||
code: str = "Z001"
|
||||
|
||||
def message(self) -> str:
|
||||
@@ -102,7 +102,7 @@ class MainKeyboardInterrupt(InfoLevel, NoFile):
|
||||
|
||||
|
||||
@dataclass
|
||||
class MainEncounteredError(ErrorLevel, NoFile):
|
||||
class MainEncounteredError(ErrorLevel):
|
||||
e: BaseException
|
||||
code: str = "Z002"
|
||||
|
||||
@@ -111,7 +111,7 @@ class MainEncounteredError(ErrorLevel, NoFile):
|
||||
|
||||
|
||||
@dataclass
|
||||
class MainStackTrace(DebugLevel, NoFile):
|
||||
class MainStackTrace(ErrorLevel):
|
||||
stack_trace: str
|
||||
code: str = "Z003"
|
||||
|
||||
@@ -1353,6 +1353,8 @@ class NodeConnectionReleaseError(ShowException, DebugLevel):
|
||||
return "Error releasing connection for node {}: {!s}".format(self.node_name, self.exc)
|
||||
|
||||
|
||||
# We don't write "clean" events to the log, because the clean command
|
||||
# may have removed the log directory.
|
||||
@dataclass
|
||||
class CheckCleanPath(InfoLevel, NoFile):
|
||||
path: str
|
||||
@@ -2382,7 +2384,7 @@ class SendEventFailure(DebugLevel):
|
||||
|
||||
|
||||
@dataclass
|
||||
class FlushEvents(DebugLevel, NoFile):
|
||||
class FlushEvents(DebugLevel):
|
||||
code: str = "Z042"
|
||||
|
||||
def message(self) -> str:
|
||||
@@ -2390,7 +2392,7 @@ class FlushEvents(DebugLevel, NoFile):
|
||||
|
||||
|
||||
@dataclass
|
||||
class FlushEventsFailure(DebugLevel, NoFile):
|
||||
class FlushEventsFailure(DebugLevel):
|
||||
code: str = "Z043"
|
||||
|
||||
def message(self) -> str:
|
||||
|
||||
@@ -142,7 +142,6 @@ def main(args=None):
|
||||
exit_code = e.code
|
||||
|
||||
except BaseException as e:
|
||||
traceback.print_exc()
|
||||
fire_event(MainEncounteredError(e=str(e)))
|
||||
fire_event(MainStackTrace(stack_trace=traceback.format_exc()))
|
||||
exit_code = ExitCodes.UnhandledError.value
|
||||
|
||||
@@ -69,6 +69,8 @@ class SchemaYamlRenderer(BaseRenderer):
|
||||
elif self.key == "metrics":
|
||||
if keypath[0] == "sql":
|
||||
return False
|
||||
elif self._is_norender_key(keypath[0:]):
|
||||
return False
|
||||
else: # models, seeds, snapshots, analyses
|
||||
if self._is_norender_key(keypath[0:]):
|
||||
return False
|
||||
|
||||
@@ -3,6 +3,7 @@ import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from typing import Optional
|
||||
|
||||
import yaml
|
||||
@@ -302,7 +303,7 @@ class InitTask(BaseTask):
|
||||
available_adapters = list(_get_adapter_plugin_names())
|
||||
if not len(available_adapters):
|
||||
print("No adapters available. Go to https://docs.getdbt.com/docs/available-adapters")
|
||||
exit(1)
|
||||
sys.exit(1)
|
||||
project_name = self.get_valid_project_name()
|
||||
project_path = Path(project_name)
|
||||
if project_path.exists():
|
||||
|
||||
@@ -235,5 +235,5 @@ def _get_adapter_plugin_names() -> Iterator[str]:
|
||||
yield plugin_name
|
||||
|
||||
|
||||
__version__ = "1.2.0b1"
|
||||
__version__ = "1.2.1"
|
||||
installed = get_installed_version()
|
||||
|
||||
@@ -25,7 +25,7 @@ with open(os.path.join(this_directory, "README.md")) as f:
|
||||
|
||||
|
||||
package_name = "dbt-core"
|
||||
package_version = "1.2.0b1"
|
||||
package_version = "1.2.1"
|
||||
description = """With dbt, data analysts and engineers can build analytics \
|
||||
the way engineers build applications."""
|
||||
|
||||
|
||||
@@ -14,12 +14,12 @@ FROM --platform=$build_for python:3.10.5-slim-bullseye as base
|
||||
# N.B. The refs updated automagically every release via bumpversion
|
||||
# N.B. dbt-postgres is currently found in the core codebase so a value of dbt-core@<some_version> is correct
|
||||
|
||||
ARG dbt_core_ref=dbt-core@v1.2.0b1
|
||||
ARG dbt_postgres_ref=dbt-core@v1.2.0b1
|
||||
ARG dbt_redshift_ref=dbt-redshift@v1.0.0
|
||||
ARG dbt_bigquery_ref=dbt-bigquery@v1.0.0
|
||||
ARG dbt_snowflake_ref=dbt-snowflake@v1.0.0
|
||||
ARG dbt_spark_ref=dbt-spark@v1.0.0
|
||||
ARG dbt_core_ref=dbt-core@v1.2.1
|
||||
ARG dbt_postgres_ref=dbt-core@v1.2.1
|
||||
ARG dbt_redshift_ref=dbt-redshift@v1.2.1
|
||||
ARG dbt_bigquery_ref=dbt-bigquery@v1.2.0
|
||||
ARG dbt_snowflake_ref=dbt-snowflake@v1.2.0
|
||||
ARG dbt_spark_ref=dbt-spark@v1.2.0
|
||||
# special case args
|
||||
ARG dbt_spark_version=all
|
||||
ARG dbt_third_party
|
||||
|
||||
@@ -1 +1 @@
|
||||
version = "1.2.0b1"
|
||||
version = "1.2.1"
|
||||
|
||||
@@ -31,6 +31,7 @@ class PostgresCredentials(Credentials):
|
||||
sslkey: Optional[str] = None
|
||||
sslrootcert: Optional[str] = None
|
||||
application_name: Optional[str] = "dbt"
|
||||
retries: int = 1
|
||||
|
||||
_ALIASES = {"dbname": "database", "pass": "password"}
|
||||
|
||||
@@ -121,7 +122,7 @@ class PostgresConnectionManager(SQLConnectionManager):
|
||||
if credentials.application_name:
|
||||
kwargs["application_name"] = credentials.application_name
|
||||
|
||||
try:
|
||||
def connect():
|
||||
handle = psycopg2.connect(
|
||||
dbname=credentials.database,
|
||||
user=credentials.user,
|
||||
@@ -131,23 +132,26 @@ class PostgresConnectionManager(SQLConnectionManager):
|
||||
connect_timeout=credentials.connect_timeout,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
if credentials.role:
|
||||
handle.cursor().execute("set role {}".format(credentials.role))
|
||||
return handle
|
||||
|
||||
connection.handle = handle
|
||||
connection.state = "open"
|
||||
except psycopg2.Error as e:
|
||||
logger.debug(
|
||||
"Got an error when attempting to open a postgres " "connection: '{}'".format(e)
|
||||
)
|
||||
retryable_exceptions = [
|
||||
# OperationalError is subclassed by all psycopg2 Connection Exceptions and it's raised
|
||||
# by generic connection timeouts without an error code. This is a limitation of
|
||||
# psycopg2 which doesn't provide subclasses for errors without a SQLSTATE error code.
|
||||
# The limitation has been known for a while and there are no efforts to tackle it.
|
||||
# See: https://github.com/psycopg/psycopg2/issues/682
|
||||
psycopg2.errors.OperationalError,
|
||||
]
|
||||
|
||||
connection.handle = None
|
||||
connection.state = "fail"
|
||||
|
||||
raise dbt.exceptions.FailedToConnectException(str(e))
|
||||
|
||||
return connection
|
||||
return cls.retry_connection(
|
||||
connection,
|
||||
connect=connect,
|
||||
logger=logger,
|
||||
retry_limit=credentials.retries,
|
||||
retryable_exceptions=retryable_exceptions,
|
||||
)
|
||||
|
||||
def cancel(self, connection):
|
||||
connection_name = connection.name
|
||||
|
||||
@@ -146,7 +146,7 @@
|
||||
{% set suffix = suffix ~ dtstring %}
|
||||
{% endif %}
|
||||
{% set suffix_length = suffix|length %}
|
||||
{% set relation_max_name_length = 63 %}
|
||||
{% set relation_max_name_length = base_relation.relation_max_name_length() %}
|
||||
{% if suffix_length > relation_max_name_length %}
|
||||
{% do exceptions.raise_compiler_error('Relation suffix is too long (' ~ suffix_length ~ ' characters). Maximum length is ' ~ relation_max_name_length ~ ' characters.') %}
|
||||
{% endif %}
|
||||
|
||||
@@ -41,7 +41,7 @@ def _dbt_psycopg2_name():
|
||||
|
||||
|
||||
package_name = "dbt-postgres"
|
||||
package_version = "1.2.0b1"
|
||||
package_version = "1.2.1"
|
||||
description = """The postgres adapter plugin for dbt (data build tool)"""
|
||||
|
||||
this_directory = os.path.abspath(os.path.dirname(__file__))
|
||||
|
||||
494
test/unit/test_adapter_connection_manager.py
Normal file
494
test/unit/test_adapter_connection_manager.py
Normal file
@@ -0,0 +1,494 @@
|
||||
import unittest
|
||||
from unittest import mock
|
||||
import sys
|
||||
|
||||
import dbt.exceptions
|
||||
|
||||
import psycopg2
|
||||
|
||||
from dbt.contracts.connection import Connection
|
||||
from dbt.adapters.base import BaseConnectionManager
|
||||
from dbt.adapters.postgres import PostgresCredentials, PostgresConnectionManager
|
||||
from dbt.events import AdapterLogger
|
||||
|
||||
|
||||
class BaseConnectionManagerTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.postgres_credentials = PostgresCredentials(
|
||||
host="localhost",
|
||||
user="test-user",
|
||||
port=1111,
|
||||
password="test-password",
|
||||
database="test-db",
|
||||
schema="test-schema",
|
||||
)
|
||||
self.logger = AdapterLogger("test")
|
||||
self.postgres_connection = Connection("postgres", None, self.postgres_credentials)
|
||||
|
||||
def test_retry_connection(self):
|
||||
"""Test a dummy handle is set on a connection on the first attempt.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects the Connection.handle attribute to be set to True and it's state to
|
||||
"open", after calling retry_connection.
|
||||
|
||||
Moreover, the attribute should be set in the first attempt as no exception would
|
||||
be raised for retrying. A mock connect function is used to simulate a real connection
|
||||
passing on the first attempt.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
return True
|
||||
|
||||
conn = BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retryable_exceptions=[],
|
||||
)
|
||||
|
||||
assert conn.state == "open"
|
||||
assert conn.handle is True
|
||||
assert attempts == 1
|
||||
|
||||
def test_retry_connection_fails_unhandled(self):
|
||||
"""Test setting a handle fails upon raising a non-handled exception.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects a ValueError to be raised by a mock connect function. As a
|
||||
result:
|
||||
* The Connection state should be "fail" and the handle None.
|
||||
* The resulting attempt count should be 1 as we are not explicitly configured to handle a
|
||||
ValueError.
|
||||
* retry_connection should raise a FailedToConnectException with the Exception message.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
raise ValueError("Something went horribly wrong")
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
|
||||
):
|
||||
|
||||
BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_limit=1,
|
||||
retry_timeout=lambda attempt: 0,
|
||||
retryable_exceptions=(TypeError,),
|
||||
)
|
||||
|
||||
assert conn.state == "fail"
|
||||
assert conn.handle is None
|
||||
assert attempts == 1
|
||||
|
||||
def test_retry_connection_fails_handled(self):
|
||||
"""Test setting a handle fails upon raising a handled exception.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects a ValueError to be raised by a mock connect function.
|
||||
As a result:
|
||||
* The Connection state should be "fail" and the handle None.
|
||||
* The resulting attempt count should be 2 as we are configured to handle a ValueError.
|
||||
* retry_connection should raise a FailedToConnectException with the Exception message.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
raise ValueError("Something went horribly wrong")
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
|
||||
):
|
||||
|
||||
BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=0,
|
||||
retryable_exceptions=(ValueError,),
|
||||
retry_limit=1,
|
||||
)
|
||||
|
||||
assert conn.state == "fail"
|
||||
assert conn.handle is None
|
||||
|
||||
def test_retry_connection_passes_handled(self):
|
||||
"""Test setting a handle fails upon raising a handled exception.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects a ValueError to be raised by a mock connect function only the first
|
||||
time is called. Upon handling the exception once, connect should return.
|
||||
As a result:
|
||||
* The Connection state should be "open" and the handle True.
|
||||
* The resulting attempt count should be 2 as we are configured to handle a ValueError.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
is_handled = False
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal is_handled
|
||||
nonlocal attempts
|
||||
|
||||
attempts += 1
|
||||
|
||||
if is_handled:
|
||||
return True
|
||||
|
||||
is_handled = True
|
||||
raise ValueError("Something went horribly wrong")
|
||||
|
||||
conn = BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=0,
|
||||
retryable_exceptions=(ValueError,),
|
||||
retry_limit=1,
|
||||
)
|
||||
|
||||
assert conn.state == "open"
|
||||
assert conn.handle is True
|
||||
assert is_handled is True
|
||||
assert attempts == 2
|
||||
|
||||
def test_retry_connection_attempts(self):
|
||||
"""Test setting a handle fails upon raising a handled exception multiple times.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects a ValueError to be raised by a mock connect function. As a result:
|
||||
* The Connection state should be "fail" and the handle None, as connect
|
||||
never returns.
|
||||
* The resulting attempt count should be 11 as we are configured to handle a ValueError.
|
||||
* retry_connection should raise a FailedToConnectException with the Exception message.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
|
||||
raise ValueError("Something went horribly wrong")
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
dbt.exceptions.FailedToConnectException, "Something went horribly wrong"
|
||||
):
|
||||
BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=0,
|
||||
retryable_exceptions=(ValueError,),
|
||||
retry_limit=10,
|
||||
)
|
||||
|
||||
assert conn.state == "fail"
|
||||
assert conn.handle is None
|
||||
assert attempts == 11
|
||||
|
||||
def test_retry_connection_fails_handling_all_exceptions(self):
|
||||
"""Test setting a handle fails after exhausting all attempts.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects a TypeError to be raised by a mock connect function. As a result:
|
||||
* The Connection state should be "fail" and the handle None, as connect
|
||||
never returns.
|
||||
* The resulting attempt count should be 11 as we are configured to handle all Exceptions.
|
||||
* retry_connection should raise a FailedToConnectException with the Exception message.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
|
||||
raise TypeError("An unhandled thing went horribly wrong")
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
dbt.exceptions.FailedToConnectException, "An unhandled thing went horribly wrong"
|
||||
):
|
||||
BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=0,
|
||||
retryable_exceptions=[Exception],
|
||||
retry_limit=15,
|
||||
)
|
||||
|
||||
assert conn.state == "fail"
|
||||
assert conn.handle is None
|
||||
assert attempts == 16
|
||||
|
||||
def test_retry_connection_passes_multiple_handled(self):
|
||||
"""Test setting a handle passes upon handling multiple exceptions.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects a mock connect to raise a ValueError in the first invocation and a
|
||||
TypeError in the second invocation. As a result:
|
||||
* The Connection state should be "open" and the handle True, as connect
|
||||
returns after both exceptions have been handled.
|
||||
* The resulting attempt count should be 3.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
is_value_err_handled = False
|
||||
is_type_err_handled = False
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal is_value_err_handled
|
||||
nonlocal is_type_err_handled
|
||||
nonlocal attempts
|
||||
|
||||
attempts += 1
|
||||
|
||||
if is_value_err_handled and is_type_err_handled:
|
||||
return True
|
||||
elif is_type_err_handled:
|
||||
is_value_err_handled = True
|
||||
raise ValueError("Something went horribly wrong")
|
||||
else:
|
||||
is_type_err_handled = True
|
||||
raise TypeError("An unhandled thing went horribly wrong")
|
||||
|
||||
conn = BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=0,
|
||||
retryable_exceptions=(ValueError, TypeError),
|
||||
retry_limit=2,
|
||||
)
|
||||
|
||||
assert conn.state == "open"
|
||||
assert conn.handle is True
|
||||
assert is_type_err_handled is True
|
||||
assert is_value_err_handled is True
|
||||
assert attempts == 3
|
||||
|
||||
def test_retry_connection_passes_none_excluded(self):
|
||||
"""Test setting a handle passes upon handling multiple exceptions.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects a mock connect to raise a ValueError in the first invocation and a
|
||||
TypeError in the second invocation. As a result:
|
||||
* The Connection state should be "open" and the handle True, as connect
|
||||
returns after both exceptions have been handled.
|
||||
* The resulting attempt count should be 3.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
is_value_err_handled = False
|
||||
is_type_err_handled = False
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal is_value_err_handled
|
||||
nonlocal is_type_err_handled
|
||||
nonlocal attempts
|
||||
|
||||
attempts += 1
|
||||
|
||||
if is_value_err_handled and is_type_err_handled:
|
||||
return True
|
||||
elif is_type_err_handled:
|
||||
is_value_err_handled = True
|
||||
raise ValueError("Something went horribly wrong")
|
||||
else:
|
||||
is_type_err_handled = True
|
||||
raise TypeError("An unhandled thing went horribly wrong")
|
||||
|
||||
conn = BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=0,
|
||||
retryable_exceptions=(ValueError, TypeError),
|
||||
retry_limit=2,
|
||||
)
|
||||
|
||||
assert conn.state == "open"
|
||||
assert conn.handle is True
|
||||
assert is_type_err_handled is True
|
||||
assert is_value_err_handled is True
|
||||
assert attempts == 3
|
||||
|
||||
def test_retry_connection_retry_limit(self):
|
||||
"""Test retry_connection raises an exception with a negative retry limit."""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
return True
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
dbt.exceptions.FailedToConnectException, "retry_limit cannot be negative"
|
||||
):
|
||||
BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=0,
|
||||
retryable_exceptions=(ValueError,),
|
||||
retry_limit=-2,
|
||||
)
|
||||
|
||||
assert conn.state == "fail"
|
||||
assert conn.handle is None
|
||||
assert attempts == 0
|
||||
|
||||
def test_retry_connection_retry_timeout(self):
|
||||
"""Test retry_connection raises an exception with a negative timeout."""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
return True
|
||||
|
||||
for retry_timeout in [-10, -2.5, lambda _: -100, lambda _: -10.1]:
|
||||
with self.assertRaisesRegex(
|
||||
dbt.exceptions.FailedToConnectException,
|
||||
"retry_timeout cannot be negative or return a negative time",
|
||||
):
|
||||
BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=-10,
|
||||
retryable_exceptions=(ValueError,),
|
||||
retry_limit=2,
|
||||
)
|
||||
|
||||
assert conn.state == "init"
|
||||
assert conn.handle is None
|
||||
assert attempts == 0
|
||||
|
||||
def test_retry_connection_exceeds_recursion_limit(self):
|
||||
"""Test retry_connection raises an exception with retries that exceed recursion limit."""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
return True
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
dbt.exceptions.FailedToConnectException,
|
||||
"retry_limit cannot be negative",
|
||||
):
|
||||
BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=2,
|
||||
retryable_exceptions=(ValueError,),
|
||||
retry_limit=sys.getrecursionlimit() + 1,
|
||||
)
|
||||
|
||||
assert conn.state == "fail"
|
||||
assert conn.handle is None
|
||||
assert attempts == 0
|
||||
|
||||
def test_retry_connection_with_exponential_backoff_timeout(self):
|
||||
"""Test retry_connection with an exponential backoff timeout.
|
||||
|
||||
We assert the provided exponential backoff function gets passed the right attempt number
|
||||
and produces the expected timeouts.
|
||||
"""
|
||||
conn = self.postgres_connection
|
||||
attempts = 0
|
||||
timeouts = []
|
||||
|
||||
def connect():
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
|
||||
if attempts < 12:
|
||||
raise ValueError("Keep trying!")
|
||||
return True
|
||||
|
||||
def exp_backoff(n):
|
||||
nonlocal timeouts
|
||||
computed = 2**n
|
||||
# We store the computed values to ensure they match the expected backoff...
|
||||
timeouts.append((n, computed))
|
||||
# but we return 0 as we don't want the test to go on forever.
|
||||
return 0
|
||||
|
||||
conn = BaseConnectionManager.retry_connection(
|
||||
conn,
|
||||
connect,
|
||||
self.logger,
|
||||
retry_timeout=exp_backoff,
|
||||
retryable_exceptions=(ValueError,),
|
||||
retry_limit=12,
|
||||
)
|
||||
|
||||
assert conn.state == "open"
|
||||
assert conn.handle is True
|
||||
assert attempts == 12
|
||||
assert timeouts == [(n, 2**n) for n in range(12)]
|
||||
|
||||
|
||||
class PostgresConnectionManagerTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.credentials = PostgresCredentials(
|
||||
host="localhost",
|
||||
user="test-user",
|
||||
port=1111,
|
||||
password="test-password",
|
||||
database="test-db",
|
||||
schema="test-schema",
|
||||
retries=2,
|
||||
)
|
||||
self.connection = Connection("postgres", None, self.credentials)
|
||||
|
||||
def test_open(self):
|
||||
"""Test opening a Postgres Connection with failures in the first 3 attempts.
|
||||
|
||||
This test uses a Connection populated with test PostgresCredentials values, and
|
||||
expects a mock connect to raise a psycopg2.errors.ConnectionFailuer
|
||||
in the first 3 invocations, after which the mock should return True. As a result:
|
||||
* The Connection state should be "open" and the handle True, as connect
|
||||
returns in the 4th attempt.
|
||||
* The resulting attempt count should be 4.
|
||||
"""
|
||||
conn = self.connection
|
||||
attempt = 0
|
||||
|
||||
def connect(*args, **kwargs):
|
||||
nonlocal attempt
|
||||
attempt += 1
|
||||
|
||||
if attempt <= 2:
|
||||
raise psycopg2.errors.ConnectionFailure("Connection has failed")
|
||||
|
||||
return True
|
||||
|
||||
with mock.patch("psycopg2.connect", wraps=connect) as mock_connect:
|
||||
PostgresConnectionManager.open(conn)
|
||||
|
||||
assert mock_connect.call_count == 3
|
||||
|
||||
assert attempt == 3
|
||||
assert conn.state == "open"
|
||||
assert conn.handle is True
|
||||
@@ -190,9 +190,9 @@ REQUIRED_BASE_KEYS = frozenset(
|
||||
"fromyaml",
|
||||
"toyaml",
|
||||
"set",
|
||||
"try_set",
|
||||
"set_strict",
|
||||
"zip",
|
||||
"try_zip",
|
||||
"zip_strict",
|
||||
"log",
|
||||
"run_started_at",
|
||||
"invocation_id",
|
||||
|
||||
@@ -101,3 +101,25 @@ class TestYamlRendering(unittest.TestCase):
|
||||
dct = renderer.render_data(dct)
|
||||
self.assertEqual(dct, expected)
|
||||
|
||||
def test__metrics(self):
|
||||
context = {
|
||||
"my_time_grains": "[day]"
|
||||
}
|
||||
renderer = SchemaYamlRenderer(context, 'metrics')
|
||||
|
||||
dct = {
|
||||
"name": "my_source",
|
||||
"description": "{{ docs('my_doc') }}",
|
||||
"sql": "select {{ var('my_var') }} from my_table",
|
||||
"time_grains": "{{ my_time_grains }}",
|
||||
}
|
||||
# We expect the sql and description will not be rendered, but
|
||||
# other fields will be
|
||||
expected = {
|
||||
"name": "my_source",
|
||||
"description": "{{ docs('my_doc') }}",
|
||||
"sql": "select {{ var('my_var') }} from my_table",
|
||||
"time_grains": "[day]"
|
||||
}
|
||||
dct = renderer.render_data(dct)
|
||||
self.assertEqual(dct, expected)
|
||||
|
||||
@@ -1 +1 @@
|
||||
version = "1.2.0b1"
|
||||
version = "1.2.1"
|
||||
|
||||
@@ -20,7 +20,7 @@ except ImportError:
|
||||
|
||||
|
||||
package_name = "dbt-tests-adapter"
|
||||
package_version = "1.2.0b1"
|
||||
package_version = "1.2.1"
|
||||
description = """The dbt adapter tests for adapter plugins"""
|
||||
|
||||
this_directory = os.path.abspath(os.path.dirname(__file__))
|
||||
|
||||
@@ -7,8 +7,8 @@ macros__validate_set_sql = """
|
||||
{% macro validate_set() %}
|
||||
{% set set_result = set([1, 2, 2, 3, 'foo', False]) %}
|
||||
{{ log("set_result: " ~ set_result) }}
|
||||
{% set try_set_result = try_set([1, 2, 2, 3, 'foo', False]) %}
|
||||
{{ log("try_set_result: " ~ try_set_result) }}
|
||||
{% set set_strict_result = set_strict([1, 2, 2, 3, 'foo', False]) %}
|
||||
{{ log("set_strict_result: " ~ set_strict_result) }}
|
||||
{% endmacro %}
|
||||
"""
|
||||
|
||||
@@ -18,17 +18,17 @@ macros__validate_zip_sql = """
|
||||
{% set list_b = ['foo', 'bar'] %}
|
||||
{% set zip_result = zip(list_a, list_b) | list %}
|
||||
{{ log("zip_result: " ~ zip_result) }}
|
||||
{% set try_zip_result = try_zip(list_a, list_b) | list %}
|
||||
{{ log("try_zip_result: " ~ try_zip_result) }}
|
||||
{% set zip_strict_result = zip_strict(list_a, list_b) | list %}
|
||||
{{ log("zip_strict_result: " ~ zip_strict_result) }}
|
||||
{% endmacro %}
|
||||
"""
|
||||
|
||||
models__set_exception_sql = """
|
||||
{% set try_set_result = try_set(1) %}
|
||||
{% set set_strict_result = set_strict(1) %}
|
||||
"""
|
||||
|
||||
models__zip_exception_sql = """
|
||||
{% set try_set_result = try_zip(1) %}
|
||||
{% set zip_strict_result = zip_strict(1) %}
|
||||
"""
|
||||
|
||||
|
||||
@@ -46,18 +46,18 @@ class TestContextBuiltins:
|
||||
# The order of the set isn't guaranteed so we can't check for the actual set in the logs
|
||||
assert "set_result: " in log_output
|
||||
assert "False" in log_output
|
||||
assert "try_set_result: " in log_output
|
||||
assert "set_strict_result: " in log_output
|
||||
|
||||
def test_builtin_zip_function(self, project):
|
||||
_, log_output = run_dbt_and_capture(["--debug", "run-operation", "validate_zip"])
|
||||
|
||||
expected_zip = [(1, "foo"), (2, "bar")]
|
||||
assert f"zip_result: {expected_zip}" in log_output
|
||||
assert f"try_zip_result: {expected_zip}" in log_output
|
||||
assert f"zip_strict_result: {expected_zip}" in log_output
|
||||
|
||||
|
||||
class TestContextBuiltinExceptions:
|
||||
# Assert compilation errors are raised with try_ equivalents
|
||||
# Assert compilation errors are raised with _strict equivalents
|
||||
def test_builtin_function_exception(self, project):
|
||||
write_file(models__set_exception_sql, project.project_root, "models", "raise.sql")
|
||||
with pytest.raises(CompilationException):
|
||||
|
||||
Reference in New Issue
Block a user