mirror of
https://github.com/dbt-labs/dbt-core
synced 2025-12-23 16:31:27 +00:00
Compare commits
4 Commits
enable-pos
...
v1.1.0rc2
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
106da05db7 | ||
|
|
09a396a731 | ||
|
|
9c233f27ab | ||
|
|
a09bc28768 |
@@ -1,5 +1,5 @@
|
|||||||
[bumpversion]
|
[bumpversion]
|
||||||
current_version = 1.1.0b1
|
current_version = 1.1.0rc2
|
||||||
parse = (?P<major>\d+)
|
parse = (?P<major>\d+)
|
||||||
\.(?P<minor>\d+)
|
\.(?P<minor>\d+)
|
||||||
\.(?P<patch>\d+)
|
\.(?P<patch>\d+)
|
||||||
|
|||||||
48
.changes/1.1.0-rc1.md
Normal file
48
.changes/1.1.0-rc1.md
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
## dbt-core 1.1.0-rc1 - April 12, 2022
|
||||||
|
### Breaking Changes
|
||||||
|
- For adapter plugin maintainers only: Internal adapter methods `set_relations_cache` + `_relations_cache_for_schemas` each take an additional argument, for use with experimental `CACHE_SELECTED_ONLY` config ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
|
||||||
|
### Features
|
||||||
|
- Add `--cache_selected_only` flag to cache schema object of selected models only. ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
|
||||||
|
- Support custom names for generic tests ([#3348](https://github.com/dbt-labs/dbt-core/issues/3348), [#4898](https://github.com/dbt-labs/dbt-core/pull/4898))
|
||||||
|
- Enable dbt jobs to run downstream models based on fresher sources. Compare the source freshness results between previous and current state. If any source is fresher and/or new in current vs. previous state, dbt will run and test the downstream models in scope. Example command: `dbt build --select source_status:fresher+` ([#4050](https://github.com/dbt-labs/dbt-core/issues/4050), [#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
|
||||||
|
- converting unique key as list tests to new pytest format ([#4882](https://github.com/dbt-labs/dbt-core/issues/4882), [#4958](https://github.com/dbt-labs/dbt-core/pull/4958))
|
||||||
|
- Add a variable called selected_resources in the Jinja context containing a list of all the resources matching the nodes for the --select, --exclude and/or --selector parameters. ([#3471](https://github.com/dbt-labs/dbt-core/issues/3471), [#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
|
||||||
|
- Support the DO_NOT_TRACK environment variable from the consoledonottrack.com initiative ([#3540](https://github.com/dbt-labs/dbt-core/issues/3540), [#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
|
||||||
|
- Add `--no-print` global flag ([#4710](https://github.com/dbt-labs/dbt-core/issues/4710), [#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
|
||||||
|
- add enabled as a source config ([#3662](https://github.com/dbt-labs/dbt-core/issues/3662), [#5008](https://github.com/dbt-labs/dbt-core/pull/5008))
|
||||||
|
### Fixes
|
||||||
|
- Inconsistent timestamps between inserted/updated and deleted rows in snapshots ([#4347](https://github.com/dbt-labs/dbt-core/issues/4347), [#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
|
||||||
|
- Catch more cases to retry package retrieval for deps pointing to the hub. Also start to cache the package requests. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4982](https://github.com/dbt-labs/dbt-core/pull/4982))
|
||||||
|
- Make the warning message for a full event deque more descriptive ([#4962](https://github.com/dbt-labs/dbt-core/issues/4962), [#5011](https://github.com/dbt-labs/dbt-core/pull/5011))
|
||||||
|
- Fix hard delete snapshot test ([#4916](https://github.com/dbt-labs/dbt-core/issues/4916), [#5020](https://github.com/dbt-labs/dbt-core/pull/5020))
|
||||||
|
### Docs
|
||||||
|
- Fixed capitalization in UI for exposures of `type: ml` ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- List packages and tags in alphabetical order ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- Bump jekyll from 3.8.7 to 3.9.0 ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- Updated docker README to reflect necessity of using BuildKit ([#4990](https://github.com/dbt-labs/dbt-core/issues/4990), [#5018](https://github.com/dbt-labs/dbt-core/pull/5018))
|
||||||
|
### Under the Hood
|
||||||
|
- add performance regression testing runner without orchestration ([#4021](https://github.com/dbt-labs/dbt-core/issues/4021), [#4602](https://github.com/dbt-labs/dbt-core/pull/4602))
|
||||||
|
- Add Graph Compilation and Adapter Cache tracking ([#4625](https://github.com/dbt-labs/dbt-core/issues/4625), [#4912](https://github.com/dbt-labs/dbt-core/pull/4912))
|
||||||
|
- Create a dbt.tests.adapter release when releasing dbt and postgres ([#4812](https://github.com/dbt-labs/dbt-core/issues/4812), [#4948](https://github.com/dbt-labs/dbt-core/pull/4948))
|
||||||
|
- update docker image to use python 3.10.3 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4963](https://github.com/dbt-labs/dbt-core/pull/4963))
|
||||||
|
- updates black to 22.3.0 which fixes dependency incompatibility when running with precommit. ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4972](https://github.com/dbt-labs/dbt-core/pull/4972))
|
||||||
|
- Adds config util for ad-hoc creation of project objs or dicts ([#4808](https://github.com/dbt-labs/dbt-core/issues/4808), [#4981](https://github.com/dbt-labs/dbt-core/pull/4981))
|
||||||
|
- Remove TableComparison and convert existing calls to use dbt.tests.util ([#4778](https://github.com/dbt-labs/dbt-core/issues/4778), [#4986](https://github.com/dbt-labs/dbt-core/pull/4986))
|
||||||
|
- Remove unneeded create_schema in snapshot materialization ([#4742](https://github.com/dbt-labs/dbt-core/issues/4742), [#4993](https://github.com/dbt-labs/dbt-core/pull/4993))
|
||||||
|
- Added .git-blame-ignore-revs file to mask re-formmating commits from git blame ([#5004](https://github.com/dbt-labs/dbt-core/issues/5004), [#5019](https://github.com/dbt-labs/dbt-core/pull/5019))
|
||||||
|
- Convert version tests to pytest ([#5024](https://github.com/dbt-labs/dbt-core/issues/5024), [#5026](https://github.com/dbt-labs/dbt-core/pull/5026))
|
||||||
|
- Updating tests and docs to show that we now support Python 3.10 ([#4974](https://github.com/dbt-labs/dbt-core/issues/4974), [#5025](https://github.com/dbt-labs/dbt-core/pull/5025))
|
||||||
|
- Update --version output and logic ([#4724](https://github.com/dbt-labs/dbt-core/issues/4724), [#5029](https://github.com/dbt-labs/dbt-core/pull/5029))
|
||||||
|
- ([#5033](https://github.com/dbt-labs/dbt-core/issues/5033), [#5032](https://github.com/dbt-labs/dbt-core/pull/5032))
|
||||||
|
|
||||||
|
### Contributors
|
||||||
|
- [@agoblet](https://github.com/agoblet) ([#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
|
||||||
|
- [@anaisvaillant](https://github.com/anaisvaillant) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
|
||||||
|
- [@b-per](https://github.com/b-per) ([#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
|
||||||
|
- [@jonstacks](https://github.com/jonstacks) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- [@kadero](https://github.com/kadero) ([#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
|
||||||
|
- [@karunpoudel](https://github.com/karunpoudel) ([#4860](https://github.com/dbt-labs/dbt-core/pull/4860), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
|
||||||
|
- [@matt-winkler](https://github.com/matt-winkler) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
|
||||||
|
- [@pgoslatara](https://github.com/pgoslatara) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- [@poloaraujo](https://github.com/poloaraujo) ([#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
|
||||||
|
- [@sungchun12](https://github.com/sungchun12) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
|
||||||
6
.changes/1.1.0-rc2.md
Normal file
6
.changes/1.1.0-rc2.md
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
## dbt-core 1.1.0-rc2 - April 21, 2022
|
||||||
|
### Fixes
|
||||||
|
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
|
||||||
|
|
||||||
|
### Contributors
|
||||||
|
- [@dbeatty10](https://github.com/dbeatty10) ([#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
|
||||||
@@ -5,6 +5,6 @@ body: 'Enable dbt jobs to run downstream models based on fresher sources. Compar
|
|||||||
models in scope. Example command: `dbt build --select source_status:fresher+` '
|
models in scope. Example command: `dbt build --select source_status:fresher+` '
|
||||||
time: 2022-03-28T13:47:43.750709-05:00
|
time: 2022-03-28T13:47:43.750709-05:00
|
||||||
custom:
|
custom:
|
||||||
Author: sungchun12, matt-winkler, anaisvaillant
|
Author: sungchun12 matt-winkler anaisvaillant
|
||||||
Issue: "4050"
|
Issue: "4050"
|
||||||
PR: "4256"
|
PR: "4256"
|
||||||
7
.changes/1.1.0/Fixes-20220415-112927.yaml
Normal file
7
.changes/1.1.0/Fixes-20220415-112927.yaml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
kind: Fixes
|
||||||
|
body: Restore ability to utilize `updated_at` for check_cols snapshots
|
||||||
|
time: 2022-04-15T11:29:27.063462-06:00
|
||||||
|
custom:
|
||||||
|
Author: dbeatty10
|
||||||
|
Issue: "5076"
|
||||||
|
PR: "5077"
|
||||||
58
CHANGELOG.md
58
CHANGELOG.md
@@ -6,6 +6,64 @@
|
|||||||
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
|
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
|
||||||
|
|
||||||
|
|
||||||
|
## dbt-core 1.1.0-rc2 - April 21, 2022
|
||||||
|
### Fixes
|
||||||
|
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
|
||||||
|
|
||||||
|
### Contributors
|
||||||
|
- [@dbeatty10](https://github.com/dbeatty10) ([#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
|
||||||
|
|
||||||
|
|
||||||
|
## dbt-core 1.1.0-rc1 - April 12, 2022
|
||||||
|
### Breaking Changes
|
||||||
|
- For adapter plugin maintainers only: Internal adapter methods `set_relations_cache` + `_relations_cache_for_schemas` each take an additional argument, for use with experimental `CACHE_SELECTED_ONLY` config ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
|
||||||
|
### Features
|
||||||
|
- Add `--cache_selected_only` flag to cache schema object of selected models only. ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
|
||||||
|
- Support custom names for generic tests ([#3348](https://github.com/dbt-labs/dbt-core/issues/3348), [#4898](https://github.com/dbt-labs/dbt-core/pull/4898))
|
||||||
|
- Enable dbt jobs to run downstream models based on fresher sources. Compare the source freshness results between previous and current state. If any source is fresher and/or new in current vs. previous state, dbt will run and test the downstream models in scope. Example command: `dbt build --select source_status:fresher+` ([#4050](https://github.com/dbt-labs/dbt-core/issues/4050), [#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
|
||||||
|
- converting unique key as list tests to new pytest format ([#4882](https://github.com/dbt-labs/dbt-core/issues/4882), [#4958](https://github.com/dbt-labs/dbt-core/pull/4958))
|
||||||
|
- Add a variable called selected_resources in the Jinja context containing a list of all the resources matching the nodes for the --select, --exclude and/or --selector parameters. ([#3471](https://github.com/dbt-labs/dbt-core/issues/3471), [#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
|
||||||
|
- Support the DO_NOT_TRACK environment variable from the consoledonottrack.com initiative ([#3540](https://github.com/dbt-labs/dbt-core/issues/3540), [#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
|
||||||
|
- Add `--no-print` global flag ([#4710](https://github.com/dbt-labs/dbt-core/issues/4710), [#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
|
||||||
|
- add enabled as a source config ([#3662](https://github.com/dbt-labs/dbt-core/issues/3662), [#5008](https://github.com/dbt-labs/dbt-core/pull/5008))
|
||||||
|
### Fixes
|
||||||
|
- Inconsistent timestamps between inserted/updated and deleted rows in snapshots ([#4347](https://github.com/dbt-labs/dbt-core/issues/4347), [#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
|
||||||
|
- Catch more cases to retry package retrieval for deps pointing to the hub. Also start to cache the package requests. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4982](https://github.com/dbt-labs/dbt-core/pull/4982))
|
||||||
|
- Make the warning message for a full event deque more descriptive ([#4962](https://github.com/dbt-labs/dbt-core/issues/4962), [#5011](https://github.com/dbt-labs/dbt-core/pull/5011))
|
||||||
|
- Fix hard delete snapshot test ([#4916](https://github.com/dbt-labs/dbt-core/issues/4916), [#5020](https://github.com/dbt-labs/dbt-core/pull/5020))
|
||||||
|
### Docs
|
||||||
|
- Fixed capitalization in UI for exposures of `type: ml` ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- List packages and tags in alphabetical order ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- Bump jekyll from 3.8.7 to 3.9.0 ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- Updated docker README to reflect necessity of using BuildKit ([#4990](https://github.com/dbt-labs/dbt-core/issues/4990), [#5018](https://github.com/dbt-labs/dbt-core/pull/5018))
|
||||||
|
### Under the Hood
|
||||||
|
- add performance regression testing runner without orchestration ([#4021](https://github.com/dbt-labs/dbt-core/issues/4021), [#4602](https://github.com/dbt-labs/dbt-core/pull/4602))
|
||||||
|
- Add Graph Compilation and Adapter Cache tracking ([#4625](https://github.com/dbt-labs/dbt-core/issues/4625), [#4912](https://github.com/dbt-labs/dbt-core/pull/4912))
|
||||||
|
- Create a dbt.tests.adapter release when releasing dbt and postgres ([#4812](https://github.com/dbt-labs/dbt-core/issues/4812), [#4948](https://github.com/dbt-labs/dbt-core/pull/4948))
|
||||||
|
- update docker image to use python 3.10.3 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4963](https://github.com/dbt-labs/dbt-core/pull/4963))
|
||||||
|
- updates black to 22.3.0 which fixes dependency incompatibility when running with precommit. ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4972](https://github.com/dbt-labs/dbt-core/pull/4972))
|
||||||
|
- Adds config util for ad-hoc creation of project objs or dicts ([#4808](https://github.com/dbt-labs/dbt-core/issues/4808), [#4981](https://github.com/dbt-labs/dbt-core/pull/4981))
|
||||||
|
- Remove TableComparison and convert existing calls to use dbt.tests.util ([#4778](https://github.com/dbt-labs/dbt-core/issues/4778), [#4986](https://github.com/dbt-labs/dbt-core/pull/4986))
|
||||||
|
- Remove unneeded create_schema in snapshot materialization ([#4742](https://github.com/dbt-labs/dbt-core/issues/4742), [#4993](https://github.com/dbt-labs/dbt-core/pull/4993))
|
||||||
|
- Added .git-blame-ignore-revs file to mask re-formmating commits from git blame ([#5004](https://github.com/dbt-labs/dbt-core/issues/5004), [#5019](https://github.com/dbt-labs/dbt-core/pull/5019))
|
||||||
|
- Convert version tests to pytest ([#5024](https://github.com/dbt-labs/dbt-core/issues/5024), [#5026](https://github.com/dbt-labs/dbt-core/pull/5026))
|
||||||
|
- Updating tests and docs to show that we now support Python 3.10 ([#4974](https://github.com/dbt-labs/dbt-core/issues/4974), [#5025](https://github.com/dbt-labs/dbt-core/pull/5025))
|
||||||
|
- Update --version output and logic ([#4724](https://github.com/dbt-labs/dbt-core/issues/4724), [#5029](https://github.com/dbt-labs/dbt-core/pull/5029))
|
||||||
|
- ([#5033](https://github.com/dbt-labs/dbt-core/issues/5033), [#5032](https://github.com/dbt-labs/dbt-core/pull/5032))
|
||||||
|
|
||||||
|
### Contributors
|
||||||
|
- [@agoblet](https://github.com/agoblet) ([#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
|
||||||
|
- [@anaisvaillant](https://github.com/anaisvaillant) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
|
||||||
|
- [@b-per](https://github.com/b-per) ([#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
|
||||||
|
- [@jonstacks](https://github.com/jonstacks) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- [@kadero](https://github.com/kadero) ([#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
|
||||||
|
- [@karunpoudel](https://github.com/karunpoudel) ([#4860](https://github.com/dbt-labs/dbt-core/pull/4860), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
|
||||||
|
- [@matt-winkler](https://github.com/matt-winkler) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
|
||||||
|
- [@pgoslatara](https://github.com/pgoslatara) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
|
||||||
|
- [@poloaraujo](https://github.com/poloaraujo) ([#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
|
||||||
|
- [@sungchun12](https://github.com/sungchun12) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
|
||||||
|
|
||||||
|
|
||||||
## dbt-core 1.1.0-b1 - March 22, 2022
|
## dbt-core 1.1.0-b1 - March 22, 2022
|
||||||
### Breaking Changes
|
### Breaking Changes
|
||||||
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter ([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
|
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter ([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
|
||||||
|
|||||||
@@ -132,7 +132,7 @@
|
|||||||
{% set check_cols_config = config['check_cols'] %}
|
{% set check_cols_config = config['check_cols'] %}
|
||||||
{% set primary_key = config['unique_key'] %}
|
{% set primary_key = config['unique_key'] %}
|
||||||
{% set invalidate_hard_deletes = config.get('invalidate_hard_deletes', false) %}
|
{% set invalidate_hard_deletes = config.get('invalidate_hard_deletes', false) %}
|
||||||
{% set updated_at = snapshot_get_time() %}
|
{% set updated_at = config.get('updated_at', snapshot_get_time()) %}
|
||||||
|
|
||||||
{% set column_added = false %}
|
{% set column_added = false %}
|
||||||
|
|
||||||
|
|||||||
@@ -273,13 +273,15 @@ def check_relation_types(adapter, relation_to_type):
|
|||||||
# by doing a separate call for each set of tables/relations.
|
# by doing a separate call for each set of tables/relations.
|
||||||
# Wraps check_relations_equal_with_relations by creating relations
|
# Wraps check_relations_equal_with_relations by creating relations
|
||||||
# from the list of names passed in.
|
# from the list of names passed in.
|
||||||
def check_relations_equal(adapter, relation_names):
|
def check_relations_equal(adapter, relation_names, compare_snapshot_cols=False):
|
||||||
if len(relation_names) < 2:
|
if len(relation_names) < 2:
|
||||||
raise TestProcessingException(
|
raise TestProcessingException(
|
||||||
"Not enough relations to compare",
|
"Not enough relations to compare",
|
||||||
)
|
)
|
||||||
relations = [relation_from_name(adapter, name) for name in relation_names]
|
relations = [relation_from_name(adapter, name) for name in relation_names]
|
||||||
return check_relations_equal_with_relations(adapter, relations)
|
return check_relations_equal_with_relations(
|
||||||
|
adapter, relations, compare_snapshot_cols=compare_snapshot_cols
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# This can be used when checking relations in different schemas, by supplying
|
# This can be used when checking relations in different schemas, by supplying
|
||||||
@@ -288,16 +290,17 @@ def check_relations_equal(adapter, relation_names):
|
|||||||
# adapter.get_columns_in_relation
|
# adapter.get_columns_in_relation
|
||||||
# adapter.get_rows_different_sql
|
# adapter.get_rows_different_sql
|
||||||
# adapter.execute
|
# adapter.execute
|
||||||
def check_relations_equal_with_relations(adapter, relations):
|
def check_relations_equal_with_relations(adapter, relations, compare_snapshot_cols=False):
|
||||||
|
|
||||||
with get_connection(adapter):
|
with get_connection(adapter):
|
||||||
basis, compares = relations[0], relations[1:]
|
basis, compares = relations[0], relations[1:]
|
||||||
# Skip columns starting with "dbt_" because we don't want to
|
# Skip columns starting with "dbt_" because we don't want to
|
||||||
# compare those, since they are time sensitive
|
# compare those, since they are time sensitive
|
||||||
|
# (unless comparing "dbt_" snapshot columns is explicitly enabled)
|
||||||
column_names = [
|
column_names = [
|
||||||
c.name
|
c.name
|
||||||
for c in adapter.get_columns_in_relation(basis)
|
for c in adapter.get_columns_in_relation(basis)
|
||||||
if not c.name.lower().startswith("dbt_")
|
if not c.name.lower().startswith("dbt_") or compare_snapshot_cols
|
||||||
]
|
]
|
||||||
|
|
||||||
for relation in compares:
|
for relation in compares:
|
||||||
|
|||||||
@@ -234,5 +234,5 @@ def _get_adapter_plugin_names() -> Iterator[str]:
|
|||||||
yield plugin_name
|
yield plugin_name
|
||||||
|
|
||||||
|
|
||||||
__version__ = "1.1.0b1"
|
__version__ = "1.1.0rc2"
|
||||||
installed = get_installed_version()
|
installed = get_installed_version()
|
||||||
|
|||||||
@@ -273,12 +273,12 @@ def parse_args(argv=None):
|
|||||||
parser.add_argument("adapter")
|
parser.add_argument("adapter")
|
||||||
parser.add_argument("--title-case", "-t", default=None)
|
parser.add_argument("--title-case", "-t", default=None)
|
||||||
parser.add_argument("--dependency", action="append")
|
parser.add_argument("--dependency", action="append")
|
||||||
parser.add_argument("--dbt-core-version", default="1.1.0b1")
|
parser.add_argument("--dbt-core-version", default="1.1.0rc2")
|
||||||
parser.add_argument("--email")
|
parser.add_argument("--email")
|
||||||
parser.add_argument("--author")
|
parser.add_argument("--author")
|
||||||
parser.add_argument("--url")
|
parser.add_argument("--url")
|
||||||
parser.add_argument("--sql", action="store_true")
|
parser.add_argument("--sql", action="store_true")
|
||||||
parser.add_argument("--package-version", default="1.1.0b1")
|
parser.add_argument("--package-version", default="1.1.0rc2")
|
||||||
parser.add_argument("--project-version", default="1.0")
|
parser.add_argument("--project-version", default="1.0")
|
||||||
parser.add_argument("--no-dependency", action="store_false", dest="set_dependency")
|
parser.add_argument("--no-dependency", action="store_false", dest="set_dependency")
|
||||||
parsed = parser.parse_args()
|
parsed = parser.parse_args()
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ with open(os.path.join(this_directory, "README.md")) as f:
|
|||||||
|
|
||||||
|
|
||||||
package_name = "dbt-core"
|
package_name = "dbt-core"
|
||||||
package_version = "1.1.0b1"
|
package_version = "1.1.0rc2"
|
||||||
description = """With dbt, data analysts and engineers can build analytics \
|
description = """With dbt, data analysts and engineers can build analytics \
|
||||||
the way engineers build applications."""
|
the way engineers build applications."""
|
||||||
|
|
||||||
|
|||||||
@@ -14,12 +14,12 @@ FROM --platform=$build_for python:3.10.3-slim-bullseye as base
|
|||||||
# N.B. The refs updated automagically every release via bumpversion
|
# N.B. The refs updated automagically every release via bumpversion
|
||||||
# N.B. dbt-postgres is currently found in the core codebase so a value of dbt-core@<some_version> is correct
|
# N.B. dbt-postgres is currently found in the core codebase so a value of dbt-core@<some_version> is correct
|
||||||
|
|
||||||
ARG dbt_core_ref=dbt-core@v1.1.0b1
|
ARG dbt_core_ref=dbt-core@v1.1.0rc2
|
||||||
ARG dbt_postgres_ref=dbt-core@v1.1.0b1
|
ARG dbt_postgres_ref=dbt-core@v1.1.0rc2
|
||||||
ARG dbt_redshift_ref=dbt-redshift@v1.0.0
|
ARG dbt_redshift_ref=dbt-redshift@v1.1.0rc2
|
||||||
ARG dbt_bigquery_ref=dbt-bigquery@v1.0.0
|
ARG dbt_bigquery_ref=dbt-bigquery@v1.1.0rc2
|
||||||
ARG dbt_snowflake_ref=dbt-snowflake@v1.0.0
|
ARG dbt_snowflake_ref=dbt-snowflake@v1.1.0rc2
|
||||||
ARG dbt_spark_ref=dbt-spark@v1.0.0
|
ARG dbt_spark_ref=dbt-spark@v1.1.0rc2
|
||||||
# special case args
|
# special case args
|
||||||
ARG dbt_spark_version=all
|
ARG dbt_spark_version=all
|
||||||
ARG dbt_third_party
|
ARG dbt_third_party
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
version = "1.1.0b1"
|
version = "1.1.0rc2"
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ def _dbt_psycopg2_name():
|
|||||||
|
|
||||||
|
|
||||||
package_name = "dbt-postgres"
|
package_name = "dbt-postgres"
|
||||||
package_version = "1.1.0b1"
|
package_version = "1.1.0rc2"
|
||||||
description = """The postgres adpter plugin for dbt (data build tool)"""
|
description = """The postgres adpter plugin for dbt (data build tool)"""
|
||||||
|
|
||||||
this_directory = os.path.abspath(os.path.dirname(__file__))
|
this_directory = os.path.abspath(os.path.dirname(__file__))
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
version = "1.1.0b1"
|
version = "1.1.0rc2"
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ except ImportError:
|
|||||||
|
|
||||||
|
|
||||||
package_name = "dbt-tests-adapter"
|
package_name = "dbt-tests-adapter"
|
||||||
package_version = "1.1.0b1"
|
package_version = "1.1.0rc2"
|
||||||
description = """The dbt adapter tests for adapter plugins"""
|
description = """The dbt adapter tests for adapter plugins"""
|
||||||
|
|
||||||
this_directory = os.path.abspath(os.path.dirname(__file__))
|
this_directory = os.path.abspath(os.path.dirname(__file__))
|
||||||
|
|||||||
@@ -0,0 +1,113 @@
|
|||||||
|
import pytest
|
||||||
|
from dbt.tests.util import run_dbt, check_relations_equal
|
||||||
|
|
||||||
|
snapshot_sql = """
|
||||||
|
{% snapshot snapshot_check_cols_updated_at_actual %}
|
||||||
|
{{
|
||||||
|
config(
|
||||||
|
target_database=database,
|
||||||
|
target_schema=schema,
|
||||||
|
unique_key='id',
|
||||||
|
strategy='check',
|
||||||
|
check_cols='all',
|
||||||
|
updated_at="'" ~ var("updated_at") ~ "'::timestamp",
|
||||||
|
)
|
||||||
|
}}
|
||||||
|
|
||||||
|
{% if var('version') == 1 %}
|
||||||
|
|
||||||
|
select 'a' as id, 10 as counter, '2016-01-01T00:00:00Z'::timestamp as timestamp_col union all
|
||||||
|
select 'b' as id, 20 as counter, '2016-01-01T00:00:00Z'::timestamp as timestamp_col
|
||||||
|
|
||||||
|
{% elif var('version') == 2 %}
|
||||||
|
|
||||||
|
select 'a' as id, 30 as counter, '2016-01-02T00:00:00Z'::timestamp as timestamp_col union all
|
||||||
|
select 'b' as id, 20 as counter, '2016-01-01T00:00:00Z'::timestamp as timestamp_col union all
|
||||||
|
select 'c' as id, 40 as counter, '2016-01-02T00:00:00Z'::timestamp as timestamp_col
|
||||||
|
|
||||||
|
{% else %}
|
||||||
|
|
||||||
|
select 'a' as id, 30 as counter, '2016-01-02T00:00:00Z'::timestamp as timestamp_col union all
|
||||||
|
select 'c' as id, 40 as counter, '2016-01-02T00:00:00Z'::timestamp as timestamp_col
|
||||||
|
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% endsnapshot %}
|
||||||
|
"""
|
||||||
|
|
||||||
|
expected_csv = """
|
||||||
|
id,counter,timestamp_col,dbt_scd_id,dbt_updated_at,dbt_valid_from,dbt_valid_to
|
||||||
|
a,10,2016-01-01 00:00:00.000,927354aa091feffd9437ead0bdae7ae1,2016-07-01 00:00:00.000,2016-07-01 00:00:00.000,2016-07-02 00:00:00.000
|
||||||
|
b,20,2016-01-01 00:00:00.000,40ace4cbf8629f1720ec8a529ed76f8c,2016-07-01 00:00:00.000,2016-07-01 00:00:00.000,
|
||||||
|
a,30,2016-01-02 00:00:00.000,e9133f2b302c50e36f43e770944cec9b,2016-07-02 00:00:00.000,2016-07-02 00:00:00.000,
|
||||||
|
c,40,2016-01-02 00:00:00.000,09d33d35101e788c152f65d0530b6837,2016-07-02 00:00:00.000,2016-07-02 00:00:00.000,
|
||||||
|
""".lstrip()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="class")
|
||||||
|
def snapshots():
|
||||||
|
return {"snapshot_check_cols_updated_at_actual.sql": snapshot_sql}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="class")
|
||||||
|
def seeds():
|
||||||
|
return {"snapshot_check_cols_updated_at_expected.csv": expected_csv}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="class")
|
||||||
|
def project_config_update():
|
||||||
|
return {
|
||||||
|
"seeds": {
|
||||||
|
"quote_columns": False,
|
||||||
|
"test": {
|
||||||
|
"snapshot_check_cols_updated_at_expected": {
|
||||||
|
"+column_types": {
|
||||||
|
"timestamp_col": "timestamp without time zone",
|
||||||
|
"dbt_updated_at": "timestamp without time zone",
|
||||||
|
"dbt_valid_from": "timestamp without time zone",
|
||||||
|
"dbt_valid_to": "timestamp without time zone",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_simple_snapshot(project):
|
||||||
|
"""
|
||||||
|
Test that the `dbt_updated_at` column reflects the `updated_at` timestamp expression in the config.
|
||||||
|
|
||||||
|
Approach:
|
||||||
|
1. Create a table that represents the expected data after a series of snapshots
|
||||||
|
- Use dbt seed to create the expected relation (`snapshot_check_cols_updated_at_expected`)
|
||||||
|
2. Execute a series of snapshots to create the data
|
||||||
|
- Use a series of (3) dbt snapshot commands to create the actual relation (`snapshot_check_cols_updated_at_actual`)
|
||||||
|
- The logic can switch between 3 different versions of the data (depending on the `version` number)
|
||||||
|
- The `updated_at` value is passed in via `--vars` and cast to a timestamp in the snapshot config
|
||||||
|
3. Compare the two relations for equality
|
||||||
|
"""
|
||||||
|
|
||||||
|
# 1. Create a table that represents the expected data after a series of snapshots
|
||||||
|
results = run_dbt(["seed", "--show", "--vars", "{version: 1, updated_at: 2016-07-01}"])
|
||||||
|
assert len(results) == 1
|
||||||
|
|
||||||
|
# 2. Execute a series of snapshots to create the data
|
||||||
|
|
||||||
|
# Snapshot day 1
|
||||||
|
results = run_dbt(["snapshot", "--vars", "{version: 1, updated_at: 2016-07-01}"])
|
||||||
|
assert len(results) == 1
|
||||||
|
|
||||||
|
# Snapshot day 2
|
||||||
|
results = run_dbt(["snapshot", "--vars", "{version: 2, updated_at: 2016-07-02}"])
|
||||||
|
assert len(results) == 1
|
||||||
|
|
||||||
|
# Snapshot day 3
|
||||||
|
results = run_dbt(["snapshot", "--vars", "{version: 3, updated_at: 2016-07-03}"])
|
||||||
|
assert len(results) == 1
|
||||||
|
|
||||||
|
# 3. Compare the two relations for equality
|
||||||
|
check_relations_equal(
|
||||||
|
project.adapter,
|
||||||
|
["snapshot_check_cols_updated_at_actual", "snapshot_check_cols_updated_at_expected"],
|
||||||
|
compare_snapshot_cols=True,
|
||||||
|
)
|
||||||
Reference in New Issue
Block a user