Compare commits

...

17 Commits

Author SHA1 Message Date
github-actions[bot]
5ef0442654 Bumping version to 1.1.1rc2 (#5342)
* Bumping version to 1.1.1rc2

* Remove whitespace

* Add Changelog

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-06-07 11:28:46 -04:00
github-actions[bot]
4d63fb2ed2 Fixing Windows color regression (#5327) (#5340)
* Fixing Windows color regression

* Cleaning up logic

* Consolidating logic to the logger

* Cleaning up vars

* Updating comment

* Removing unused import

* Fixing whitespace

* Adding changelog

(cherry picked from commit e48f7ab32e)

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-06-07 09:32:28 -04:00
Jeremy Cohen
413a19abe6 Backports: networkx, context cleanup (#5335)
* Update context readme, small code cleanup (#5334)

* Pin networkx dependency to <2.8.4 for v1.1
2022-06-07 11:00:45 +02:00
github-actions[bot]
a3dcc14f87 [Backport 1.1.latest] CT-159 Remove docs file from manifest when deleting doc node (#5293)
* CT-159 Remove docs file from manifest when deleting doc node (#5270)

* Convert partial parsing docs tests

* Failing test

* Remove doc file from manifest when doc node is removed

* Changie

(cherry picked from commit cda88d1948)

* Fix rm_file method

* comment out pip upgrade

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-06-01 10:31:38 -04:00
github-actions[bot]
5e298b1854 Bumping version to 1.1.1rc1 (#5285)
* Bumping version to 1.1.1rc1

* Remove whitespace

* Changelog update

* Fixing adapter versions

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-05-20 13:28:51 -04:00
github-actions[bot]
c74e9dde2b When parsing 'all_sources' should be a list of unique dirs (#5176) (#5256)
* When parsing 'all_sources' should be a list of unique dirs

* Changie

* Fix some unit tests of all_source_paths

* Convert 039_config_tests

* Remove old 039_config_tests

* Add test for duplicate directories in 'all_source_files'

(cherry picked from commit f633e9936f)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-05-16 16:00:53 -04:00
github-actions[bot]
9c6bdcabf8 Bumping hologram version (#5218) (#5226)
* Bumping hologram version

* Add automated changelog yaml from template

* Updating issue

* Loosen requirement range

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
(cherry picked from commit aa8115aa5e)

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-05-11 11:56:43 -04:00
leahwicz
4ae2e41cc6 Flexibilize MarkupSafe pinned version (#5039) (#5111)
* Flexibilize MarkupSafe pinned version

The current `MarkupSafe` pinned version has been added in #4746 as a
temporary fix for #4745.

However, the current restrictive approach isn't compatible with other
libraries that could require an even older version of `MarkupSafe`, like
Airflow `2.2.2` [0], which requires `markupsafe>=1.1.1, <2.0`.

To avoid that issue, we can allow a greater range of supported
`MarkupSafe` versions. Considering the direct dependency `dbt-core` has
is `Jinja2==2.11.3`, we can use its pinning as the lower bound, which is
`MarkupSafe>=0.23` [1].

This fix should be also backported this to `1.0.latest` for inclusion in
the next v1.0 patch.

[0] https://github.com/adamantike/airflow/blob/2.2.2/setup.cfg#L125
[1] https://github.com/pallets/jinja/blob/2.11.3/setup.py#L53

Co-authored-by: Michael Manganiello <adamantike@users.noreply.github.com>
2022-05-09 14:49:56 -04:00
github-actions[bot]
cbd4570c67 Bumping version to 1.1.0 (#5186)
* Bumping version to 1.1.0

* Updating changelog

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-04-28 12:54:30 -04:00
github-actions[bot]
5ae33b90bf Bumping version to 1.1.0rc3 (#5166)
* Bumping version to 1.1.0rc3

* Update changelog

* Fixing spacing issue

* Removing more whitespace

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-04-26 15:25:39 -04:00
github-actions[bot]
37d78338c2 [Backport 1.1.latest] Use yaml renderer (with target context) for rendering selectors (#5161)
* Use yaml renderer (with target context) for rendering selectors (#5136)

* Use yaml renderer (with target context) for rendering selectors

* Changie

* Convert cli_vars tests

* Add test for var in profiles

* Add test for cli vars in packages

* Add test for vars in selectors

(cherry picked from commit 1f898c859a)

* Tweak cli_vars_in_packages test to do a run and build an adapter

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-04-26 14:13:16 -04:00
github-actions[bot]
698420b420 Even more scrubbing (#5152) (#5156)
* Even more scrubbing

* Changelog entry

* Even more

* remove reduendent scrub

* remove reduendent scrub

* fix encoding issue

* keep scrubbed log in args

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
(cherry picked from commit ce0bcc08a6)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2022-04-26 10:59:07 -06:00
Emily Rockman
e23f0e0747 fix retry logic failures (#5137) (#5148)
* fix retry logic failures

* changelog

* add tests to make sure data is getting where it needs to

* rename file

* remove duplicate file

(cherry picked from commit 31a3f2bdee)
2022-04-25 10:45:35 -05:00
github-actions[bot]
106da05db7 Bumping version to 1.1.0rc2 (#5129)
* Bumping version to 1.1.0rc2

* release changelog

* fix trailing space

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2022-04-21 09:52:08 -06:00
Chenyu Li
09a396a731 Restore ability to utilize updated_at for check_cols snapshots (#5077) (#5126)
* Restore ability to configure and utilize `updated_at` for snapshots using the check_cols strategy

* Changelog entry

* Optional comparison of column names starting with `dbt_`

* Functional test for check cols snapshots using `updated_at`

* Comments to explain the test implementation

(cherry picked from commit d09459c980)

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2022-04-21 09:06:04 -06:00
Emily Rockman
9c233f27ab remove commas from changelog yaml (#5113) 2022-04-19 19:57:14 -05:00
github-actions[bot]
a09bc28768 Bumping version to 1.1.0rc1 (#5044)
* Bumping version to 1.1.0rc1

* Adding changelog

* Updating adapter versions for Docker

* Remove extra spaces

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-04-12 17:10:57 -04:00
148 changed files with 1849 additions and 1624 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.1.0b1
current_version = 1.1.1rc2
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)

View File

@@ -1,50 +0,0 @@
## dbt-core 1.1.0-b1 - March 22, 2022
### Breaking Changes
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter ([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
### Features
- Change behaviour of `non_null` test so that it only `select`s all columns if `--store-failures` is enabled. ([#4769](https://github.com/dbt-labs/dbt-core/issues/4769), [#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
- Testing framework for dbt adapter testing ([#4730](https://github.com/dbt-labs/dbt-core/issues/4730), [#4846](https://github.com/dbt-labs/dbt-core/pull/4846))
- Allow unique key to take a list implementation for postgres/redshift ([#4738](https://github.com/dbt-labs/dbt-core/issues/4738), [#4858](https://github.com/dbt-labs/dbt-core/pull/4858))
- Added Support for Semantic Versioning ([#4453](https://github.com/dbt-labs/dbt-core/issues/4453), [#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- New Dockerfile to support specific db adapters and platforms. See docker/README.md for details ([#4495](https://github.com/dbt-labs/dbt-core/issues/4495), [#4487](https://github.com/dbt-labs/dbt-core/pull/4487))
- Allow unique_key to take a list ([#2479](https://github.com/dbt-labs/dbt-core/issues/2479), [#4618](https://github.com/dbt-labs/dbt-core/pull/4618))
- Add `--quiet` global flag and `print` Jinja function ([#3451](https://github.com/dbt-labs/dbt-core/issues/3451), [#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- Add space before justification periods ([#4737](https://github.com/dbt-labs/dbt-core/issues/4737), [#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
### Fixes
- Fix bug causing empty node level meta, snapshot config errors ([#4459](https://github.com/dbt-labs/dbt-core/issues/4459), [#4726](https://github.com/dbt-labs/dbt-core/pull/4726))
- Catch all Requests Exceptions on deps install to attempt retries. Also log the exceptions hit. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4865](https://github.com/dbt-labs/dbt-core/pull/4865))
- Use cli_vars instead of context to create package and selector renderers ([#4876](https://github.com/dbt-labs/dbt-core/issues/4876), [#4878](https://github.com/dbt-labs/dbt-core/pull/4878))
- depend on new dbt-extractor version with fixed github links ([#4891](https://github.com/dbt-labs/dbt-core/issues/4891), [#4890](https://github.com/dbt-labs/dbt-core/pull/4890))
- Update bumpervsion config to stop looking for missing setup.py ([#-1](https://github.com/dbt-labs/dbt-core/issues/-1), [#4896](https://github.com/dbt-labs/dbt-core/pull/4896))
- Corrected permissions settings for docker release workflow ([#4902](https://github.com/dbt-labs/dbt-core/issues/4902), [#4903](https://github.com/dbt-labs/dbt-core/pull/4903))
- User wasn't asked for permission to overwite a profile entry when running init inside an existing project ([#4375](https://github.com/dbt-labs/dbt-core/issues/4375), [#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- Add project name validation to `dbt init` ([#4490](https://github.com/dbt-labs/dbt-core/issues/4490), [#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- Allow override of string and numeric types for adapters. ([#4603](https://github.com/dbt-labs/dbt-core/issues/4603), [#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- A change in secret environment variables won't trigger a full reparse ([#4650](https://github.com/dbt-labs/dbt-core/issues/4650), [#4665](https://github.com/dbt-labs/dbt-core/pull/4665))
- Fix misspellings and typos in docstrings ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
### Under the Hood
- Automate changelog generation with changie ([#4652](https://github.com/dbt-labs/dbt-core/issues/4652), [#4743](https://github.com/dbt-labs/dbt-core/pull/4743))
- Fix broken links for changelog generation and tweak GHA to only post a comment once when changelog entry is missing. ([#4848](https://github.com/dbt-labs/dbt-core/issues/4848), [#4857](https://github.com/dbt-labs/dbt-core/pull/4857))
- Add support for Python 3.10 ([#4562](https://github.com/dbt-labs/dbt-core/issues/4562), [#4866](https://github.com/dbt-labs/dbt-core/pull/4866))
- Enable more dialects to snapshot sources with added columns, even those that don't support boolean datatypes ([#4488](https://github.com/dbt-labs/dbt-core/issues/4488), [#4871](https://github.com/dbt-labs/dbt-core/pull/4871))
- Testing cleanup ([#3648](https://github.com/dbt-labs/dbt-core/issues/3648), [#4509](https://github.com/dbt-labs/dbt-core/pull/4509))
- Clean up test deprecation warnings ([#3988](https://github.com/dbt-labs/dbt-core/issues/3988), [#4556](https://github.com/dbt-labs/dbt-core/pull/4556))
- Use mashumaro for serialization in event logging ([#4504](https://github.com/dbt-labs/dbt-core/issues/4504), [#4505](https://github.com/dbt-labs/dbt-core/pull/4505))
- Drop support for Python 3.7.0 + 3.7.1 ([#4584](https://github.com/dbt-labs/dbt-core/issues/4584), [#4585](https://github.com/dbt-labs/dbt-core/pull/4585))
- Drop support for Python <3.7.2 ([#4584](https://github.com/dbt-labs/dbt-core/issues/4584), [#4643](https://github.com/dbt-labs/dbt-core/pull/4643))
- Re-format codebase (except tests) using pre-commit hooks ([#3195](https://github.com/dbt-labs/dbt-core/issues/3195), [#4697](https://github.com/dbt-labs/dbt-core/pull/4697))
- Add deps module README ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4686](https://github.com/dbt-labs/dbt-core/pull/4686))
- Initial conversion of tests to pytest ([#4690](https://github.com/dbt-labs/dbt-core/issues/4690), [#4691](https://github.com/dbt-labs/dbt-core/pull/4691))
- Fix errors in Windows for tests/functions ([#4782](https://github.com/dbt-labs/dbt-core/issues/4782), [#4767](https://github.com/dbt-labs/dbt-core/pull/4767))
### Docs
- Resolve errors related to operations preventing DAG from generating in the docs. Also patch a spark issue to allow search to filter accurately past the missing columns. ([#4578](https://github.com/dbt-labs/dbt-core/issues/4578), [#4763](https://github.com/dbt-labs/dbt-core/pull/4763))
### Contributors
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- [@alswang18](https://github.com/alswang18) ([#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- [@amirkdv](https://github.com/amirkdv) ([#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- [@ehmartens](https://github.com/ehmartens) ([#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- [@joellabes](https://github.com/joellabes) ([#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- [@kazanzhy](https://github.com/kazanzhy) ([#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- [@mdesmet](https://github.com/mdesmet) ([#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- [@versusfacit](https://github.com/versusfacit) ([#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- [@willbowditch](https://github.com/willbowditch) ([#4777](https://github.com/dbt-labs/dbt-core/pull/4777))

96
.changes/1.1.0.md Normal file
View File

@@ -0,0 +1,96 @@
## dbt-core 1.1.0 - April 28, 2022
### Breaking Changes
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter
([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- For adapter plugin maintainers only: Internal adapter methods `set_relations_cache` + `_relations_cache_for_schemas` each take an additional argument, for use with experimental `CACHE_SELECTED_ONLY` config ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
### Features
- Change behaviour of `non_null` test so that it only `select`s all columns if `--store-failures` is enabled. ([#4769](https://github.com/dbt-labs/dbt-core/issues/4769), [#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
- Testing framework for dbt adapter testing ([#4730](https://github.com/dbt-labs/dbt-core/issues/4730), [#4846](https://github.com/dbt-labs/dbt-core/pull/4846))
- Allow unique key to take a list implementation for postgres/redshift ([#4738](https://github.com/dbt-labs/dbt-core/issues/4738), [#4858](https://github.com/dbt-labs/dbt-core/pull/4858))
- Add `--cache_selected_only` flag to cache schema object of selected models only. ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
- Support custom names for generic tests ([#3348](https://github.com/dbt-labs/dbt-core/issues/3348), [#4898](https://github.com/dbt-labs/dbt-core/pull/4898))
- Added Support for Semantic Versioning ([#4453](https://github.com/dbt-labs/dbt-core/issues/4453), [#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- New Dockerfile to support specific db adapters and platforms. See docker/README.md for details ([#4495](https://github.com/dbt-labs/dbt-core/issues/4495), [#4487](https://github.com/dbt-labs/dbt-core/pull/4487))
- Allow unique_key to take a list ([#2479](https://github.com/dbt-labs/dbt-core/issues/2479), [#4618](https://github.com/dbt-labs/dbt-core/pull/4618))
- Add `--quiet` global flag and `print` Jinja function ([#3451](https://github.com/dbt-labs/dbt-core/issues/3451), [#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- Add space before justification periods ([#4737](https://github.com/dbt-labs/dbt-core/issues/4737), [#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- Enable dbt jobs to run downstream models based on fresher sources. Compare the source freshness results between previous and current state. If any source is fresher and/or new in current vs. previous state, dbt will run and test the downstream models in scope. Example command: `dbt build --select source_status:fresher+` ([#4050](https://github.com/dbt-labs/dbt-core/issues/4050), [#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- converting unique key as list tests to new pytest format ([#4882](https://github.com/dbt-labs/dbt-core/issues/4882), [#4958](https://github.com/dbt-labs/dbt-core/pull/4958))
- Add a variable called selected_resources in the Jinja context containing a list of all the resources matching the nodes for the --select, --exclude and/or --selector parameters. ([#3471](https://github.com/dbt-labs/dbt-core/issues/3471), [#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
- Support the DO_NOT_TRACK environment variable from the consoledonottrack.com initiative ([#3540](https://github.com/dbt-labs/dbt-core/issues/3540), [#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
- Add `--no-print` global flag ([#4710](https://github.com/dbt-labs/dbt-core/issues/4710), [#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
- add enabled as a source config ([#3662](https://github.com/dbt-labs/dbt-core/issues/3662), [#5008](https://github.com/dbt-labs/dbt-core/pull/5008))
### Fixes
- Fix bug causing empty node level meta, snapshot config errors ([#4459](https://github.com/dbt-labs/dbt-core/issues/4459), [#4726](https://github.com/dbt-labs/dbt-core/pull/4726))
- Inconsistent timestamps between inserted/updated and deleted rows in snapshots ([#4347](https://github.com/dbt-labs/dbt-core/issues/4347), [#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
- Catch all Requests Exceptions on deps install to attempt retries. Also log the exceptions hit. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4865](https://github.com/dbt-labs/dbt-core/pull/4865))
- Use cli_vars instead of context to create package and selector renderers ([#4876](https://github.com/dbt-labs/dbt-core/issues/4876), [#4878](https://github.com/dbt-labs/dbt-core/pull/4878))
- depend on new dbt-extractor version with fixed github links ([#4891](https://github.com/dbt-labs/dbt-core/issues/4891), [#4890](https://github.com/dbt-labs/dbt-core/pull/4890))
- Update bumpervsion config to stop looking for missing setup.py ([#-1](https://github.com/dbt-labs/dbt-core/issues/-1), [#4896](https://github.com/dbt-labs/dbt-core/pull/4896))
- Corrected permissions settings for docker release workflow ([#4902](https://github.com/dbt-labs/dbt-core/issues/4902), [#4903](https://github.com/dbt-labs/dbt-core/pull/4903))
- User wasn't asked for permission to overwite a profile entry when running init inside an existing project ([#4375](https://github.com/dbt-labs/dbt-core/issues/4375), [#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- Add project name validation to `dbt init` ([#4490](https://github.com/dbt-labs/dbt-core/issues/4490), [#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- Allow override of string and numeric types for adapters. ([#4603](https://github.com/dbt-labs/dbt-core/issues/4603), [#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- A change in secret environment variables won't trigger a full reparse ([#4650](https://github.com/dbt-labs/dbt-core/issues/4650), [#4665](https://github.com/dbt-labs/dbt-core/pull/4665))
- Fix misspellings and typos in docstrings ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- Catch more cases to retry package retrieval for deps pointing to the hub. Also start to cache the package requests. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4982](https://github.com/dbt-labs/dbt-core/pull/4982))
- Make the warning message for a full event deque more descriptive ([#4962](https://github.com/dbt-labs/dbt-core/issues/4962), [#5011](https://github.com/dbt-labs/dbt-core/pull/5011))
- Fix hard delete snapshot test ([#4916](https://github.com/dbt-labs/dbt-core/issues/4916), [#5020](https://github.com/dbt-labs/dbt-core/pull/5020))
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- Use yaml renderer (with target context) for rendering selectors ([#5131](https://github.com/dbt-labs/dbt-core/issues/5131), [#5136](https://github.com/dbt-labs/dbt-core/pull/5136))
- Fix retry logic to return values after initial try ([#5023](https://github.com/dbt-labs/dbt-core/issues/5023), [#5137](https://github.com/dbt-labs/dbt-core/pull/5137))
- Scrub secret env vars from CommandError in exception stacktrace ([#5151](https://github.com/dbt-labs/dbt-core/issues/5151), [#5152](https://github.com/dbt-labs/dbt-core/pull/5152))
### Docs
- Resolve errors related to operations preventing DAG from generating in the docs. Also patch a spark issue to allow search to filter accurately past the missing columns. ([#4578](https://github.com/dbt-labs/dbt-core/issues/4578), [#4763](https://github.com/dbt-labs/dbt-core/pull/4763))
- Fixed capitalization in UI for exposures of `type: ml` ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- List packages and tags in alphabetical order ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- Bump jekyll from 3.8.7 to 3.9.0 ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- Updated docker README to reflect necessity of using BuildKit ([#4990](https://github.com/dbt-labs/dbt-core/issues/4990), [#5018](https://github.com/dbt-labs/dbt-core/pull/5018))
### Under the Hood
- Automate changelog generation with changie ([#4652](https://github.com/dbt-labs/dbt-core/issues/4652), [#4743](https://github.com/dbt-labs/dbt-core/pull/4743))
- add performance regression testing runner without orchestration ([#4021](https://github.com/dbt-labs/dbt-core/issues/4021), [#4602](https://github.com/dbt-labs/dbt-core/pull/4602))
- Fix broken links for changelog generation and tweak GHA to only post a comment once when changelog entry is missing. ([#4848](https://github.com/dbt-labs/dbt-core/issues/4848), [#4857](https://github.com/dbt-labs/dbt-core/pull/4857))
- Add support for Python 3.10 ([#4562](https://github.com/dbt-labs/dbt-core/issues/4562), [#4866](https://github.com/dbt-labs/dbt-core/pull/4866))
- Enable more dialects to snapshot sources with added columns, even those that don't support boolean datatypes ([#4488](https://github.com/dbt-labs/dbt-core/issues/4488), [#4871](https://github.com/dbt-labs/dbt-core/pull/4871))
- Add Graph Compilation and Adapter Cache tracking ([#4625](https://github.com/dbt-labs/dbt-core/issues/4625), [#4912](https://github.com/dbt-labs/dbt-core/pull/4912))
- Testing cleanup ([#3648](https://github.com/dbt-labs/dbt-core/issues/3648), [#4509](https://github.com/dbt-labs/dbt-core/pull/4509))
- Clean up test deprecation warnings ([#3988](https://github.com/dbt-labs/dbt-core/issues/3988), [#4556](https://github.com/dbt-labs/dbt-core/pull/4556))
- Use mashumaro for serialization in event logging ([#4504](https://github.com/dbt-labs/dbt-core/issues/4504), [#4505](https://github.com/dbt-labs/dbt-core/pull/4505))
- Drop support for Python 3.7.0 + 3.7.1 ([#4584](https://github.com/dbt-labs/dbt-core/issues/4584), [#4585](https://github.com/dbt-labs/dbt-core/pull/4585))
- Drop support for Python <3.7.2 ([#4584](https://github.com/dbt-labs/dbt-core/issues/4584), [#4643](https://github.com/dbt-labs/dbt-core/pull/4643))
- Re-format codebase (except tests) using pre-commit hooks ([#3195](https://github.com/dbt-labs/dbt-core/issues/3195), [#4697](https://github.com/dbt-labs/dbt-core/pull/4697))
- Add deps module README ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4686](https://github.com/dbt-labs/dbt-core/pull/4686))
- Initial conversion of tests to pytest ([#4690](https://github.com/dbt-labs/dbt-core/issues/4690), [#4691](https://github.com/dbt-labs/dbt-core/pull/4691))
- Fix errors in Windows for tests/functions ([#4782](https://github.com/dbt-labs/dbt-core/issues/4782), [#4767](https://github.com/dbt-labs/dbt-core/pull/4767))
- Create a dbt.tests.adapter release when releasing dbt and postgres ([#4812](https://github.com/dbt-labs/dbt-core/issues/4812), [#4948](https://github.com/dbt-labs/dbt-core/pull/4948))
- update docker image to use python 3.10.3 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4963](https://github.com/dbt-labs/dbt-core/pull/4963))
- updates black to 22.3.0 which fixes dependency incompatibility when running with precommit. ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4972](https://github.com/dbt-labs/dbt-core/pull/4972))
- Adds config util for ad-hoc creation of project objs or dicts ([#4808](https://github.com/dbt-labs/dbt-core/issues/4808), [#4981](https://github.com/dbt-labs/dbt-core/pull/4981))
- Remove TableComparison and convert existing calls to use dbt.tests.util ([#4778](https://github.com/dbt-labs/dbt-core/issues/4778), [#4986](https://github.com/dbt-labs/dbt-core/pull/4986))
- Remove unneeded create_schema in snapshot materialization ([#4742](https://github.com/dbt-labs/dbt-core/issues/4742), [#4993](https://github.com/dbt-labs/dbt-core/pull/4993))
- Added .git-blame-ignore-revs file to mask re-formmating commits from git blame ([#5004](https://github.com/dbt-labs/dbt-core/issues/5004), [#5019](https://github.com/dbt-labs/dbt-core/pull/5019))
- Convert version tests to pytest ([#5024](https://github.com/dbt-labs/dbt-core/issues/5024), [#5026](https://github.com/dbt-labs/dbt-core/pull/5026))
- Updating tests and docs to show that we now support Python 3.10 ([#4974](https://github.com/dbt-labs/dbt-core/issues/4974), [#5025](https://github.com/dbt-labs/dbt-core/pull/5025))
- Update --version output and logic ([#4724](https://github.com/dbt-labs/dbt-core/issues/4724), [#5029](https://github.com/dbt-labs/dbt-core/pull/5029))
- ([#5033](https://github.com/dbt-labs/dbt-core/issues/5033), [#5032](https://github.com/dbt-labs/dbt-core/pull/5032))
### Contributors
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- [@agoblet](https://github.com/agoblet) ([#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
- [@alswang18](https://github.com/alswang18) ([#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- [@amirkdv](https://github.com/amirkdv) ([#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- [@anaisvaillant](https://github.com/anaisvaillant) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@b-per](https://github.com/b-per) ([#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
- [@dbeatty10](https://github.com/dbeatty10) ([#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@ehmartens](https://github.com/ehmartens) ([#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- [@joellabes](https://github.com/joellabes) ([#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- [@jonstacks](https://github.com/jonstacks) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- [@kadero](https://github.com/kadero) ([#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
- [@karunpoudel](https://github.com/karunpoudel) ([#4860](https://github.com/dbt-labs/dbt-core/pull/4860), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
- [@kazanzhy](https://github.com/kazanzhy) ([#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- [@matt-winkler](https://github.com/matt-winkler) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@mdesmet](https://github.com/mdesmet) ([#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- [@pgoslatara](https://github.com/pgoslatara) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- [@poloaraujo](https://github.com/poloaraujo) ([#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
- [@sungchun12](https://github.com/sungchun12) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@willbowditch](https://github.com/willbowditch) ([#4777](https://github.com/dbt-labs/dbt-core/pull/4777))

View File

@@ -1,11 +0,0 @@
kind: Breaking Changes
body: >
**Relevant to maintainers of adapter plugins _only_:**
The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints.
(Previously, they could return a string.) We encourage you to update your methods to return an object of
class `AdapterResponse`, or implement a subclass specific to your adapter
time: 2022-03-16T03:13:11.656007-07:00
custom:
Author: versusfacit
Issue: "4499"
PR: "4869"

View File

@@ -1,8 +0,0 @@
kind: Docs
body: Resolve errors related to operations preventing DAG from generating in the docs. Also
patch a spark issue to allow search to filter accurately past the missing columns.
time: 2022-03-07T20:31:05.557064-06:00
custom:
Author: emmyoop
Issue: "4578"
PR: "4763"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Testing framework for dbt adapter testing
time: 2022-03-09T14:26:45.828295-05:00
custom:
Author: gshank
Issue: "4730"
PR: "4846"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Change behaviour of `non_null` test so that it only `select`s all columns if
`--store-failures` is enabled.
time: 2022-03-09T16:38:17.160568Z
custom:
Author: willbowditch
Issue: "4769"
PR: "4777"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Allow unique key to take a list implementation for postgres/redshift
time: 2022-03-14T11:23:41.293726-05:00
custom:
Author: McKnight-42
Issue: "4738"
PR: "4858"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Added Support for Semantic Versioning
time: 2022-03-21T13:59:49.725046-05:00
custom:
Author: alswang18
Issue: "4453"
PR: "4644"

View File

@@ -1,8 +0,0 @@
kind: Features
body: New Dockerfile to support specific db adapters and platforms. See docker/README.md
for details
time: 2022-03-21T14:00:46.670045-05:00
custom:
Author: iknox-fa
Issue: "4495"
PR: "4487"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Allow unique_key to take a list
time: 2022-03-21T14:01:39.918897-05:00
custom:
Author: gshank
Issue: "2479"
PR: "4618"

View File

@@ -1,7 +0,0 @@
kind: Features
body: "Add `--quiet` global flag and `print` Jinja function"
time: 2022-03-21T14:03:06.565157-05:00
custom:
Author: ehmartens
Issue: "3451"
PR: "4701"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add space before justification periods
time: 2022-03-22T14:48:22.64892-04:00
custom:
Author: joellabes
Issue: "4737"
PR: "4744"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix bug causing empty node level meta, snapshot config errors
time: 2022-03-07T20:30:22.624709-06:00
custom:
Author: gshank
Issue: "4459"
PR: "4726"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Catch all Requests Exceptions on deps install to attempt retries. Also log
the exceptions hit.
time: 2022-03-15T10:53:31.637963-05:00
custom:
Author: emmyoop
Issue: "4849"
PR: "4865"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Use cli_vars instead of context to create package and selector renderers
time: 2022-03-16T15:54:20.608384-04:00
custom:
Author: gshank
Issue: "4876"
PR: "4878"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: depend on new dbt-extractor version with fixed github links
time: 2022-03-17T13:24:18.419599-04:00
custom:
Author: nathaniel-may
Issue: "4891"
PR: "4890"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Update bumpervsion config to stop looking for missing setup.py
time: 2022-03-17T16:30:51.299919-05:00
custom:
Author: iknox-fa
Issue: "-1"
PR: "4896"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Corrected permissions settings for docker release workflow
time: 2022-03-18T09:58:46.061468-05:00
custom:
Author: iknox-fa
Issue: "4902"
PR: "4903"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: "User wasn't asked for permission to overwite a profile entry when running init inside an existing project"
time: 2022-03-21T14:05:21.238521-05:00
custom:
Author: NiallRees
Issue: "4375"
PR: "4447"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add project name validation to `dbt init`
time: 2022-03-21T14:07:20.153071-05:00
custom:
Author: amirkdv
Issue: "4490"
PR: "4536"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Allow override of string and numeric types for adapters.
time: 2022-03-21T14:08:33.384372-05:00
custom:
Author: mdesmet
Issue: "4603"
PR: "4604"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: A change in secret environment variables won't trigger a full reparse
time: 2022-03-21T14:09:21.004543-05:00
custom:
Author: nathaniel-may
Issue: "4650"
PR: "4665"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix misspellings and typos in docstrings
time: 2022-03-21T14:10:07.962502-05:00
custom:
Author: kazanzhy
Issue: "4904"
PR: "4545"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Automate changelog generation with changie
time: 2022-02-18T16:13:19.882436-06:00
custom:
Author: emmyoop
Issue: "4652"
PR: "4743"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Fix broken links for changelog generation and tweak GHA to only post a comment
once when changelog entry is missing.
time: 2022-03-11T10:18:51.404524-06:00
custom:
Author: emmyoop
Issue: "4848"
PR: "4857"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add support for Python 3.10
time: 2022-03-15T15:54:36.759434-05:00
custom:
Author: iknox-fa
Issue: "4562"
PR: "4866"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Enable more dialects to snapshot sources with added columns, even those that
don't support boolean datatypes
time: 2022-03-16T10:03:05.96768-05:00
custom:
Author: McKnight-42
Issue: "4488"
PR: "4871"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Testing cleanup
time: 2022-03-21T14:11:58.750419-05:00
custom:
Author: emmyoop
Issue: "3648"
PR: "4509"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Clean up test deprecation warnings
time: 2022-03-21T14:13:00.557425-05:00
custom:
Author: gshank
Issue: "3988"
PR: "4556"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Use mashumaro for serialization in event logging
time: 2022-03-21T14:13:38.758158-05:00
custom:
Author: gshank
Issue: "4504"
PR: "4505"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Drop support for Python 3.7.0 + 3.7.1
time: 2022-03-21T14:14:51.840637-05:00
custom:
Author: nathaniel-may
Issue: "4584"
PR: "4585"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Drop support for Python <3.7.2
time: 2022-03-21T14:15:23.893773-05:00
custom:
Author: jtcohen6
Issue: "4584"
PR: "4643"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Re-format codebase (except tests) using pre-commit hooks
time: 2022-03-21T14:16:10.285506-05:00
custom:
Author: iknox-fa
Issue: "3195"
PR: "4697"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add deps module README
time: 2022-03-21T14:17:07.884314-05:00
custom:
Author: emmyoop
Issue: "4904"
PR: "4686"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Initial conversion of tests to pytest
time: 2022-03-21T14:17:47.852378-05:00
custom:
Author: gshank
Issue: "4690"
PR: "4691"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix errors in Windows for tests/functions
time: 2022-03-21T14:20:55.648765-05:00
custom:
Author: gshank
Issue: "4782"
PR: "4767"

9
.changes/1.1.1-rc1.md Normal file
View File

@@ -0,0 +1,9 @@
## dbt-core 1.1.1-rc1 - May 20, 2022
### Fixes
- Relax minimum supported version of MarkupSafe ([#4745](https://github.com/dbt-labs/dbt-core/issues/4745), [#5039](https://github.com/dbt-labs/dbt-core/pull/5039))
- When parsing 'all_sources' should be a list of unique dirs ([#5120](https://github.com/dbt-labs/dbt-core/issues/5120), [#5176](https://github.com/dbt-labs/dbt-core/pull/5176))
### Dependencies
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
### Contributors
- [@adamantike](https://github.com/adamantike) ([#5039](https://github.com/dbt-labs/dbt-core/pull/5039))

8
.changes/1.1.1-rc2.md Normal file
View File

@@ -0,0 +1,8 @@
## dbt-core 1.1.1-rc2 - June 07, 2022
### Fixes
- Remove docs file from manifest when removing doc node ([#4146](https://github.com/dbt-labs/dbt-core/issues/4146), [#5270](https://github.com/dbt-labs/dbt-core/pull/5270))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
### Under the Hood
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Dependencies
- Pin networkx to <2.8.4 for v1.1 patches ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))

View File

@@ -0,0 +1,7 @@
kind: Dependencies
body: "Bumping hologram version"
time: 2022-05-06T16:09:07.000000-05:00
custom:
Author: leahwicz
Issue: "5219"
PR: "5218"

View File

@@ -0,0 +1,7 @@
kind: Dependencies
body: Pin networkx to <2.8.4 for v1.1 patches
time: 2022-06-06T23:42:58.406625+02:00
custom:
Author: jtcohen6
Issue: "5286"
PR: "5334"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Relax minimum supported version of MarkupSafe
time: 2022-04-13T15:17:35.055274145-03:00
custom:
Author: adamantike
Issue: "4745"
PR: "5039"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: When parsing 'all_sources' should be a list of unique dirs
time: 2022-04-27T10:26:48.648388-04:00
custom:
Author: gshank
Issue: "5120"
PR: "5176"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Remove docs file from manifest when removing doc node
time: 2022-05-18T13:46:10.167143-04:00
custom:
Author: gshank
Issue: "4146"
PR: "5270"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Fixing Windows color regression
time: 2022-06-01T19:42:34.263009-04:00
custom:
Author: leahwicz
Issue: "5191"
PR: "5327"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Update context readme + clean up context code"
time: 2022-06-06T23:03:53.022568+02:00
custom:
Author: jtcohen6
Issue: "4796"
PR: "5334"

View File

@@ -1,9 +0,0 @@
kind: Breaking Changes
body: 'For adapter plugin maintainers only: Internal adapter methods `set_relations_cache` + `_relations_cache_for_schemas`
each take an additional argument, for use with experimental `CACHE_SELECTED_ONLY`
config'
time: 2022-04-12T15:24:50.159572+02:00
custom:
Author: karunpoudel
Issue: "4688"
PR: "4860"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: 'Fixed capitalization in UI for exposures of `type: ml`'
time: 2022-04-04T20:10:21.941618-05:00
custom:
Author: jonstacks
Issue: "4984"
PR: "4995"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: List packages and tags in alphabetical order
time: 2022-04-04T20:10:57.066016-05:00
custom:
Author: pgoslatara
Issue: "4984"
PR: "4995"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Bump jekyll from 3.8.7 to 3.9.0
time: 2022-04-04T20:12:03.146811-05:00
custom:
Author: emmyoop
Issue: "4984"
PR: "4995"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Updated docker README to reflect necessity of using BuildKit
time: 2022-04-08T13:35:40.361554-05:00
custom:
Author: iknox-fa
Issue: "4990"
PR: "5018"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Add `--cache_selected_only` flag to cache schema object of selected models
only.
time: 2022-03-16T00:38:47.8468296-05:00
custom:
Author: karunpoudel
Issue: "4688"
PR: "4860"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Support custom names for generic tests
time: 2022-03-18T08:57:56.05584+01:00
custom:
Author: jtcohen6
Issue: "3348"
PR: "4898"

View File

@@ -1,10 +0,0 @@
kind: Features
body: 'Enable dbt jobs to run downstream models based on fresher sources. Compare
the source freshness results between previous and current state. If any source is
fresher and/or new in current vs. previous state, dbt will run and test the downstream
models in scope. Example command: `dbt build --select source_status:fresher+` '
time: 2022-03-28T13:47:43.750709-05:00
custom:
Author: sungchun12, matt-winkler, anaisvaillant
Issue: "4050"
PR: "4256"

View File

@@ -1,7 +0,0 @@
kind: Features
body: converting unique key as list tests to new pytest format
time: 2022-03-28T15:08:12.70006-05:00
custom:
Author: McKnight-42
Issue: "4882"
PR: "4958"

View File

@@ -1,9 +0,0 @@
kind: Features
body: Add a variable called selected_resources in the Jinja context containing a list
of all the resources matching the nodes for the --select, --exclude and/or --selector
parameters.
time: 2022-04-04T19:04:39.347479+02:00
custom:
Author: b-per
Issue: "3471"
PR: "5001"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Support the DO_NOT_TRACK environment variable from the consoledonottrack.com initiative
time: 2022-04-06T08:03:28.877054+02:00
custom:
Author: agoblet
Issue: "3540"
PR: "5000"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add `--no-print` global flag
time: 2022-04-08T11:41:18.934373+01:00
custom:
Author: poloaraujo
Issue: "4710"
PR: "4854"

View File

@@ -1,7 +0,0 @@
kind: Features
body: add enabled as a source config
time: 2022-04-08T13:27:25.292454-04:00
custom:
Author: nathaniel-may
Issue: "3662"
PR: "5008"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Inconsistent timestamps between inserted/updated and deleted rows in snapshots
time: 2022-03-14T20:02:36.119598832+01:00
custom:
Author: kadero
Issue: "4347"
PR: "4513"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Catch more cases to retry package retrieval for deps pointing to the hub. Also start to cache the package requests.
time: 2022-03-31T14:39:23.952705-05:00
custom:
Author: emmyoop
Issue: "4849"
PR: "4982"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Make the warning message for a full event deque more descriptive
time: 2022-04-07T16:11:34.614988-05:00
custom:
Author: iknox-fa
Issue: "4962"
PR: "5011"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix hard delete snapshot test
time: 2022-04-08T15:55:12.552138-04:00
custom:
Author: gshank
Issue: "4916"
PR: "5020"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: add performance regression testing runner without orchestration
time: 2022-03-09T14:21:33.884043-05:00
custom:
Author: nathaniel-may
Issue: "4021"
PR: "4602"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add Graph Compilation and Adapter Cache tracking
time: 2022-03-21T14:28:54.160087-04:00
custom:
Author: ChenyuLInx
Issue: "4625"
PR: "4912"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Create a dbt.tests.adapter release when releasing dbt and postgres
time: 2022-03-29T11:24:48.34465-04:00
custom:
Author: gshank
Issue: "4812"
PR: "4948"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: update docker image to use python 3.10.3
time: 2022-03-29T11:26:53.359066-04:00
custom:
Author: nathaniel-may
Issue: "4904"
PR: "4963"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: updates black to 22.3.0 which fixes dependency incompatibility when running
with precommit.
time: 2022-03-29T11:35:42.38155-04:00
custom:
Author: nathaniel-may
Issue: "4904"
PR: "4972"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Adds config util for ad-hoc creation of project objs or dicts
time: 2022-03-30T10:14:39.746196-05:00
custom:
Author: iknox-fa
Issue: "4808"
PR: "4981"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Remove TableComparison and convert existing calls to use dbt.tests.util
time: 2022-04-01T15:47:29.054759-04:00
custom:
Author: gshank
Issue: "4778"
PR: "4986"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Remove unneeded create_schema in snapshot materialization
time: 2022-04-04T14:47:08.960642+02:00
custom:
Author: jtcohen6
Issue: "4742"
PR: "4993"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Added .git-blame-ignore-revs file to mask re-formmating commits from git blame
time: 2022-04-08T13:55:25.790513-05:00
custom:
Author: iknox-fa
Issue: "5004"
PR: "5019"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert version tests to pytest
time: 2022-04-11T09:05:36.956877-05:00
custom:
Author: stu-k
Issue: "5024"
PR: "5026"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Updating tests and docs to show that we now support Python 3.10
time: 2022-04-11T10:37:24.806649-04:00
custom:
Author: leahwicz
Issue: "4974"
PR: "5025"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Update --version output and logic
time: 2022-04-11T15:46:26.113705-05:00
custom:
Author: stu-k
Issue: "4724"
PR: "5029"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: ""
time: 2022-04-11T21:39:14.402268-04:00
custom:
Author: leahwicz
Issue: "5033"
PR: "5032"

View File

@@ -148,8 +148,8 @@ jobs:
- name: Install python tools
run: |
pip install --user --upgrade pip
pip --version
# pip install --user --upgrade pip
# pip --version
pip install tox
tox --version

View File

@@ -6,20 +6,53 @@
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.1.0-b1 - March 22, 2022
## dbt-core 1.1.1-rc2 - June 07, 2022
### Fixes
- Remove docs file from manifest when removing doc node ([#4146](https://github.com/dbt-labs/dbt-core/issues/4146), [#5270](https://github.com/dbt-labs/dbt-core/pull/5270))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
### Under the Hood
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Dependencies
- Pin networkx to <2.8.4 for v1.1 patches ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
## dbt-core 1.1.1-rc1 - May 20, 2022
### Fixes
- Relax minimum supported version of MarkupSafe ([#4745](https://github.com/dbt-labs/dbt-core/issues/4745), [#5039](https://github.com/dbt-labs/dbt-core/pull/5039))
- When parsing 'all_sources' should be a list of unique dirs ([#5120](https://github.com/dbt-labs/dbt-core/issues/5120), [#5176](https://github.com/dbt-labs/dbt-core/pull/5176))
### Dependencies
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
### Contributors
- [@adamantike](https://github.com/adamantike) ([#5039](https://github.com/dbt-labs/dbt-core/pull/5039))
## dbt-core 1.1.0 - April 28, 2022
### Breaking Changes
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter ([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter
([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- For adapter plugin maintainers only: Internal adapter methods `set_relations_cache` + `_relations_cache_for_schemas` each take an additional argument, for use with experimental `CACHE_SELECTED_ONLY` config ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
### Features
- Change behaviour of `non_null` test so that it only `select`s all columns if `--store-failures` is enabled. ([#4769](https://github.com/dbt-labs/dbt-core/issues/4769), [#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
- Testing framework for dbt adapter testing ([#4730](https://github.com/dbt-labs/dbt-core/issues/4730), [#4846](https://github.com/dbt-labs/dbt-core/pull/4846))
- Allow unique key to take a list implementation for postgres/redshift ([#4738](https://github.com/dbt-labs/dbt-core/issues/4738), [#4858](https://github.com/dbt-labs/dbt-core/pull/4858))
- Add `--cache_selected_only` flag to cache schema object of selected models only. ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
- Support custom names for generic tests ([#3348](https://github.com/dbt-labs/dbt-core/issues/3348), [#4898](https://github.com/dbt-labs/dbt-core/pull/4898))
- Added Support for Semantic Versioning ([#4453](https://github.com/dbt-labs/dbt-core/issues/4453), [#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- New Dockerfile to support specific db adapters and platforms. See docker/README.md for details ([#4495](https://github.com/dbt-labs/dbt-core/issues/4495), [#4487](https://github.com/dbt-labs/dbt-core/pull/4487))
- Allow unique_key to take a list ([#2479](https://github.com/dbt-labs/dbt-core/issues/2479), [#4618](https://github.com/dbt-labs/dbt-core/pull/4618))
- Add `--quiet` global flag and `print` Jinja function ([#3451](https://github.com/dbt-labs/dbt-core/issues/3451), [#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- Add space before justification periods ([#4737](https://github.com/dbt-labs/dbt-core/issues/4737), [#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- Enable dbt jobs to run downstream models based on fresher sources. Compare the source freshness results between previous and current state. If any source is fresher and/or new in current vs. previous state, dbt will run and test the downstream models in scope. Example command: `dbt build --select source_status:fresher+` ([#4050](https://github.com/dbt-labs/dbt-core/issues/4050), [#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- converting unique key as list tests to new pytest format ([#4882](https://github.com/dbt-labs/dbt-core/issues/4882), [#4958](https://github.com/dbt-labs/dbt-core/pull/4958))
- Add a variable called selected_resources in the Jinja context containing a list of all the resources matching the nodes for the --select, --exclude and/or --selector parameters. ([#3471](https://github.com/dbt-labs/dbt-core/issues/3471), [#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
- Support the DO_NOT_TRACK environment variable from the consoledonottrack.com initiative ([#3540](https://github.com/dbt-labs/dbt-core/issues/3540), [#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
- Add `--no-print` global flag ([#4710](https://github.com/dbt-labs/dbt-core/issues/4710), [#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
- add enabled as a source config ([#3662](https://github.com/dbt-labs/dbt-core/issues/3662), [#5008](https://github.com/dbt-labs/dbt-core/pull/5008))
### Fixes
- Fix bug causing empty node level meta, snapshot config errors ([#4459](https://github.com/dbt-labs/dbt-core/issues/4459), [#4726](https://github.com/dbt-labs/dbt-core/pull/4726))
- Inconsistent timestamps between inserted/updated and deleted rows in snapshots ([#4347](https://github.com/dbt-labs/dbt-core/issues/4347), [#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
- Catch all Requests Exceptions on deps install to attempt retries. Also log the exceptions hit. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4865](https://github.com/dbt-labs/dbt-core/pull/4865))
- Use cli_vars instead of context to create package and selector renderers ([#4876](https://github.com/dbt-labs/dbt-core/issues/4876), [#4878](https://github.com/dbt-labs/dbt-core/pull/4878))
- depend on new dbt-extractor version with fixed github links ([#4891](https://github.com/dbt-labs/dbt-core/issues/4891), [#4890](https://github.com/dbt-labs/dbt-core/pull/4890))
@@ -30,11 +63,26 @@
- Allow override of string and numeric types for adapters. ([#4603](https://github.com/dbt-labs/dbt-core/issues/4603), [#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- A change in secret environment variables won't trigger a full reparse ([#4650](https://github.com/dbt-labs/dbt-core/issues/4650), [#4665](https://github.com/dbt-labs/dbt-core/pull/4665))
- Fix misspellings and typos in docstrings ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- Catch more cases to retry package retrieval for deps pointing to the hub. Also start to cache the package requests. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4982](https://github.com/dbt-labs/dbt-core/pull/4982))
- Make the warning message for a full event deque more descriptive ([#4962](https://github.com/dbt-labs/dbt-core/issues/4962), [#5011](https://github.com/dbt-labs/dbt-core/pull/5011))
- Fix hard delete snapshot test ([#4916](https://github.com/dbt-labs/dbt-core/issues/4916), [#5020](https://github.com/dbt-labs/dbt-core/pull/5020))
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- Use yaml renderer (with target context) for rendering selectors ([#5131](https://github.com/dbt-labs/dbt-core/issues/5131), [#5136](https://github.com/dbt-labs/dbt-core/pull/5136))
- Fix retry logic to return values after initial try ([#5023](https://github.com/dbt-labs/dbt-core/issues/5023), [#5137](https://github.com/dbt-labs/dbt-core/pull/5137))
- Scrub secret env vars from CommandError in exception stacktrace ([#5151](https://github.com/dbt-labs/dbt-core/issues/5151), [#5152](https://github.com/dbt-labs/dbt-core/pull/5152))
### Docs
- Resolve errors related to operations preventing DAG from generating in the docs. Also patch a spark issue to allow search to filter accurately past the missing columns. ([#4578](https://github.com/dbt-labs/dbt-core/issues/4578), [#4763](https://github.com/dbt-labs/dbt-core/pull/4763))
- Fixed capitalization in UI for exposures of `type: ml` ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- List packages and tags in alphabetical order ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- Bump jekyll from 3.8.7 to 3.9.0 ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- Updated docker README to reflect necessity of using BuildKit ([#4990](https://github.com/dbt-labs/dbt-core/issues/4990), [#5018](https://github.com/dbt-labs/dbt-core/pull/5018))
### Under the Hood
- Automate changelog generation with changie ([#4652](https://github.com/dbt-labs/dbt-core/issues/4652), [#4743](https://github.com/dbt-labs/dbt-core/pull/4743))
- add performance regression testing runner without orchestration ([#4021](https://github.com/dbt-labs/dbt-core/issues/4021), [#4602](https://github.com/dbt-labs/dbt-core/pull/4602))
- Fix broken links for changelog generation and tweak GHA to only post a comment once when changelog entry is missing. ([#4848](https://github.com/dbt-labs/dbt-core/issues/4848), [#4857](https://github.com/dbt-labs/dbt-core/pull/4857))
- Add support for Python 3.10 ([#4562](https://github.com/dbt-labs/dbt-core/issues/4562), [#4866](https://github.com/dbt-labs/dbt-core/pull/4866))
- Enable more dialects to snapshot sources with added columns, even those that don't support boolean datatypes ([#4488](https://github.com/dbt-labs/dbt-core/issues/4488), [#4871](https://github.com/dbt-labs/dbt-core/pull/4871))
- Add Graph Compilation and Adapter Cache tracking ([#4625](https://github.com/dbt-labs/dbt-core/issues/4625), [#4912](https://github.com/dbt-labs/dbt-core/pull/4912))
- Testing cleanup ([#3648](https://github.com/dbt-labs/dbt-core/issues/3648), [#4509](https://github.com/dbt-labs/dbt-core/pull/4509))
- Clean up test deprecation warnings ([#3988](https://github.com/dbt-labs/dbt-core/issues/3988), [#4556](https://github.com/dbt-labs/dbt-core/pull/4556))
- Use mashumaro for serialization in event logging ([#4504](https://github.com/dbt-labs/dbt-core/issues/4504), [#4505](https://github.com/dbt-labs/dbt-core/pull/4505))
@@ -44,18 +92,38 @@
- Add deps module README ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4686](https://github.com/dbt-labs/dbt-core/pull/4686))
- Initial conversion of tests to pytest ([#4690](https://github.com/dbt-labs/dbt-core/issues/4690), [#4691](https://github.com/dbt-labs/dbt-core/pull/4691))
- Fix errors in Windows for tests/functions ([#4782](https://github.com/dbt-labs/dbt-core/issues/4782), [#4767](https://github.com/dbt-labs/dbt-core/pull/4767))
### Docs
- Resolve errors related to operations preventing DAG from generating in the docs. Also patch a spark issue to allow search to filter accurately past the missing columns. ([#4578](https://github.com/dbt-labs/dbt-core/issues/4578), [#4763](https://github.com/dbt-labs/dbt-core/pull/4763))
- Create a dbt.tests.adapter release when releasing dbt and postgres ([#4812](https://github.com/dbt-labs/dbt-core/issues/4812), [#4948](https://github.com/dbt-labs/dbt-core/pull/4948))
- update docker image to use python 3.10.3 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4963](https://github.com/dbt-labs/dbt-core/pull/4963))
- updates black to 22.3.0 which fixes dependency incompatibility when running with precommit. ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4972](https://github.com/dbt-labs/dbt-core/pull/4972))
- Adds config util for ad-hoc creation of project objs or dicts ([#4808](https://github.com/dbt-labs/dbt-core/issues/4808), [#4981](https://github.com/dbt-labs/dbt-core/pull/4981))
- Remove TableComparison and convert existing calls to use dbt.tests.util ([#4778](https://github.com/dbt-labs/dbt-core/issues/4778), [#4986](https://github.com/dbt-labs/dbt-core/pull/4986))
- Remove unneeded create_schema in snapshot materialization ([#4742](https://github.com/dbt-labs/dbt-core/issues/4742), [#4993](https://github.com/dbt-labs/dbt-core/pull/4993))
- Added .git-blame-ignore-revs file to mask re-formmating commits from git blame ([#5004](https://github.com/dbt-labs/dbt-core/issues/5004), [#5019](https://github.com/dbt-labs/dbt-core/pull/5019))
- Convert version tests to pytest ([#5024](https://github.com/dbt-labs/dbt-core/issues/5024), [#5026](https://github.com/dbt-labs/dbt-core/pull/5026))
- Updating tests and docs to show that we now support Python 3.10 ([#4974](https://github.com/dbt-labs/dbt-core/issues/4974), [#5025](https://github.com/dbt-labs/dbt-core/pull/5025))
- Update --version output and logic ([#4724](https://github.com/dbt-labs/dbt-core/issues/4724), [#5029](https://github.com/dbt-labs/dbt-core/pull/5029))
- ([#5033](https://github.com/dbt-labs/dbt-core/issues/5033), [#5032](https://github.com/dbt-labs/dbt-core/pull/5032))
### Contributors
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- [@alswang18](https://github.com/alswang18) ([#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- [@amirkdv](https://github.com/amirkdv) ([#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- [@ehmartens](https://github.com/ehmartens) ([#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- [@joellabes](https://github.com/joellabes) ([#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- [@kazanzhy](https://github.com/kazanzhy) ([#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- [@mdesmet](https://github.com/mdesmet) ([#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- [@versusfacit](https://github.com/versusfacit) ([#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- [@willbowditch](https://github.com/willbowditch) ([#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- [@agoblet](https://github.com/agoblet) ([#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
- [@alswang18](https://github.com/alswang18) ([#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- [@amirkdv](https://github.com/amirkdv) ([#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- [@anaisvaillant](https://github.com/anaisvaillant) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@b-per](https://github.com/b-per) ([#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
- [@dbeatty10](https://github.com/dbeatty10) ([#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@ehmartens](https://github.com/ehmartens) ([#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- [@joellabes](https://github.com/joellabes) ([#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- [@jonstacks](https://github.com/jonstacks) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- [@kadero](https://github.com/kadero) ([#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
- [@karunpoudel](https://github.com/karunpoudel) ([#4860](https://github.com/dbt-labs/dbt-core/pull/4860), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
- [@kazanzhy](https://github.com/kazanzhy) ([#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- [@matt-winkler](https://github.com/matt-winkler) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@mdesmet](https://github.com/mdesmet) ([#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- [@pgoslatara](https://github.com/pgoslatara) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- [@poloaraujo](https://github.com/poloaraujo) ([#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
- [@sungchun12](https://github.com/sungchun12) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@willbowditch](https://github.com/willbowditch) ([#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
## Previous Releases

View File

@@ -28,7 +28,7 @@ def _is_commit(revision: str) -> bool:
def _raise_git_cloning_error(repo, revision, error):
stderr = error.stderr.decode("utf-8").strip()
stderr = error.stderr.strip()
if "usage: git" in stderr:
stderr = stderr.split("\nusage: git")[0]
if re.match("fatal: destination path '(.+)' already exists", stderr):
@@ -115,8 +115,8 @@ def checkout(cwd, repo, revision=None):
try:
return _checkout(cwd, repo, revision)
except CommandResultError as exc:
stderr = exc.stderr.decode("utf-8").strip()
bad_package_spec(repo, revision, stderr)
stderr = exc.stderr.strip()
bad_package_spec(repo, revision, stderr)
def get_current_sha(cwd):
@@ -142,7 +142,7 @@ def clone_and_checkout(
subdirectory=subdirectory,
)
except CommandResultError as exc:
err = exc.stderr.decode("utf-8")
err = exc.stderr
exists = re.match("fatal: destination path '(.+)' already exists", err)
if not exists:
raise_git_cloning_problem(repo)

View File

@@ -132,7 +132,11 @@ def _all_source_paths(
analysis_paths: List[str],
macro_paths: List[str],
) -> List[str]:
return list(chain(model_paths, seed_paths, snapshot_paths, analysis_paths, macro_paths))
# We need to turn a list of lists into just a list, then convert to a set to
# get only unique elements, then back to a list
return list(
set(list(chain(model_paths, seed_paths, snapshot_paths, analysis_paths, macro_paths)))
)
T = TypeVar("T")

View File

@@ -1,12 +1,15 @@
from typing import Dict, Any, Tuple, Optional, Union, Callable
import re
import os
from dbt.clients.jinja import get_rendered, catch_jinja
from dbt.context.target import TargetContext
from dbt.context.secret import SecretContext
from dbt.context.secret import SecretContext, SECRET_PLACEHOLDER
from dbt.context.base import BaseContext
from dbt.contracts.connection import HasCredentials
from dbt.exceptions import DbtProjectError, CompilationException, RecursionException
from dbt.utils import deep_map_render
from dbt.logger import SECRET_ENV_PREFIX
Keypath = Tuple[Union[str, int], ...]
@@ -114,12 +117,10 @@ class DbtProjectYamlRenderer(BaseRenderer):
def name(self):
"Project config"
# Uses SecretRenderer
def get_package_renderer(self) -> BaseRenderer:
return PackageRenderer(self.ctx_obj.cli_vars)
def get_selector_renderer(self) -> BaseRenderer:
return SelectorRenderer(self.ctx_obj.cli_vars)
def render_project(
self,
project: Dict[str, Any],
@@ -136,8 +137,7 @@ class DbtProjectYamlRenderer(BaseRenderer):
return package_renderer.render_data(packages)
def render_selectors(self, selectors: Dict[str, Any]):
selector_renderer = self.get_selector_renderer()
return selector_renderer.render_data(selectors)
return self.render_data(selectors)
def render_entry(self, value: Any, keypath: Keypath) -> Any:
result = super().render_entry(value, keypath)
@@ -165,18 +165,10 @@ class DbtProjectYamlRenderer(BaseRenderer):
return True
class SelectorRenderer(BaseRenderer):
@property
def name(self):
return "Selector config"
class SecretRenderer(BaseRenderer):
def __init__(self, cli_vars: Optional[Dict[str, Any]] = None) -> None:
def __init__(self, cli_vars: Dict[str, Any] = {}) -> None:
# Generate contexts here because we want to save the context
# object in order to retrieve the env_vars.
if cli_vars is None:
cli_vars = {}
self.ctx_obj = SecretContext(cli_vars)
context = self.ctx_obj.to_dict()
super().__init__(context)
@@ -185,6 +177,23 @@ class SecretRenderer(BaseRenderer):
def name(self):
return "Secret"
def render_value(self, value: Any, keypath: Optional[Keypath] = None) -> Any:
rendered = super().render_value(value, keypath)
if SECRET_ENV_PREFIX in str(rendered):
search_group = f"({SECRET_ENV_PREFIX}(.*))"
pattern = SECRET_PLACEHOLDER.format(search_group).replace("$", r"\$")
m = re.search(
pattern,
rendered,
)
if m:
found = m.group(1)
value = os.environ[found]
replace_this = SECRET_PLACEHOLDER.format(found)
return rendered.replace(replace_this, value)
else:
return rendered
class ProfileRenderer(SecretRenderer):
@property

View File

@@ -3,7 +3,7 @@ from typing import Dict, Any, Union
from dbt.clients.yaml_helper import yaml, Loader, Dumper, load_yaml_text # noqa: F401
from dbt.dataclass_schema import ValidationError
from .renderer import SelectorRenderer
from .renderer import BaseRenderer
from dbt.clients.system import (
load_file_contents,
@@ -57,7 +57,7 @@ class SelectorConfig(Dict[str, Dict[str, Union[SelectionSpec, bool]]]):
def render_from_dict(
cls,
data: Dict[str, Any],
renderer: SelectorRenderer,
renderer: BaseRenderer,
) -> "SelectorConfig":
try:
rendered = renderer.render_data(data)
@@ -72,7 +72,7 @@ class SelectorConfig(Dict[str, Dict[str, Union[SelectionSpec, bool]]]):
def from_path(
cls,
path: Path,
renderer: SelectorRenderer,
renderer: BaseRenderer,
) -> "SelectorConfig":
try:
data = load_yaml_text(load_file_contents(str(path)))

View File

@@ -62,6 +62,8 @@ def get_project_config(
user_config = read_user_config(flags.PROFILES_DIR)
# Update flags
flags.set_from_args(args, user_config)
if cli_vars is None:
cli_vars = {}
profile = Profile.render_from_args(args, ProfileRenderer(cli_vars), profile_name)
# Generate a project
project = Project.from_project_root(

View File

@@ -1 +1,51 @@
# Contexts and Jinja rendering
Contexts are used for Jinja rendering. They include context methods, executable macros, and various settings that are available in Jinja.
The most common entrypoint to Jinja rendering in dbt is a method named `get_rendered`, which takes two arguments: templated code (string), and a context used to render it (dictionary).
The context is the bundle of information that is in "scope" when rendering Jinja-templated code. For instance, imagine a simple Jinja template:
```
{% set new_value = some_macro(some_variable) %}
```
Both `some_macro()` and `some_variable` must be defined in that context. Otherwise, it will raise an error when rendering.
Different contexts are used in different places because we allow access to different methods and data in different places. Executable SQL, for example, includes all available macros and the model being run. The variables and macros in scope for Jinja defined in yaml files is much more limited.
### Implementation
The context that is passed to Jinja is always in a dictionary format, not an actual class, so a `to_dict()` is executed on a context class before it is used for rendering.
Each context has a `generate_<name>_context` function to create the context. `ProviderContext` subclasses have different generate functions for parsing and for execution, so that certain functions (notably `ref`, `source`, and `config`) can return different results
### Hierarchy
All contexts inherit from the `BaseContext`, which includes "pure" methods (e.g. `tojson`), `env_var()`, and `var()` (but only CLI values, passed via `--vars`).
Methods available in parent contexts are also available in child contexts.
```
BaseContext -- core/dbt/context/base.py
SecretContext -- core/dbt/context/secret.py
TargetContext -- core/dbt/context/target.py
ConfiguredContext -- core/dbt/context/configured.py
SchemaYamlContext -- core/dbt/context/configured.py
DocsRuntimeContext -- core/dbt/context/configured.py
MacroResolvingContext -- core/dbt/context/configured.py
ManifestContext -- core/dbt/context/manifest.py
QueryHeaderContext -- core/dbt/context/manifest.py
ProviderContext -- core/dbt/context/provider.py
MacroContext -- core/dbt/context/provider.py
ModelContext -- core/dbt/context/provider.py
TestContext -- core/dbt/context/provider.py
```
### Contexts for configuration
Contexts for rendering "special" `.yml` (configuration) files:
- `SecretContext`: Supports "secret" env vars, which are prefixed with `DBT_ENV_SECRET_`. Used for rendering in `profiles.yml` and `packages.yml` ONLY. Secrets defined elsewhere will raise explicit errors.
- `TargetContext`: The same as `Base`, plus `target` (connection profile). Used most notably in `dbt_project.yml` and `selectors.yml`.
Contexts for other `.yml` files in the project:
- `SchemaYamlContext`: Supports `vars` declared on the CLI and in `dbt_project.yml`. Does not support custom macros, beyond `var()` + `env_var()` methods. Used for all `.yml` files, to define properties and configuration.
- `DocsRuntimeContext`: Standard `.yml` file context, plus `doc()` method (with all `docs` blocks in scope). Used to resolve `description` properties.

View File

@@ -24,38 +24,7 @@ import pytz
import datetime
import re
# Contexts in dbt Core
# Contexts are used for Jinja rendering. They include context methods,
# executable macros, and various settings that are available in Jinja.
#
# Different contexts are used in different places because we allow access
# to different methods and data in different places. Executable SQL, for
# example, includes the available macros and the model, while Jinja in
# yaml files is more limited.
#
# The context that is passed to Jinja is always in a dictionary format,
# not an actual class, so a 'to_dict()' is executed on a context class
# before it is used for rendering.
#
# Each context has a generate_<name>_context function to create the context.
# ProviderContext subclasses have different generate functions for
# parsing and for execution.
#
# Context class hierarchy
#
# BaseContext -- core/dbt/context/base.py
# SecretContext -- core/dbt/context/secret.py
# TargetContext -- core/dbt/context/target.py
# ConfiguredContext -- core/dbt/context/configured.py
# SchemaYamlContext -- core/dbt/context/configured.py
# DocsRuntimeContext -- core/dbt/context/configured.py
# MacroResolvingContext -- core/dbt/context/configured.py
# ManifestContext -- core/dbt/context/manifest.py
# QueryHeaderContext -- core/dbt/context/manifest.py
# ProviderContext -- core/dbt/context/provider.py
# MacroContext -- core/dbt/context/provider.py
# ModelContext -- core/dbt/context/provider.py
# TestContext -- core/dbt/context/provider.py
# See the `contexts` module README for more information on how contexts work
def get_pytz_module_context() -> Dict[str, Any]:

View File

@@ -7,6 +7,9 @@ from dbt.exceptions import raise_parsing_error
from dbt.logger import SECRET_ENV_PREFIX
SECRET_PLACEHOLDER = "$$$DBT_SECRET_START$$${}$$$DBT_SECRET_END$$$"
class SecretContext(BaseContext):
"""This context is used in profiles.yml + packages.yml. It can render secret
env vars that aren't usable elsewhere"""
@@ -18,21 +21,29 @@ class SecretContext(BaseContext):
If the default is None, raise an exception for an undefined variable.
In this context *only*, env_var will return the actual values of
env vars prefixed with DBT_ENV_SECRET_
In this context *only*, env_var will accept env vars prefixed with DBT_ENV_SECRET_.
It will return the name of the secret env var, wrapped in 'start' and 'end' identifiers.
The actual value will be subbed in later in SecretRenderer.render_value()
"""
return_value = None
if var in os.environ:
# if this is a 'secret' env var, just return the name of the env var
# instead of rendering the actual value here, to avoid any risk of
# Jinja manipulation. it will be subbed out later, in SecretRenderer.render_value
if var in os.environ and var.startswith(SECRET_ENV_PREFIX):
return SECRET_PLACEHOLDER.format(var)
elif var in os.environ:
return_value = os.environ[var]
elif default is not None:
return_value = default
if return_value is not None:
# do not save secret environment variables
# store env vars in the internal manifest to power partial parsing
# if it's a 'secret' env var, we shouldn't even get here
# but just to be safe — don't save secrets
if not var.startswith(SECRET_ENV_PREFIX):
self.env_vars[var] = return_value
# return the value even if its a secret
return return_value
else:
msg = f"Env var required but not provided: '{var}'"

View File

@@ -1,4 +1,3 @@
import colorama
from colorama import Style
import dbt.events.functions as this # don't worry I hate it too.
from dbt.events.base_types import NoStdOut, Event, NoFile, ShowException, Cache
@@ -50,14 +49,6 @@ format_color = True
format_json = False
invocation_id: Optional[str] = None
# Colorama needs some help on windows because we're using logger.info
# intead of print(). If the Windows env doesn't have a TERM var set,
# then we should override the logging stream to use the colorama
# converter. If the TERM var is set (as with Git Bash), then it's safe
# to send escape characters and no log handler injection is needed.
if sys.platform == "win32":
colorama.init(wrap=False)
def setup_event_logger(log_path, level_override=None):
# flags have been resolved, and log_path is known

View File

@@ -383,10 +383,11 @@ class FailedToConnectException(DatabaseException):
class CommandError(RuntimeException):
def __init__(self, cwd, cmd, message="Error running command"):
cmd_scrubbed = list(scrub_secrets(cmd_txt, env_secrets()) for cmd_txt in cmd)
super().__init__(message)
self.cwd = cwd
self.cmd = cmd
self.args = (cwd, cmd, message)
self.cmd = cmd_scrubbed
self.args = (cwd, cmd_scrubbed, message)
def __str__(self):
if len(self.cmd) == 0:
@@ -411,9 +412,9 @@ class CommandResultError(CommandError):
def __init__(self, cwd, cmd, returncode, stdout, stderr, message="Got a non-zero returncode"):
super().__init__(cwd, cmd, message)
self.returncode = returncode
self.stdout = stdout
self.stderr = stderr
self.args = (cwd, cmd, returncode, stdout, stderr, message)
self.stdout = scrub_secrets(stdout.decode("utf-8"), env_secrets())
self.stderr = scrub_secrets(stderr.decode("utf-8"), env_secrets())
self.args = (cwd, self.cmd, returncode, self.stdout, self.stderr, message)
def __str__(self):
return "{} running: {}".format(self.msg, self.cmd)
@@ -704,7 +705,6 @@ def missing_materialization(model, adapter_type):
def bad_package_spec(repo, spec, error_message):
msg = "Error checking out spec='{}' for repo {}\n{}".format(spec, repo, error_message)
raise InternalException(scrub_secrets(msg, env_secrets()))

View File

@@ -132,7 +132,7 @@
{% set check_cols_config = config['check_cols'] %}
{% set primary_key = config['unique_key'] %}
{% set invalidate_hard_deletes = config.get('invalidate_hard_deletes', false) %}
{% set updated_at = snapshot_get_time() %}
{% set updated_at = config.get('updated_at', snapshot_get_time()) %}
{% set column_added = false %}

View File

@@ -15,17 +15,16 @@ import colorama
import logbook
from dbt.dataclass_schema import dbtClassMixin
# Colorama needs some help on windows because we're using logger.info
# intead of print(). If the Windows env doesn't have a TERM var set,
# then we should override the logging stream to use the colorama
# converter. If the TERM var is set (as with Git Bash), then it's safe
# to send escape characters and no log handler injection is needed.
logging_stdout = sys.stdout
if sys.platform == "win32":
if not os.getenv("TERM"):
logging_stdout = colorama.AnsiToWin32(sys.stdout).stream
colorama.init(wrap=False)
# Colorama is needed for colored logs on Windows because we're using logger.info
# intead of print(). If the Windows env doesn't have a TERM var set or it is set to None
# (i.e. in the case of Git Bash on Windows- this emulates Unix), then it's safe to initialize
# Colorama with wrapping turned on which allows us to strip ANSI sequences from stdout.
# You can safely initialize Colorama for any OS and the coloring stays the same except
# when piped to anoter process for Linux and MacOS, then it loses the coloring. To combat
# that, we will just initialize Colorama when needed on Windows using a non-Unix terminal.
if sys.platform == "win32" and (not os.getenv("TERM") or os.getenv("TERM") == "None"):
colorama.init(wrap=True)
STDOUT_LOG_FORMAT = "{record.message}"
DEBUG_LOG_FORMAT = (
@@ -455,7 +454,7 @@ class DelayedFileHandler(logbook.RotatingFileHandler, FormatterMixin):
class LogManager(logbook.NestedSetup):
def __init__(self, stdout=logging_stdout, stderr=sys.stderr):
def __init__(self, stdout=sys.stdout, stderr=sys.stderr):
self.stdout = stdout
self.stderr = stderr
self._null_handler = logbook.NullHandler()

View File

@@ -573,6 +573,10 @@ class PartialParsing:
# doc source_file.nodes
self.schedule_nodes_for_parsing(source_file.nodes)
source_file.nodes = []
# Remove the file object
self.deleted_manifest.files[source_file.file_id] = self.saved_manifest.files.pop(
source_file.file_id
)
# Schema files -----------------------
# Changed schema files

View File

@@ -107,6 +107,9 @@ def test_data_dir(request):
# This contains the profile target information, for simplicity in setting
# up different profiles, particularly in the adapter repos.
# Note: because we load the profile to create the adapter, this
# fixture can't be used to test vars and env_vars or errors. The
# profile must be written out after the test starts.
@pytest.fixture(scope="class")
def dbt_profile_target():
return {
@@ -179,6 +182,7 @@ def dbt_project_yml(project_root, project_config_update, logs_dir):
if project_config_update:
project_config.update(project_config_update)
write_file(yaml.safe_dump(project_config), project_root, "dbt_project.yml")
return project_config
# Fixture to provide packages as either yaml or dictionary

View File

@@ -23,6 +23,7 @@ from dbt.events.test_types import IntegrationTestDebug
# read_file
# get_artifact
# update_config_file
# write_config_file
# get_unique_ids_in_results
# check_result_nodes_by_name
# check_result_nodes_by_unique_id
@@ -104,9 +105,9 @@ def copy_file(src_path, src, dest_path, dest) -> None:
# Used in tests when you want to remove a file from the project directory
def rm_file(src_path, src) -> None:
def rm_file(*paths) -> None:
# remove files from proj_path
os.remove(os.path.join(src_path, src))
os.remove(os.path.join(*paths))
# Used in tests to write out the string contents of a file to a
@@ -143,6 +144,13 @@ def update_config_file(updates, *paths):
write_file(new_yaml, *paths)
# Write new config file
def write_config_file(data, *paths):
if type(data) is dict:
data = yaml.safe_dump(data)
write_file(data, *paths)
# Get the unique_ids in dbt command results
def get_unique_ids_in_results(results):
unique_ids = []
@@ -273,13 +281,15 @@ def check_relation_types(adapter, relation_to_type):
# by doing a separate call for each set of tables/relations.
# Wraps check_relations_equal_with_relations by creating relations
# from the list of names passed in.
def check_relations_equal(adapter, relation_names):
def check_relations_equal(adapter, relation_names, compare_snapshot_cols=False):
if len(relation_names) < 2:
raise TestProcessingException(
"Not enough relations to compare",
)
relations = [relation_from_name(adapter, name) for name in relation_names]
return check_relations_equal_with_relations(adapter, relations)
return check_relations_equal_with_relations(
adapter, relations, compare_snapshot_cols=compare_snapshot_cols
)
# This can be used when checking relations in different schemas, by supplying
@@ -288,16 +298,17 @@ def check_relations_equal(adapter, relation_names):
# adapter.get_columns_in_relation
# adapter.get_rows_different_sql
# adapter.execute
def check_relations_equal_with_relations(adapter, relations):
def check_relations_equal_with_relations(adapter, relations, compare_snapshot_cols=False):
with get_connection(adapter):
basis, compares = relations[0], relations[1:]
# Skip columns starting with "dbt_" because we don't want to
# compare those, since they are time sensitive
# (unless comparing "dbt_" snapshot columns is explicitly enabled)
column_names = [
c.name
for c in adapter.get_columns_in_relation(basis)
if not c.name.lower().startswith("dbt_")
if not c.name.lower().startswith("dbt_") or compare_snapshot_cols
]
for relation in compares:

View File

@@ -615,7 +615,7 @@ def _connection_exception_retry(fn, max_attempts: int, attempt: int = 0):
fire_event(RecordRetryException(exc=exc))
fire_event(RetryExternalCall(attempt=attempt, max=max_attempts))
time.sleep(1)
_connection_exception_retry(fn, max_attempts, attempt + 1)
return _connection_exception_retry(fn, max_attempts, attempt + 1)
else:
raise ConnectionException("External connection exception occurred: " + str(exc))

View File

@@ -234,5 +234,5 @@ def _get_adapter_plugin_names() -> Iterator[str]:
yield plugin_name
__version__ = "1.1.0b1"
__version__ = "1.1.1rc2"
installed = get_installed_version()

View File

@@ -273,12 +273,12 @@ def parse_args(argv=None):
parser.add_argument("adapter")
parser.add_argument("--title-case", "-t", default=None)
parser.add_argument("--dependency", action="append")
parser.add_argument("--dbt-core-version", default="1.1.0b1")
parser.add_argument("--dbt-core-version", default="1.1.1rc2")
parser.add_argument("--email")
parser.add_argument("--author")
parser.add_argument("--url")
parser.add_argument("--sql", action="store_true")
parser.add_argument("--package-version", default="1.1.0b1")
parser.add_argument("--package-version", default="1.1.1rc2")
parser.add_argument("--project-version", default="1.0")
parser.add_argument("--no-dependency", action="store_false", dest="set_dependency")
parsed = parser.parse_args()

View File

@@ -25,7 +25,7 @@ with open(os.path.join(this_directory, "README.md")) as f:
package_name = "dbt-core"
package_version = "1.1.0b1"
package_version = "1.1.1rc2"
description = """With dbt, data analysts and engineers can build analytics \
the way engineers build applications."""
@@ -52,16 +52,16 @@ setup(
],
install_requires=[
"Jinja2==2.11.3",
"MarkupSafe==2.0.1",
"MarkupSafe>=0.23,<2.1",
"agate>=1.6,<1.6.4",
"click>=7.0,<9",
"colorama>=0.3.9,<0.4.5",
"hologram==0.0.14",
"hologram>=0.0.14,<=0.0.15",
"isodate>=0.6,<0.7",
"logbook>=1.5,<1.6",
"mashumaro==2.9",
"minimal-snowplow-tracker==0.0.2",
"networkx>=2.3,<3",
"networkx>=2.3,<2.8.4",
"packaging>=20.9,<22.0",
"sqlparse>=0.2.3,<0.5",
"dbt-extractor~=0.4.1",

View File

@@ -14,12 +14,12 @@ FROM --platform=$build_for python:3.10.3-slim-bullseye as base
# N.B. The refs updated automagically every release via bumpversion
# N.B. dbt-postgres is currently found in the core codebase so a value of dbt-core@<some_version> is correct
ARG dbt_core_ref=dbt-core@v1.1.0b1
ARG dbt_postgres_ref=dbt-core@v1.1.0b1
ARG dbt_redshift_ref=dbt-redshift@v1.0.0
ARG dbt_bigquery_ref=dbt-bigquery@v1.0.0
ARG dbt_snowflake_ref=dbt-snowflake@v1.0.0
ARG dbt_spark_ref=dbt-spark@v1.0.0
ARG dbt_core_ref=dbt-core@v1.1.1rc2
ARG dbt_postgres_ref=dbt-core@v1.1.1rc2
ARG dbt_redshift_ref=dbt-redshift@v1.1.0
ARG dbt_bigquery_ref=dbt-bigquery@v1.1.0
ARG dbt_snowflake_ref=dbt-snowflake@v1.1.0
ARG dbt_spark_ref=dbt-spark@v1.1.0
# special case args
ARG dbt_spark_version=all
ARG dbt_third_party

View File

@@ -1 +1 @@
version = "1.1.0b1"
version = "1.1.1rc2"

View File

@@ -41,7 +41,7 @@ def _dbt_psycopg2_name():
package_name = "dbt-postgres"
package_version = "1.1.0b1"
package_version = "1.1.1rc2"
description = """The postgres adpter plugin for dbt (data build tool)"""
this_directory = os.path.abspath(os.path.dirname(__file__))

Some files were not shown because too many files have changed in this diff Show More