Compare commits

...

39 Commits

Author SHA1 Message Date
Github Build Bot
193a9dcb18 Bumping version to 1.1.5 and generate changelog 2023-04-19 13:16:22 +00:00
Michelle Ark
924175938c sqlparse <0.4.4 (#7402)
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-04-19 05:13:37 -07:00
Quigley Malcolm
cdc0886bb9 [BACKPORT 1.1.latest] Improved event log msg stringification (#7338)
* Add tests for logging jinja2.Undefined objects

[CT-2259](https://github.com/dbt-labs/dbt-core/issues/7108) identifies
an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined
object is attempted to be logged. This generally happened in the form
of `{{ log(undefined_variable, info=True) }}`. This commit adding this
test exists for two reasons
1. Ensure we don't have a regression in this going forward
2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3

* Add tests for checking `DBT_ENV_SECRET_`s don't break logging

[CT-1783](https://github.com/dbt-labs/dbt-core/issues/6568) describes
a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all
`{{ log("logging stuff", info=True) }}` invocations break. This commit
adds a test for this for two reasons:
1. Ensure we don't regress to this behavior going forward
2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3

* Add changie info for changes

* Ensure `log()` calls from jinja ensure the msg is stringified

The simplest way to resolve [CT-2259](https://github.com/dbt-labs/dbt-core/issues/7108)
and [CT-1783](https://github.com/dbt-labs/dbt-core/issues/6568) in backports
to dbt-core < 1.4 is to ensure `msg` in `BaseContext.log()` is stringified.
2023-04-18 13:33:38 -07:00
FishtownBuildBot
8187947458 [Automated] Merged prep-release/1.1.4_4294186265 into target 1.1.latest during release process 2023-02-28 09:15:31 -06:00
Github Build Bot
96028df5fc Bumping version to 1.1.4 and generate changelog 2023-02-28 14:50:16 +00:00
Sam Debruyn
9a7527aacf fix: add pytz dependency (#7081) 2023-02-28 09:32:08 -05:00
Alexander Smolyakov
29b78a747d [CI/CD] Backport release workflow to 1.1.latest (#6790)
* [CI/CD] Update release workflow and introduce workflow for nightly releases (#6602)

* Add release workflows

* Update nightly-release.yml

* Set default `test_run` value to `true`

* Update .bumpversion.cfg

* Resolve review comment

- Update workflow docs
- Change workflow name
- Set `test_run` default value to `true`

* Update Slack secret

* PyPI

* Update release workflow (#6778)

- Update AWS secrets
- Rework condition for Slack notification

* update regex to match all iterations (#6839)

* update regex to match all iterations

* convert to num to match all adapters

* add comments, remove extra .

* clarify with more comments

* Update .bumpversion.cfg

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>

---------

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
# Conflicts:
#	.bumpversion.cfg

* put back correct version

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-02-03 13:02:15 -06:00
Emily Rockman
bea3ce90c7 [BACKPORT] fix contributor list generation (#6799) (#6807)
* Remove PR from most changelog kinds (#6374)

* update changie to require issue or pr, and allow multiple

* remove extraneous data from changelog files.

* allow for multiple PR/issues to be entered

* update contributing guide

* remove issue number from bot changelogs

* update format of PR

* fix dependency changelogs

* remove extra line

* remove extra lines, tweak contributor wording

* Update CONTRIBUTING.md

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* fix contributor list generation (#6799)

* fix merge conflict

---------

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-01-31 21:39:23 -06:00
github-actions[bot]
e660fefc52 Bumping version to 1.1.3 and generate changelog (#6524)
* Bumping version to 1.1.3rc1 and generate CHANGELOG

* Bumping version to 1.1.3 and generate CHANGELOG (#6525)

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-01-04 21:53:07 -05:00
github-actions[bot]
7e23a3e05d Update Version Bump to include Homebrew in PATH (#5963) (#5993) (#6523)
(cherry picked from commit f903aed3e0)

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-01-04 21:16:00 -05:00
Gerda Shank
6dbea380de [Backport 1.1.latest] Partial parsing bug with empty schema file - en… (#6516)
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-01-04 16:38:52 -05:00
Peter Webb
cfd6a6cc93 Paw/ct 1643 tox passenv (#6445)
* Break tox's passenv parameters into multiple lines

* ct-1643: remove trailing whitespace -_-

* CT-1643: Add missed parameters

* CT-1643: more whitespace fixing
2022-12-14 10:47:31 -05:00
Jeremy Cohen
2298c9fbfa Disable test_postgres_package_redirect_fail (#6410) 2022-12-08 15:27:30 +01:00
leahwicz
9f865fb148 Updating version to remove rc1 (#6373) 2022-12-05 09:06:58 -05:00
github-actions[bot]
b6c9221100 s/gitlab/github for flake8 precommit repo (#6252) (#6253)
(cherry picked from commit eae98677b9)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2022-11-15 11:24:06 -05:00
Emily Rockman
ad710f51cd add python version and upgrade action (#6204) (#6209)
(cherry picked from commit c3ccbe3357)
2022-11-04 09:33:29 -05:00
github-actions[bot]
8ef7259d9c Bumping version to 1.1.2 and generate changelog (#5575)
* Bumping version to 1.1.2 and generate CHANGELOG

* Remove newline

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-07-28 15:40:23 -04:00
leahwicz
424b454918 Add changelog and whitespace fix to version bump Action (#5563) (#5573)
* Add changelog and whitespace fix to version bump Action

* Fixing whitespace

* Remove tabs

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Update .github/workflows/version-bump.yml

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Updating per comments

* Fix whitespace

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-07-28 14:37:55 -04:00
github-actions[bot]
437b763390 Bumping version to 1.1.2rc1 (#5498)
* Bumping version to 1.1.2rc1

* Fix whitespace

* Update changelog

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-07-20 09:00:30 -04:00
Jeremy Cohen
d10ebecdb6 Backport #5382 to 1.1.latest (#5415)
* Backport #5382 to 1.1.latest

* Rm changelog entry for previous change
2022-06-28 17:26:47 +02:00
Jeremy Cohen
8052331439 Declare compatibility for previous artifact versions (#5346) (#5383)
* Add functional test

* Add is_compatible_version logic

* Add changelog entry

* Fix mypy
2022-06-16 18:22:43 +02:00
github-actions[bot]
50d2981320 Bumping version to 1.1.1 (#5377)
* Bumping version to 1.1.1

* Fix whitespace

* Add Changelog

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-06-15 10:39:50 -04:00
github-actions[bot]
5ef0442654 Bumping version to 1.1.1rc2 (#5342)
* Bumping version to 1.1.1rc2

* Remove whitespace

* Add Changelog

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-06-07 11:28:46 -04:00
github-actions[bot]
4d63fb2ed2 Fixing Windows color regression (#5327) (#5340)
* Fixing Windows color regression

* Cleaning up logic

* Consolidating logic to the logger

* Cleaning up vars

* Updating comment

* Removing unused import

* Fixing whitespace

* Adding changelog

(cherry picked from commit e48f7ab32e)

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-06-07 09:32:28 -04:00
Jeremy Cohen
413a19abe6 Backports: networkx, context cleanup (#5335)
* Update context readme, small code cleanup (#5334)

* Pin networkx dependency to <2.8.4 for v1.1
2022-06-07 11:00:45 +02:00
github-actions[bot]
a3dcc14f87 [Backport 1.1.latest] CT-159 Remove docs file from manifest when deleting doc node (#5293)
* CT-159 Remove docs file from manifest when deleting doc node (#5270)

* Convert partial parsing docs tests

* Failing test

* Remove doc file from manifest when doc node is removed

* Changie

(cherry picked from commit cda88d1948)

* Fix rm_file method

* comment out pip upgrade

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-06-01 10:31:38 -04:00
github-actions[bot]
5e298b1854 Bumping version to 1.1.1rc1 (#5285)
* Bumping version to 1.1.1rc1

* Remove whitespace

* Changelog update

* Fixing adapter versions

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-05-20 13:28:51 -04:00
github-actions[bot]
c74e9dde2b When parsing 'all_sources' should be a list of unique dirs (#5176) (#5256)
* When parsing 'all_sources' should be a list of unique dirs

* Changie

* Fix some unit tests of all_source_paths

* Convert 039_config_tests

* Remove old 039_config_tests

* Add test for duplicate directories in 'all_source_files'

(cherry picked from commit f633e9936f)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-05-16 16:00:53 -04:00
github-actions[bot]
9c6bdcabf8 Bumping hologram version (#5218) (#5226)
* Bumping hologram version

* Add automated changelog yaml from template

* Updating issue

* Loosen requirement range

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
(cherry picked from commit aa8115aa5e)

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-05-11 11:56:43 -04:00
leahwicz
4ae2e41cc6 Flexibilize MarkupSafe pinned version (#5039) (#5111)
* Flexibilize MarkupSafe pinned version

The current `MarkupSafe` pinned version has been added in #4746 as a
temporary fix for #4745.

However, the current restrictive approach isn't compatible with other
libraries that could require an even older version of `MarkupSafe`, like
Airflow `2.2.2` [0], which requires `markupsafe>=1.1.1, <2.0`.

To avoid that issue, we can allow a greater range of supported
`MarkupSafe` versions. Considering the direct dependency `dbt-core` has
is `Jinja2==2.11.3`, we can use its pinning as the lower bound, which is
`MarkupSafe>=0.23` [1].

This fix should be also backported this to `1.0.latest` for inclusion in
the next v1.0 patch.

[0] https://github.com/adamantike/airflow/blob/2.2.2/setup.cfg#L125
[1] https://github.com/pallets/jinja/blob/2.11.3/setup.py#L53

Co-authored-by: Michael Manganiello <adamantike@users.noreply.github.com>
2022-05-09 14:49:56 -04:00
github-actions[bot]
cbd4570c67 Bumping version to 1.1.0 (#5186)
* Bumping version to 1.1.0

* Updating changelog

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-04-28 12:54:30 -04:00
github-actions[bot]
5ae33b90bf Bumping version to 1.1.0rc3 (#5166)
* Bumping version to 1.1.0rc3

* Update changelog

* Fixing spacing issue

* Removing more whitespace

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-04-26 15:25:39 -04:00
github-actions[bot]
37d78338c2 [Backport 1.1.latest] Use yaml renderer (with target context) for rendering selectors (#5161)
* Use yaml renderer (with target context) for rendering selectors (#5136)

* Use yaml renderer (with target context) for rendering selectors

* Changie

* Convert cli_vars tests

* Add test for var in profiles

* Add test for cli vars in packages

* Add test for vars in selectors

(cherry picked from commit 1f898c859a)

* Tweak cli_vars_in_packages test to do a run and build an adapter

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-04-26 14:13:16 -04:00
github-actions[bot]
698420b420 Even more scrubbing (#5152) (#5156)
* Even more scrubbing

* Changelog entry

* Even more

* remove reduendent scrub

* remove reduendent scrub

* fix encoding issue

* keep scrubbed log in args

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
(cherry picked from commit ce0bcc08a6)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2022-04-26 10:59:07 -06:00
Emily Rockman
e23f0e0747 fix retry logic failures (#5137) (#5148)
* fix retry logic failures

* changelog

* add tests to make sure data is getting where it needs to

* rename file

* remove duplicate file

(cherry picked from commit 31a3f2bdee)
2022-04-25 10:45:35 -05:00
github-actions[bot]
106da05db7 Bumping version to 1.1.0rc2 (#5129)
* Bumping version to 1.1.0rc2

* release changelog

* fix trailing space

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2022-04-21 09:52:08 -06:00
Chenyu Li
09a396a731 Restore ability to utilize updated_at for check_cols snapshots (#5077) (#5126)
* Restore ability to configure and utilize `updated_at` for snapshots using the check_cols strategy

* Changelog entry

* Optional comparison of column names starting with `dbt_`

* Functional test for check cols snapshots using `updated_at`

* Comments to explain the test implementation

(cherry picked from commit d09459c980)

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2022-04-21 09:06:04 -06:00
Emily Rockman
9c233f27ab remove commas from changelog yaml (#5113) 2022-04-19 19:57:14 -05:00
github-actions[bot]
a09bc28768 Bumping version to 1.1.0rc1 (#5044)
* Bumping version to 1.1.0rc1

* Adding changelog

* Updating adapter versions for Docker

* Remove extra spaces

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-04-12 17:10:57 -04:00
168 changed files with 2574 additions and 1907 deletions

View File

@@ -1,13 +1,15 @@
[bumpversion]
current_version = 1.1.0b1
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
((?P<prekind>a|b|rc)
(?P<pre>\d+) # pre-release version num
current_version = 1.1.5
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
(((?P<prekind>a|b|rc) # optional pre-release type
?(?P<num>[\d]+?)) # optional pre-release version number
\.?(?P<nightly>[a-z0-9]+\+[a-z]+)? # optional nightly release indicator
)?
serialize =
{major}.{minor}.{patch}{prekind}{pre}
{major}.{minor}.{patch}{prekind}{num}.{nightly}
{major}.{minor}.{patch}{prekind}{num}
{major}.{minor}.{patch}
commit = False
tag = False
@@ -21,9 +23,11 @@ values =
rc
final
[bumpversion:part:pre]
[bumpversion:part:num]
first_value = 1
[bumpversion:part:nightly]
[bumpversion:file:core/setup.py]
[bumpversion:file:core/dbt/version.py]

View File

@@ -1,50 +0,0 @@
## dbt-core 1.1.0-b1 - March 22, 2022
### Breaking Changes
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter ([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
### Features
- Change behaviour of `non_null` test so that it only `select`s all columns if `--store-failures` is enabled. ([#4769](https://github.com/dbt-labs/dbt-core/issues/4769), [#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
- Testing framework for dbt adapter testing ([#4730](https://github.com/dbt-labs/dbt-core/issues/4730), [#4846](https://github.com/dbt-labs/dbt-core/pull/4846))
- Allow unique key to take a list implementation for postgres/redshift ([#4738](https://github.com/dbt-labs/dbt-core/issues/4738), [#4858](https://github.com/dbt-labs/dbt-core/pull/4858))
- Added Support for Semantic Versioning ([#4453](https://github.com/dbt-labs/dbt-core/issues/4453), [#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- New Dockerfile to support specific db adapters and platforms. See docker/README.md for details ([#4495](https://github.com/dbt-labs/dbt-core/issues/4495), [#4487](https://github.com/dbt-labs/dbt-core/pull/4487))
- Allow unique_key to take a list ([#2479](https://github.com/dbt-labs/dbt-core/issues/2479), [#4618](https://github.com/dbt-labs/dbt-core/pull/4618))
- Add `--quiet` global flag and `print` Jinja function ([#3451](https://github.com/dbt-labs/dbt-core/issues/3451), [#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- Add space before justification periods ([#4737](https://github.com/dbt-labs/dbt-core/issues/4737), [#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
### Fixes
- Fix bug causing empty node level meta, snapshot config errors ([#4459](https://github.com/dbt-labs/dbt-core/issues/4459), [#4726](https://github.com/dbt-labs/dbt-core/pull/4726))
- Catch all Requests Exceptions on deps install to attempt retries. Also log the exceptions hit. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4865](https://github.com/dbt-labs/dbt-core/pull/4865))
- Use cli_vars instead of context to create package and selector renderers ([#4876](https://github.com/dbt-labs/dbt-core/issues/4876), [#4878](https://github.com/dbt-labs/dbt-core/pull/4878))
- depend on new dbt-extractor version with fixed github links ([#4891](https://github.com/dbt-labs/dbt-core/issues/4891), [#4890](https://github.com/dbt-labs/dbt-core/pull/4890))
- Update bumpervsion config to stop looking for missing setup.py ([#-1](https://github.com/dbt-labs/dbt-core/issues/-1), [#4896](https://github.com/dbt-labs/dbt-core/pull/4896))
- Corrected permissions settings for docker release workflow ([#4902](https://github.com/dbt-labs/dbt-core/issues/4902), [#4903](https://github.com/dbt-labs/dbt-core/pull/4903))
- User wasn't asked for permission to overwite a profile entry when running init inside an existing project ([#4375](https://github.com/dbt-labs/dbt-core/issues/4375), [#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- Add project name validation to `dbt init` ([#4490](https://github.com/dbt-labs/dbt-core/issues/4490), [#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- Allow override of string and numeric types for adapters. ([#4603](https://github.com/dbt-labs/dbt-core/issues/4603), [#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- A change in secret environment variables won't trigger a full reparse ([#4650](https://github.com/dbt-labs/dbt-core/issues/4650), [#4665](https://github.com/dbt-labs/dbt-core/pull/4665))
- Fix misspellings and typos in docstrings ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
### Under the Hood
- Automate changelog generation with changie ([#4652](https://github.com/dbt-labs/dbt-core/issues/4652), [#4743](https://github.com/dbt-labs/dbt-core/pull/4743))
- Fix broken links for changelog generation and tweak GHA to only post a comment once when changelog entry is missing. ([#4848](https://github.com/dbt-labs/dbt-core/issues/4848), [#4857](https://github.com/dbt-labs/dbt-core/pull/4857))
- Add support for Python 3.10 ([#4562](https://github.com/dbt-labs/dbt-core/issues/4562), [#4866](https://github.com/dbt-labs/dbt-core/pull/4866))
- Enable more dialects to snapshot sources with added columns, even those that don't support boolean datatypes ([#4488](https://github.com/dbt-labs/dbt-core/issues/4488), [#4871](https://github.com/dbt-labs/dbt-core/pull/4871))
- Testing cleanup ([#3648](https://github.com/dbt-labs/dbt-core/issues/3648), [#4509](https://github.com/dbt-labs/dbt-core/pull/4509))
- Clean up test deprecation warnings ([#3988](https://github.com/dbt-labs/dbt-core/issues/3988), [#4556](https://github.com/dbt-labs/dbt-core/pull/4556))
- Use mashumaro for serialization in event logging ([#4504](https://github.com/dbt-labs/dbt-core/issues/4504), [#4505](https://github.com/dbt-labs/dbt-core/pull/4505))
- Drop support for Python 3.7.0 + 3.7.1 ([#4584](https://github.com/dbt-labs/dbt-core/issues/4584), [#4585](https://github.com/dbt-labs/dbt-core/pull/4585))
- Drop support for Python <3.7.2 ([#4584](https://github.com/dbt-labs/dbt-core/issues/4584), [#4643](https://github.com/dbt-labs/dbt-core/pull/4643))
- Re-format codebase (except tests) using pre-commit hooks ([#3195](https://github.com/dbt-labs/dbt-core/issues/3195), [#4697](https://github.com/dbt-labs/dbt-core/pull/4697))
- Add deps module README ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4686](https://github.com/dbt-labs/dbt-core/pull/4686))
- Initial conversion of tests to pytest ([#4690](https://github.com/dbt-labs/dbt-core/issues/4690), [#4691](https://github.com/dbt-labs/dbt-core/pull/4691))
- Fix errors in Windows for tests/functions ([#4782](https://github.com/dbt-labs/dbt-core/issues/4782), [#4767](https://github.com/dbt-labs/dbt-core/pull/4767))
### Docs
- Resolve errors related to operations preventing DAG from generating in the docs. Also patch a spark issue to allow search to filter accurately past the missing columns. ([#4578](https://github.com/dbt-labs/dbt-core/issues/4578), [#4763](https://github.com/dbt-labs/dbt-core/pull/4763))
### Contributors
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- [@alswang18](https://github.com/alswang18) ([#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- [@amirkdv](https://github.com/amirkdv) ([#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- [@ehmartens](https://github.com/ehmartens) ([#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- [@joellabes](https://github.com/joellabes) ([#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- [@kazanzhy](https://github.com/kazanzhy) ([#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- [@mdesmet](https://github.com/mdesmet) ([#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- [@versusfacit](https://github.com/versusfacit) ([#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- [@willbowditch](https://github.com/willbowditch) ([#4777](https://github.com/dbt-labs/dbt-core/pull/4777))

96
.changes/1.1.0.md Normal file
View File

@@ -0,0 +1,96 @@
## dbt-core 1.1.0 - April 28, 2022
### Breaking Changes
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter
([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- For adapter plugin maintainers only: Internal adapter methods `set_relations_cache` + `_relations_cache_for_schemas` each take an additional argument, for use with experimental `CACHE_SELECTED_ONLY` config ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
### Features
- Change behaviour of `non_null` test so that it only `select`s all columns if `--store-failures` is enabled. ([#4769](https://github.com/dbt-labs/dbt-core/issues/4769), [#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
- Testing framework for dbt adapter testing ([#4730](https://github.com/dbt-labs/dbt-core/issues/4730), [#4846](https://github.com/dbt-labs/dbt-core/pull/4846))
- Allow unique key to take a list implementation for postgres/redshift ([#4738](https://github.com/dbt-labs/dbt-core/issues/4738), [#4858](https://github.com/dbt-labs/dbt-core/pull/4858))
- Add `--cache_selected_only` flag to cache schema object of selected models only. ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
- Support custom names for generic tests ([#3348](https://github.com/dbt-labs/dbt-core/issues/3348), [#4898](https://github.com/dbt-labs/dbt-core/pull/4898))
- Added Support for Semantic Versioning ([#4453](https://github.com/dbt-labs/dbt-core/issues/4453), [#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- New Dockerfile to support specific db adapters and platforms. See docker/README.md for details ([#4495](https://github.com/dbt-labs/dbt-core/issues/4495), [#4487](https://github.com/dbt-labs/dbt-core/pull/4487))
- Allow unique_key to take a list ([#2479](https://github.com/dbt-labs/dbt-core/issues/2479), [#4618](https://github.com/dbt-labs/dbt-core/pull/4618))
- Add `--quiet` global flag and `print` Jinja function ([#3451](https://github.com/dbt-labs/dbt-core/issues/3451), [#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- Add space before justification periods ([#4737](https://github.com/dbt-labs/dbt-core/issues/4737), [#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- Enable dbt jobs to run downstream models based on fresher sources. Compare the source freshness results between previous and current state. If any source is fresher and/or new in current vs. previous state, dbt will run and test the downstream models in scope. Example command: `dbt build --select source_status:fresher+` ([#4050](https://github.com/dbt-labs/dbt-core/issues/4050), [#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- converting unique key as list tests to new pytest format ([#4882](https://github.com/dbt-labs/dbt-core/issues/4882), [#4958](https://github.com/dbt-labs/dbt-core/pull/4958))
- Add a variable called selected_resources in the Jinja context containing a list of all the resources matching the nodes for the --select, --exclude and/or --selector parameters. ([#3471](https://github.com/dbt-labs/dbt-core/issues/3471), [#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
- Support the DO_NOT_TRACK environment variable from the consoledonottrack.com initiative ([#3540](https://github.com/dbt-labs/dbt-core/issues/3540), [#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
- Add `--no-print` global flag ([#4710](https://github.com/dbt-labs/dbt-core/issues/4710), [#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
- add enabled as a source config ([#3662](https://github.com/dbt-labs/dbt-core/issues/3662), [#5008](https://github.com/dbt-labs/dbt-core/pull/5008))
### Fixes
- Fix bug causing empty node level meta, snapshot config errors ([#4459](https://github.com/dbt-labs/dbt-core/issues/4459), [#4726](https://github.com/dbt-labs/dbt-core/pull/4726))
- Inconsistent timestamps between inserted/updated and deleted rows in snapshots ([#4347](https://github.com/dbt-labs/dbt-core/issues/4347), [#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
- Catch all Requests Exceptions on deps install to attempt retries. Also log the exceptions hit. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4865](https://github.com/dbt-labs/dbt-core/pull/4865))
- Use cli_vars instead of context to create package and selector renderers ([#4876](https://github.com/dbt-labs/dbt-core/issues/4876), [#4878](https://github.com/dbt-labs/dbt-core/pull/4878))
- depend on new dbt-extractor version with fixed github links ([#4891](https://github.com/dbt-labs/dbt-core/issues/4891), [#4890](https://github.com/dbt-labs/dbt-core/pull/4890))
- Update bumpervsion config to stop looking for missing setup.py ([#-1](https://github.com/dbt-labs/dbt-core/issues/-1), [#4896](https://github.com/dbt-labs/dbt-core/pull/4896))
- Corrected permissions settings for docker release workflow ([#4902](https://github.com/dbt-labs/dbt-core/issues/4902), [#4903](https://github.com/dbt-labs/dbt-core/pull/4903))
- User wasn't asked for permission to overwite a profile entry when running init inside an existing project ([#4375](https://github.com/dbt-labs/dbt-core/issues/4375), [#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- Add project name validation to `dbt init` ([#4490](https://github.com/dbt-labs/dbt-core/issues/4490), [#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- Allow override of string and numeric types for adapters. ([#4603](https://github.com/dbt-labs/dbt-core/issues/4603), [#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- A change in secret environment variables won't trigger a full reparse ([#4650](https://github.com/dbt-labs/dbt-core/issues/4650), [#4665](https://github.com/dbt-labs/dbt-core/pull/4665))
- Fix misspellings and typos in docstrings ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- Catch more cases to retry package retrieval for deps pointing to the hub. Also start to cache the package requests. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4982](https://github.com/dbt-labs/dbt-core/pull/4982))
- Make the warning message for a full event deque more descriptive ([#4962](https://github.com/dbt-labs/dbt-core/issues/4962), [#5011](https://github.com/dbt-labs/dbt-core/pull/5011))
- Fix hard delete snapshot test ([#4916](https://github.com/dbt-labs/dbt-core/issues/4916), [#5020](https://github.com/dbt-labs/dbt-core/pull/5020))
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- Use yaml renderer (with target context) for rendering selectors ([#5131](https://github.com/dbt-labs/dbt-core/issues/5131), [#5136](https://github.com/dbt-labs/dbt-core/pull/5136))
- Fix retry logic to return values after initial try ([#5023](https://github.com/dbt-labs/dbt-core/issues/5023), [#5137](https://github.com/dbt-labs/dbt-core/pull/5137))
- Scrub secret env vars from CommandError in exception stacktrace ([#5151](https://github.com/dbt-labs/dbt-core/issues/5151), [#5152](https://github.com/dbt-labs/dbt-core/pull/5152))
### Docs
- Resolve errors related to operations preventing DAG from generating in the docs. Also patch a spark issue to allow search to filter accurately past the missing columns. ([#4578](https://github.com/dbt-labs/dbt-core/issues/4578), [#4763](https://github.com/dbt-labs/dbt-core/pull/4763))
- Fixed capitalization in UI for exposures of `type: ml` ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- List packages and tags in alphabetical order ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- Bump jekyll from 3.8.7 to 3.9.0 ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- Updated docker README to reflect necessity of using BuildKit ([#4990](https://github.com/dbt-labs/dbt-core/issues/4990), [#5018](https://github.com/dbt-labs/dbt-core/pull/5018))
### Under the Hood
- Automate changelog generation with changie ([#4652](https://github.com/dbt-labs/dbt-core/issues/4652), [#4743](https://github.com/dbt-labs/dbt-core/pull/4743))
- add performance regression testing runner without orchestration ([#4021](https://github.com/dbt-labs/dbt-core/issues/4021), [#4602](https://github.com/dbt-labs/dbt-core/pull/4602))
- Fix broken links for changelog generation and tweak GHA to only post a comment once when changelog entry is missing. ([#4848](https://github.com/dbt-labs/dbt-core/issues/4848), [#4857](https://github.com/dbt-labs/dbt-core/pull/4857))
- Add support for Python 3.10 ([#4562](https://github.com/dbt-labs/dbt-core/issues/4562), [#4866](https://github.com/dbt-labs/dbt-core/pull/4866))
- Enable more dialects to snapshot sources with added columns, even those that don't support boolean datatypes ([#4488](https://github.com/dbt-labs/dbt-core/issues/4488), [#4871](https://github.com/dbt-labs/dbt-core/pull/4871))
- Add Graph Compilation and Adapter Cache tracking ([#4625](https://github.com/dbt-labs/dbt-core/issues/4625), [#4912](https://github.com/dbt-labs/dbt-core/pull/4912))
- Testing cleanup ([#3648](https://github.com/dbt-labs/dbt-core/issues/3648), [#4509](https://github.com/dbt-labs/dbt-core/pull/4509))
- Clean up test deprecation warnings ([#3988](https://github.com/dbt-labs/dbt-core/issues/3988), [#4556](https://github.com/dbt-labs/dbt-core/pull/4556))
- Use mashumaro for serialization in event logging ([#4504](https://github.com/dbt-labs/dbt-core/issues/4504), [#4505](https://github.com/dbt-labs/dbt-core/pull/4505))
- Drop support for Python 3.7.0 + 3.7.1 ([#4584](https://github.com/dbt-labs/dbt-core/issues/4584), [#4585](https://github.com/dbt-labs/dbt-core/pull/4585))
- Drop support for Python <3.7.2 ([#4584](https://github.com/dbt-labs/dbt-core/issues/4584), [#4643](https://github.com/dbt-labs/dbt-core/pull/4643))
- Re-format codebase (except tests) using pre-commit hooks ([#3195](https://github.com/dbt-labs/dbt-core/issues/3195), [#4697](https://github.com/dbt-labs/dbt-core/pull/4697))
- Add deps module README ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4686](https://github.com/dbt-labs/dbt-core/pull/4686))
- Initial conversion of tests to pytest ([#4690](https://github.com/dbt-labs/dbt-core/issues/4690), [#4691](https://github.com/dbt-labs/dbt-core/pull/4691))
- Fix errors in Windows for tests/functions ([#4782](https://github.com/dbt-labs/dbt-core/issues/4782), [#4767](https://github.com/dbt-labs/dbt-core/pull/4767))
- Create a dbt.tests.adapter release when releasing dbt and postgres ([#4812](https://github.com/dbt-labs/dbt-core/issues/4812), [#4948](https://github.com/dbt-labs/dbt-core/pull/4948))
- update docker image to use python 3.10.3 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4963](https://github.com/dbt-labs/dbt-core/pull/4963))
- updates black to 22.3.0 which fixes dependency incompatibility when running with precommit. ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4972](https://github.com/dbt-labs/dbt-core/pull/4972))
- Adds config util for ad-hoc creation of project objs or dicts ([#4808](https://github.com/dbt-labs/dbt-core/issues/4808), [#4981](https://github.com/dbt-labs/dbt-core/pull/4981))
- Remove TableComparison and convert existing calls to use dbt.tests.util ([#4778](https://github.com/dbt-labs/dbt-core/issues/4778), [#4986](https://github.com/dbt-labs/dbt-core/pull/4986))
- Remove unneeded create_schema in snapshot materialization ([#4742](https://github.com/dbt-labs/dbt-core/issues/4742), [#4993](https://github.com/dbt-labs/dbt-core/pull/4993))
- Added .git-blame-ignore-revs file to mask re-formmating commits from git blame ([#5004](https://github.com/dbt-labs/dbt-core/issues/5004), [#5019](https://github.com/dbt-labs/dbt-core/pull/5019))
- Convert version tests to pytest ([#5024](https://github.com/dbt-labs/dbt-core/issues/5024), [#5026](https://github.com/dbt-labs/dbt-core/pull/5026))
- Updating tests and docs to show that we now support Python 3.10 ([#4974](https://github.com/dbt-labs/dbt-core/issues/4974), [#5025](https://github.com/dbt-labs/dbt-core/pull/5025))
- Update --version output and logic ([#4724](https://github.com/dbt-labs/dbt-core/issues/4724), [#5029](https://github.com/dbt-labs/dbt-core/pull/5029))
- ([#5033](https://github.com/dbt-labs/dbt-core/issues/5033), [#5032](https://github.com/dbt-labs/dbt-core/pull/5032))
### Contributors
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- [@agoblet](https://github.com/agoblet) ([#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
- [@alswang18](https://github.com/alswang18) ([#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- [@amirkdv](https://github.com/amirkdv) ([#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- [@anaisvaillant](https://github.com/anaisvaillant) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@b-per](https://github.com/b-per) ([#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
- [@dbeatty10](https://github.com/dbeatty10) ([#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@ehmartens](https://github.com/ehmartens) ([#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- [@joellabes](https://github.com/joellabes) ([#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- [@jonstacks](https://github.com/jonstacks) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- [@kadero](https://github.com/kadero) ([#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
- [@karunpoudel](https://github.com/karunpoudel) ([#4860](https://github.com/dbt-labs/dbt-core/pull/4860), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
- [@kazanzhy](https://github.com/kazanzhy) ([#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- [@matt-winkler](https://github.com/matt-winkler) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@mdesmet](https://github.com/mdesmet) ([#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- [@pgoslatara](https://github.com/pgoslatara) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- [@poloaraujo](https://github.com/poloaraujo) ([#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
- [@sungchun12](https://github.com/sungchun12) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@willbowditch](https://github.com/willbowditch) ([#4777](https://github.com/dbt-labs/dbt-core/pull/4777))

View File

@@ -1,11 +0,0 @@
kind: Breaking Changes
body: >
**Relevant to maintainers of adapter plugins _only_:**
The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints.
(Previously, they could return a string.) We encourage you to update your methods to return an object of
class `AdapterResponse`, or implement a subclass specific to your adapter
time: 2022-03-16T03:13:11.656007-07:00
custom:
Author: versusfacit
Issue: "4499"
PR: "4869"

View File

@@ -1,8 +0,0 @@
kind: Docs
body: Resolve errors related to operations preventing DAG from generating in the docs. Also
patch a spark issue to allow search to filter accurately past the missing columns.
time: 2022-03-07T20:31:05.557064-06:00
custom:
Author: emmyoop
Issue: "4578"
PR: "4763"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Testing framework for dbt adapter testing
time: 2022-03-09T14:26:45.828295-05:00
custom:
Author: gshank
Issue: "4730"
PR: "4846"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Change behaviour of `non_null` test so that it only `select`s all columns if
`--store-failures` is enabled.
time: 2022-03-09T16:38:17.160568Z
custom:
Author: willbowditch
Issue: "4769"
PR: "4777"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Allow unique key to take a list implementation for postgres/redshift
time: 2022-03-14T11:23:41.293726-05:00
custom:
Author: McKnight-42
Issue: "4738"
PR: "4858"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Added Support for Semantic Versioning
time: 2022-03-21T13:59:49.725046-05:00
custom:
Author: alswang18
Issue: "4453"
PR: "4644"

View File

@@ -1,8 +0,0 @@
kind: Features
body: New Dockerfile to support specific db adapters and platforms. See docker/README.md
for details
time: 2022-03-21T14:00:46.670045-05:00
custom:
Author: iknox-fa
Issue: "4495"
PR: "4487"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Allow unique_key to take a list
time: 2022-03-21T14:01:39.918897-05:00
custom:
Author: gshank
Issue: "2479"
PR: "4618"

View File

@@ -1,7 +0,0 @@
kind: Features
body: "Add `--quiet` global flag and `print` Jinja function"
time: 2022-03-21T14:03:06.565157-05:00
custom:
Author: ehmartens
Issue: "3451"
PR: "4701"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add space before justification periods
time: 2022-03-22T14:48:22.64892-04:00
custom:
Author: joellabes
Issue: "4737"
PR: "4744"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix bug causing empty node level meta, snapshot config errors
time: 2022-03-07T20:30:22.624709-06:00
custom:
Author: gshank
Issue: "4459"
PR: "4726"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Catch all Requests Exceptions on deps install to attempt retries. Also log
the exceptions hit.
time: 2022-03-15T10:53:31.637963-05:00
custom:
Author: emmyoop
Issue: "4849"
PR: "4865"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Use cli_vars instead of context to create package and selector renderers
time: 2022-03-16T15:54:20.608384-04:00
custom:
Author: gshank
Issue: "4876"
PR: "4878"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: depend on new dbt-extractor version with fixed github links
time: 2022-03-17T13:24:18.419599-04:00
custom:
Author: nathaniel-may
Issue: "4891"
PR: "4890"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Update bumpervsion config to stop looking for missing setup.py
time: 2022-03-17T16:30:51.299919-05:00
custom:
Author: iknox-fa
Issue: "-1"
PR: "4896"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Corrected permissions settings for docker release workflow
time: 2022-03-18T09:58:46.061468-05:00
custom:
Author: iknox-fa
Issue: "4902"
PR: "4903"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: "User wasn't asked for permission to overwite a profile entry when running init inside an existing project"
time: 2022-03-21T14:05:21.238521-05:00
custom:
Author: NiallRees
Issue: "4375"
PR: "4447"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add project name validation to `dbt init`
time: 2022-03-21T14:07:20.153071-05:00
custom:
Author: amirkdv
Issue: "4490"
PR: "4536"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Allow override of string and numeric types for adapters.
time: 2022-03-21T14:08:33.384372-05:00
custom:
Author: mdesmet
Issue: "4603"
PR: "4604"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: A change in secret environment variables won't trigger a full reparse
time: 2022-03-21T14:09:21.004543-05:00
custom:
Author: nathaniel-may
Issue: "4650"
PR: "4665"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix misspellings and typos in docstrings
time: 2022-03-21T14:10:07.962502-05:00
custom:
Author: kazanzhy
Issue: "4904"
PR: "4545"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Automate changelog generation with changie
time: 2022-02-18T16:13:19.882436-06:00
custom:
Author: emmyoop
Issue: "4652"
PR: "4743"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Fix broken links for changelog generation and tweak GHA to only post a comment
once when changelog entry is missing.
time: 2022-03-11T10:18:51.404524-06:00
custom:
Author: emmyoop
Issue: "4848"
PR: "4857"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add support for Python 3.10
time: 2022-03-15T15:54:36.759434-05:00
custom:
Author: iknox-fa
Issue: "4562"
PR: "4866"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Enable more dialects to snapshot sources with added columns, even those that
don't support boolean datatypes
time: 2022-03-16T10:03:05.96768-05:00
custom:
Author: McKnight-42
Issue: "4488"
PR: "4871"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Testing cleanup
time: 2022-03-21T14:11:58.750419-05:00
custom:
Author: emmyoop
Issue: "3648"
PR: "4509"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Clean up test deprecation warnings
time: 2022-03-21T14:13:00.557425-05:00
custom:
Author: gshank
Issue: "3988"
PR: "4556"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Use mashumaro for serialization in event logging
time: 2022-03-21T14:13:38.758158-05:00
custom:
Author: gshank
Issue: "4504"
PR: "4505"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Drop support for Python 3.7.0 + 3.7.1
time: 2022-03-21T14:14:51.840637-05:00
custom:
Author: nathaniel-may
Issue: "4584"
PR: "4585"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Drop support for Python <3.7.2
time: 2022-03-21T14:15:23.893773-05:00
custom:
Author: jtcohen6
Issue: "4584"
PR: "4643"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Re-format codebase (except tests) using pre-commit hooks
time: 2022-03-21T14:16:10.285506-05:00
custom:
Author: iknox-fa
Issue: "3195"
PR: "4697"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add deps module README
time: 2022-03-21T14:17:07.884314-05:00
custom:
Author: emmyoop
Issue: "4904"
PR: "4686"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Initial conversion of tests to pytest
time: 2022-03-21T14:17:47.852378-05:00
custom:
Author: gshank
Issue: "4690"
PR: "4691"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix errors in Windows for tests/functions
time: 2022-03-21T14:20:55.648765-05:00
custom:
Author: gshank
Issue: "4782"
PR: "4767"

14
.changes/1.1.1.md Normal file
View File

@@ -0,0 +1,14 @@
## dbt-core 1.1.1 - June 15, 2022
### Fixes
- Relax minimum supported version of MarkupSafe ([#4745](https://github.com/dbt-labs/dbt-core/issues/4745), [#5039](https://github.com/dbt-labs/dbt-core/pull/5039))
- When parsing 'all_sources' should be a list of unique dirs ([#5120](https://github.com/dbt-labs/dbt-core/issues/5120), [#5176](https://github.com/dbt-labs/dbt-core/pull/5176))
- Remove docs file from manifest when removing doc node ([#4146](https://github.com/dbt-labs/dbt-core/issues/4146), [#5270](https://github.com/dbt-labs/dbt-core/pull/5270))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
### Under the Hood
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Dependencies
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Pin networkx to <2.8.4 for v1.1 patches ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Contributors
- [@adamantike](https://github.com/adamantike) ([#5039](https://github.com/dbt-labs/dbt-core/pull/5039))

5
.changes/1.1.2.md Normal file
View File

@@ -0,0 +1,5 @@
## dbt-core 1.1.2 - July 28, 2022
### Fixes
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
### Under the Hood
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))

3
.changes/1.1.3.md Normal file
View File

@@ -0,0 +1,3 @@
## dbt-core 1.1.3 - January 05, 2023
### Fixes
- Bug when partial parsing with an empty schema file ([#4850](https://github.com/dbt-labs/dbt-core/issues/4850), [#<no value>](https://github.com/dbt-labs/dbt-core/pull/<no value>))

8
.changes/1.1.4.md Normal file
View File

@@ -0,0 +1,8 @@
## dbt-core 1.1.4 - February 28, 2023
### Fixes
- add pytz dependency ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
### Contributors
- [@sdebruyn](https://github.com/sdebruyn) ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))

8
.changes/1.1.5.md Normal file
View File

@@ -0,0 +1,8 @@
## dbt-core 1.1.5 - April 19, 2023
### Fixes
- Improved failed event serialization handling and associated tests ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))
### Contributors
- [@QMalcolm](https://github.com/QMalcolm) ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))

View File

@@ -1,9 +0,0 @@
kind: Breaking Changes
body: 'For adapter plugin maintainers only: Internal adapter methods `set_relations_cache` + `_relations_cache_for_schemas`
each take an additional argument, for use with experimental `CACHE_SELECTED_ONLY`
config'
time: 2022-04-12T15:24:50.159572+02:00
custom:
Author: karunpoudel
Issue: "4688"
PR: "4860"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: 'Fixed capitalization in UI for exposures of `type: ml`'
time: 2022-04-04T20:10:21.941618-05:00
custom:
Author: jonstacks
Issue: "4984"
PR: "4995"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: List packages and tags in alphabetical order
time: 2022-04-04T20:10:57.066016-05:00
custom:
Author: pgoslatara
Issue: "4984"
PR: "4995"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Bump jekyll from 3.8.7 to 3.9.0
time: 2022-04-04T20:12:03.146811-05:00
custom:
Author: emmyoop
Issue: "4984"
PR: "4995"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Updated docker README to reflect necessity of using BuildKit
time: 2022-04-08T13:35:40.361554-05:00
custom:
Author: iknox-fa
Issue: "4990"
PR: "5018"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Add `--cache_selected_only` flag to cache schema object of selected models
only.
time: 2022-03-16T00:38:47.8468296-05:00
custom:
Author: karunpoudel
Issue: "4688"
PR: "4860"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Support custom names for generic tests
time: 2022-03-18T08:57:56.05584+01:00
custom:
Author: jtcohen6
Issue: "3348"
PR: "4898"

View File

@@ -1,10 +0,0 @@
kind: Features
body: 'Enable dbt jobs to run downstream models based on fresher sources. Compare
the source freshness results between previous and current state. If any source is
fresher and/or new in current vs. previous state, dbt will run and test the downstream
models in scope. Example command: `dbt build --select source_status:fresher+` '
time: 2022-03-28T13:47:43.750709-05:00
custom:
Author: sungchun12, matt-winkler, anaisvaillant
Issue: "4050"
PR: "4256"

View File

@@ -1,7 +0,0 @@
kind: Features
body: converting unique key as list tests to new pytest format
time: 2022-03-28T15:08:12.70006-05:00
custom:
Author: McKnight-42
Issue: "4882"
PR: "4958"

View File

@@ -1,9 +0,0 @@
kind: Features
body: Add a variable called selected_resources in the Jinja context containing a list
of all the resources matching the nodes for the --select, --exclude and/or --selector
parameters.
time: 2022-04-04T19:04:39.347479+02:00
custom:
Author: b-per
Issue: "3471"
PR: "5001"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Support the DO_NOT_TRACK environment variable from the consoledonottrack.com initiative
time: 2022-04-06T08:03:28.877054+02:00
custom:
Author: agoblet
Issue: "3540"
PR: "5000"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add `--no-print` global flag
time: 2022-04-08T11:41:18.934373+01:00
custom:
Author: poloaraujo
Issue: "4710"
PR: "4854"

View File

@@ -1,7 +0,0 @@
kind: Features
body: add enabled as a source config
time: 2022-04-08T13:27:25.292454-04:00
custom:
Author: nathaniel-may
Issue: "3662"
PR: "5008"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Inconsistent timestamps between inserted/updated and deleted rows in snapshots
time: 2022-03-14T20:02:36.119598832+01:00
custom:
Author: kadero
Issue: "4347"
PR: "4513"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Catch more cases to retry package retrieval for deps pointing to the hub. Also start to cache the package requests.
time: 2022-03-31T14:39:23.952705-05:00
custom:
Author: emmyoop
Issue: "4849"
PR: "4982"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Make the warning message for a full event deque more descriptive
time: 2022-04-07T16:11:34.614988-05:00
custom:
Author: iknox-fa
Issue: "4962"
PR: "5011"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix hard delete snapshot test
time: 2022-04-08T15:55:12.552138-04:00
custom:
Author: gshank
Issue: "4916"
PR: "5020"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: add performance regression testing runner without orchestration
time: 2022-03-09T14:21:33.884043-05:00
custom:
Author: nathaniel-may
Issue: "4021"
PR: "4602"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add Graph Compilation and Adapter Cache tracking
time: 2022-03-21T14:28:54.160087-04:00
custom:
Author: ChenyuLInx
Issue: "4625"
PR: "4912"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Create a dbt.tests.adapter release when releasing dbt and postgres
time: 2022-03-29T11:24:48.34465-04:00
custom:
Author: gshank
Issue: "4812"
PR: "4948"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: update docker image to use python 3.10.3
time: 2022-03-29T11:26:53.359066-04:00
custom:
Author: nathaniel-may
Issue: "4904"
PR: "4963"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: updates black to 22.3.0 which fixes dependency incompatibility when running
with precommit.
time: 2022-03-29T11:35:42.38155-04:00
custom:
Author: nathaniel-may
Issue: "4904"
PR: "4972"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Adds config util for ad-hoc creation of project objs or dicts
time: 2022-03-30T10:14:39.746196-05:00
custom:
Author: iknox-fa
Issue: "4808"
PR: "4981"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Remove TableComparison and convert existing calls to use dbt.tests.util
time: 2022-04-01T15:47:29.054759-04:00
custom:
Author: gshank
Issue: "4778"
PR: "4986"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Remove unneeded create_schema in snapshot materialization
time: 2022-04-04T14:47:08.960642+02:00
custom:
Author: jtcohen6
Issue: "4742"
PR: "4993"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Added .git-blame-ignore-revs file to mask re-formmating commits from git blame
time: 2022-04-08T13:55:25.790513-05:00
custom:
Author: iknox-fa
Issue: "5004"
PR: "5019"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert version tests to pytest
time: 2022-04-11T09:05:36.956877-05:00
custom:
Author: stu-k
Issue: "5024"
PR: "5026"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Updating tests and docs to show that we now support Python 3.10
time: 2022-04-11T10:37:24.806649-04:00
custom:
Author: leahwicz
Issue: "4974"
PR: "5025"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Update --version output and logic
time: 2022-04-11T15:46:26.113705-05:00
custom:
Author: stu-k
Issue: "4724"
PR: "5029"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: ""
time: 2022-04-11T21:39:14.402268-04:00
custom:
Author: leahwicz
Issue: "5033"
PR: "5032"

View File

@@ -6,55 +6,128 @@ changelogPath: CHANGELOG.md
versionExt: md
versionFormat: '## dbt-core {{.Version}} - {{.Time.Format "January 02, 2006"}}'
kindFormat: '### {{.Kind}}'
changeFormat: '- {{.Body}} ([#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), [#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
kinds:
- label: Breaking Changes
- label: Features
- label: Fixes
- label: Docs
- label: Under the Hood
- label: Dependencies
- label: Breaking Changes
- label: Features
- label: Fixes
- label: Docs
changeFormat: |-
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[dbt-docs/#nbr](https://github.com/dbt-labs/dbt-docs/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
afterKind: 1
afterChangelogVersion: 1
beforeKind: 1
endOfVersion: 1
custom:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: Issue
label: GitHub Issue Number
type: int
minLength: 4
- key: PR
label: GitHub Pull Request Number
type: int
minLength: 4
label: GitHub Issue Number (separated by a single space if multiple)
type: string
minLength: 1
footerFormat: |
{{- $contributorDict := dict }}
{{- /* any names added to this list should be all lowercase for later matching purposes */}}
{{- $core_team := list "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "dependabot" }}
{{- $core_team := list "michelleark" "peterallenwebb" "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "aranke" "dependabot[bot]" "snyk-bot" "colin-rogers-dbt" }}
{{- range $change := .Changes }}
{{- $authorList := splitList " " $change.Custom.Author }}
{{- /* loop through all authors for a PR */}}
{{- /* loop through all authors for a single changelog */}}
{{- range $author := $authorList }}
{{- $authorLower := lower $author }}
{{- /* we only want to include non-core team contributors */}}
{{- if not (has $authorLower $core_team)}}
{{- $pr := $change.Custom.PR }}
{{- /* check if this contributor has other PRs associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $prList := get $contributorDict $author }}
{{- $prList = append $prList $pr }}
{{- $contributorDict := set $contributorDict $author $prList }}
{{- else }}
{{- $prList := list $change.Custom.PR }}
{{- $contributorDict := set $contributorDict $author $prList }}
{{- end }}
{{- end}}
{{- $changeList := splitList " " $change.Custom.Author }}
{{- $IssueList := list }}
{{- $changeLink := $change.Kind }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }}
{{- $changes := splitList " " $change.Custom.PR }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- else }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- end }}
{{- /* check if this contributor has other changes associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $contributionList := get $contributorDict $author }}
{{- $contributionList = concat $contributionList $IssueList }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- else }}
{{- $contributionList := $IssueList }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- end }}
{{- end}}
{{- end}}
{{- end }}
{{- /* no indentation here for formatting so the final markdown doesn't have unneeded indentations */}}
{{- if $contributorDict}}
### Contributors
{{- range $k,$v := $contributorDict }}
- [@{{$k}}](https://github.com/{{$k}}) ({{ range $index, $element := $v }}{{if $index}}, {{end}}[#{{$element}}](https://github.com/dbt-labs/dbt-core/pull/{{$element}}){{end}})
- [@{{$k}}](https://github.com/{{$k}}) ({{ range $index, $element := $v }}{{if $index}}, {{end}}{{$element}}{{end}})
{{- end }}
{{- end }}

61
.github/workflows/bot-changelog.yml vendored Normal file
View File

@@ -0,0 +1,61 @@
# **what?**
# When bots create a PR, this action will add a corresponding changie yaml file to that
# PR when a specific label is added.
#
# The file is created off a template:
#
# kind: <per action matrix>
# body: <PR title>
# time: <current timestamp>
# custom:
# Author: <PR User Login (generally the bot)>
# Issue: 4904
# PR: <PR number>
#
# **why?**
# Automate changelog generation for more visability with automated bot PRs.
#
# **when?**
# Once a PR is created, label should be added to PR before or after creation. You can also
# manually trigger this by adding the appropriate label at any time.
#
# **how to add another bot?**
# Add the label and changie kind to the include matrix. That's it!
#
name: Bot Changelog
on:
pull_request:
# catch when the PR is opened with the label or when the label is added
types: [labeled]
permissions:
contents: write
pull-requests: read
jobs:
generate_changelog:
strategy:
matrix:
include:
- label: "dependencies"
changie_kind: "Dependencies"
- label: "snyk"
changie_kind: "Security"
runs-on: ubuntu-latest
steps:
- name: Create and commit changelog on bot PR
if: ${{ contains(github.event.pull_request.labels.*.name, matrix.label) }}
id: bot_changelog
uses: emmyoop/changie_bot@v1.0.1
with:
GITHUB_TOKEN: ${{ secrets.FISHTOWN_BOT_PAT }}
commit_author_name: "Github Build Bot"
commit_author_email: "<buildbot@fishtownanalytics.com>"
commit_message: "Add automated changelog yaml from template for bot PR"
changie_kind: ${{ matrix.changie_kind }}
label: ${{ matrix.label }}
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n PR: ${{ github.event.pull_request.number }}"

View File

@@ -37,14 +37,16 @@ jobs:
code-quality:
name: code-quality
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- name: Check out the repository
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: '3.8'
- name: Install python dependencies
run: |
@@ -63,7 +65,7 @@ jobs:
unit:
name: unit test / python ${{ matrix.python-version }}
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
strategy:
fail-fast: false
@@ -79,7 +81,7 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: ${{ matrix.python-version }}
@@ -113,7 +115,7 @@ jobs:
fail-fast: false
matrix:
python-version: ['3.7', '3.8', '3.9', '3.10']
os: [ubuntu-latest]
os: [ubuntu-20.04]
include:
- python-version: 3.8
os: windows-latest
@@ -130,7 +132,7 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: ${{ matrix.python-version }}
@@ -148,8 +150,8 @@ jobs:
- name: Install python tools
run: |
pip install --user --upgrade pip
pip --version
# pip install --user --upgrade pip
# pip --version
pip install tox
tox --version
@@ -176,16 +178,16 @@ jobs:
build:
name: build packages
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- name: Check out the repository
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: 3.8
python-version: '3.8'
- name: Install python dependencies
run: |

109
.github/workflows/nightly-release.yml vendored Normal file
View File

@@ -0,0 +1,109 @@
# **what?**
# Nightly releases to GitHub and PyPI. This workflow produces the following outcome:
# - generate and validate data for night release (commit SHA, version number, release branch);
# - pass data to release workflow;
# - night release will be pushed to GitHub as a draft release;
# - night build will be pushed to test PyPI;
#
# **why?**
# Ensure an automated and tested release process for nightly builds
#
# **when?**
# This workflow runs on schedule or can be run manually on demand.
name: Nightly Test Release to GitHub and PyPI
on:
workflow_dispatch: # for manual triggering
schedule:
- cron: 0 9 * * *
permissions:
contents: write # this is the permission that allows creating a new release
defaults:
run:
shell: bash
env:
RELEASE_BRANCH: "main"
jobs:
aggregate-release-data:
runs-on: ubuntu-latest
outputs:
commit_sha: ${{ steps.resolve-commit-sha.outputs.release_commit }}
version_number: ${{ steps.nightly-release-version.outputs.number }}
release_branch: ${{ steps.release-branch.outputs.name }}
steps:
- name: "Checkout ${{ github.repository }} Branch ${{ env.RELEASE_BRANCH }}"
uses: actions/checkout@v3
with:
ref: ${{ env.RELEASE_BRANCH }}
- name: "Resolve Commit To Release"
id: resolve-commit-sha
run: |
commit_sha=$(git rev-parse HEAD)
echo "release_commit=$commit_sha" >> $GITHUB_OUTPUT
- name: "Get Current Version Number"
id: version-number-sources
run: |
current_version=`awk -F"current_version = " '{print $2}' .bumpversion.cfg | tr '\n' ' '`
echo "current_version=$current_version" >> $GITHUB_OUTPUT
- name: "Audit Version And Parse Into Parts"
id: semver
uses: dbt-labs/actions/parse-semver@v1.1.0
with:
version: ${{ steps.version-number-sources.outputs.current_version }}
- name: "Get Current Date"
id: current-date
run: echo "date=$(date +'%m%d%Y')" >> $GITHUB_OUTPUT
- name: "Generate Nightly Release Version Number"
id: nightly-release-version
run: |
number="${{ steps.semver.outputs.version }}.dev${{ steps.current-date.outputs.date }}+nightly"
echo "number=$number" >> $GITHUB_OUTPUT
- name: "Audit Nightly Release Version And Parse Into Parts"
uses: dbt-labs/actions/parse-semver@v1.1.0
with:
version: ${{ steps.nightly-release-version.outputs.number }}
- name: "Set Release Branch"
id: release-branch
run: |
echo "name=${{ env.RELEASE_BRANCH }}" >> $GITHUB_OUTPUT
log-outputs-aggregate-release-data:
runs-on: ubuntu-latest
needs: [aggregate-release-data]
steps:
- name: "[DEBUG] Log Outputs"
run: |
echo commit_sha : ${{ needs.aggregate-release-data.outputs.commit_sha }}
echo version_number: ${{ needs.aggregate-release-data.outputs.version_number }}
echo release_branch: ${{ needs.aggregate-release-data.outputs.release_branch }}
release-github-pypi:
needs: [aggregate-release-data]
uses: ./.github/workflows/release.yml
with:
sha: ${{ needs.aggregate-release-data.outputs.commit_sha }}
target_branch: ${{ needs.aggregate-release-data.outputs.release-branch }}
version_number: ${{ needs.aggregate-release-data.outputs.version_number }}
build_script_path: "scripts/build-dist.sh"
env_setup_script_path: "scripts/env-setup.sh"
s3_bucket_name: "core-team-artifacts"
package_test_command: "dbt --version"
test_run: true
nightly_release: true
secrets: inherit

View File

@@ -1,199 +1,227 @@
# **what?**
# Take the given commit, run unit tests specifically on that sha, build and
# package it, and then release to GitHub and PyPi with that specific build
# Release workflow provides the following steps:
# - checkout the given commit;
# - validate version in sources and changelog file for given version;
# - bump the version and generate a changelog if needed;
# - merge all changes to the target branch if needed;
# - run unit and integration tests against given commit;
# - build and package that SHA;
# - release it to GitHub and PyPI with that specific build;
#
# **why?**
# Ensure an automated and tested release process
#
# **when?**
# This will only run manually with a given sha and version
# This workflow can be run manually on demand or can be called by other workflows
name: Release to GitHub and PyPi
name: Release to GitHub and PyPI
on:
workflow_dispatch:
inputs:
sha:
description: 'The last commit sha in the release'
required: true
description: "The last commit sha in the release"
type: string
required: true
target_branch:
description: "The branch to release from"
type: string
required: true
version_number:
description: 'The release version number (i.e. 1.0.0b1)'
required: true
description: "The release version number (i.e. 1.0.0b1)"
type: string
required: true
build_script_path:
description: "Build script path"
type: string
default: "scripts/build-dist.sh"
required: true
env_setup_script_path:
description: "Environment setup script path"
type: string
default: "scripts/env-setup.sh"
required: false
s3_bucket_name:
description: "AWS S3 bucket name"
type: string
default: "core-team-artifacts"
required: true
package_test_command:
description: "Package test command"
type: string
default: "dbt --version"
required: true
test_run:
description: "Test run (Publish release as draft)"
type: boolean
default: true
required: false
nightly_release:
description: "Nightly release to dev environment"
type: boolean
default: false
required: false
workflow_call:
inputs:
sha:
description: "The last commit sha in the release"
type: string
required: true
target_branch:
description: "The branch to release from"
type: string
required: true
version_number:
description: "The release version number (i.e. 1.0.0b1)"
type: string
required: true
build_script_path:
description: "Build script path"
type: string
default: "scripts/build-dist.sh"
required: true
env_setup_script_path:
description: "Environment setup script path"
type: string
default: "scripts/env-setup.sh"
required: false
s3_bucket_name:
description: "AWS S3 bucket name"
type: string
default: "core-team-artifacts"
required: true
package_test_command:
description: "Package test command"
type: string
default: "dbt --version"
required: true
test_run:
description: "Test run (Publish release as draft)"
type: boolean
default: true
required: false
nightly_release:
description: "Nightly release to dev environment"
type: boolean
default: false
required: false
defaults:
run:
shell: bash
jobs:
unit:
name: Unit test
log-inputs:
name: Log Inputs
runs-on: ubuntu-latest
env:
TOXENV: "unit"
steps:
- name: Check out the repository
uses: actions/checkout@v2
with:
persist-credentials: false
ref: ${{ github.event.inputs.sha }}
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
- name: "[DEBUG] Print Variables"
run: |
pip install --user --upgrade pip
pip install tox
pip --version
tox --version
echo The last commit sha in the release: ${{ inputs.sha }}
echo The branch to release from: ${{ inputs.target_branch }}
echo The release version number: ${{ inputs.version_number }}
echo Build script path: ${{ inputs.build_script_path }}
echo Environment setup script path: ${{ inputs.env_setup_script_path }}
echo AWS S3 bucket name: ${{ inputs.s3_bucket_name }}
echo Package test command: ${{ inputs.package_test_command }}
echo Test run: ${{ inputs.test_run }}
echo Nightly release: ${{ inputs.nightly_release }}
- name: Run tox
run: tox
bump-version-generate-changelog:
name: Bump package version, Generate changelog
build:
name: build packages
uses: dbt-labs/dbt-release/.github/workflows/release-prep.yml@main
with:
sha: ${{ inputs.sha }}
version_number: ${{ inputs.version_number }}
target_branch: ${{ inputs.target_branch }}
env_setup_script_path: ${{ inputs.env_setup_script_path }}
test_run: ${{ inputs.test_run }}
nightly_release: ${{ inputs.nightly_release }}
secrets:
FISHTOWN_BOT_PAT: ${{ secrets.FISHTOWN_BOT_PAT }}
log-outputs-bump-version-generate-changelog:
name: "[Log output] Bump package version, Generate changelog"
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog]
runs-on: ubuntu-latest
steps:
- name: Check out the repository
uses: actions/checkout@v2
with:
persist-credentials: false
ref: ${{ github.event.inputs.sha }}
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
- name: Print variables
run: |
pip install --user --upgrade pip
pip install --upgrade setuptools wheel twine check-wheel-contents
pip --version
echo Final SHA : ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
echo Changelog path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
- name: Build distributions
run: ./scripts/build-dist.sh
build-test-package:
name: Build, Test, Package
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog]
- name: Show distributions
run: ls -lh dist/
uses: dbt-labs/dbt-release/.github/workflows/build.yml@main
- name: Check distribution descriptions
run: |
twine check dist/*
with:
sha: ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
changelog_path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
build_script_path: ${{ inputs.build_script_path }}
s3_bucket_name: ${{ inputs.s3_bucket_name }}
package_test_command: ${{ inputs.package_test_command }}
test_run: ${{ inputs.test_run }}
nightly_release: ${{ inputs.nightly_release }}
- name: Check wheel contents
run: |
check-wheel-contents dist/*.whl --ignore W007,W008
- uses: actions/upload-artifact@v2
with:
name: dist
path: |
dist/
!dist/dbt-${{github.event.inputs.version_number}}.tar.gz
test-build:
name: verify packages
needs: [build, unit]
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
run: |
pip install --user --upgrade pip
pip install --upgrade wheel
pip --version
- uses: actions/download-artifact@v2
with:
name: dist
path: dist/
- name: Show distributions
run: ls -lh dist/
- name: Install wheel distributions
run: |
find ./dist/*.whl -maxdepth 1 -type f | xargs pip install --force-reinstall --find-links=dist/
- name: Check wheel distributions
run: |
dbt --version
- name: Install source distributions
run: |
find ./dist/*.gz -maxdepth 1 -type f | xargs pip install --force-reinstall --find-links=dist/
- name: Check source distributions
run: |
dbt --version
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
github-release:
name: GitHub Release
if: ${{ !failure() && !cancelled() }}
needs: test-build
needs: [bump-version-generate-changelog, build-test-package]
runs-on: ubuntu-latest
uses: dbt-labs/dbt-release/.github/workflows/github-release.yml@main
steps:
- uses: actions/download-artifact@v2
with:
name: dist
path: '.'
# Need to set an output variable because env variables can't be taken as input
# This is needed for the next step with releasing to GitHub
- name: Find release type
id: release_type
env:
IS_PRERELEASE: ${{ contains(github.event.inputs.version_number, 'rc') || contains(github.event.inputs.version_number, 'b') }}
run: |
echo ::set-output name=isPrerelease::$IS_PRERELEASE
- name: Creating GitHub Release
uses: softprops/action-gh-release@v1
with:
name: dbt-core v${{github.event.inputs.version_number}}
tag_name: v${{github.event.inputs.version_number}}
prerelease: ${{ steps.release_type.outputs.isPrerelease }}
target_commitish: ${{github.event.inputs.sha}}
body: |
[Release notes](https://github.com/dbt-labs/dbt-core/blob/main/CHANGELOG.md)
files: |
dbt_postgres-${{github.event.inputs.version_number}}-py3-none-any.whl
dbt_core-${{github.event.inputs.version_number}}-py3-none-any.whl
dbt-postgres-${{github.event.inputs.version_number}}.tar.gz
dbt-core-${{github.event.inputs.version_number}}.tar.gz
with:
sha: ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
changelog_path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
test_run: ${{ inputs.test_run }}
pypi-release:
name: Pypi release
name: PyPI Release
runs-on: ubuntu-latest
needs: [github-release]
needs: github-release
uses: dbt-labs/dbt-release/.github/workflows/pypi-release.yml@main
environment: PypiProd
steps:
- uses: actions/download-artifact@v2
with:
name: dist
path: 'dist'
with:
version_number: ${{ inputs.version_number }}
test_run: ${{ inputs.test_run }}
- name: Publish distribution to PyPI
uses: pypa/gh-action-pypi-publish@v1.4.2
with:
password: ${{ secrets.PYPI_API_TOKEN }}
secrets:
PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
TEST_PYPI_API_TOKEN: ${{ secrets.TEST_PYPI_API_TOKEN }}
slack-notification:
name: Slack Notification
if: ${{ failure() && (!inputs.test_run || inputs.nightly_release) }}
needs:
[
bump-version-generate-changelog,
build-test-package,
github-release,
pypi-release,
]
uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}

View File

@@ -22,7 +22,7 @@ jobs:
# run the performance measurements on the current or default branch
test-schema:
name: Test Log Schema
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
env:
# turns warnings into errors
RUSTFLAGS: "-D warnings"

View File

@@ -1,18 +1,15 @@
# **what?**
# This workflow will take a version number and a dry run flag. With that
# This workflow will take the new version number to bump to. With that
# it will run versionbump to update the version number everywhere in the
# code base and then generate an update Docker requirements file. If this
# is a dry run, a draft PR will open with the changes. If this isn't a dry
# run, the changes will be committed to the branch this is run on.
# code base and then run changie to create the corresponding changelog.
# A PR will be created with the changes that can be reviewed before committing.
# **why?**
# This is to aid in releasing dbt and making sure we have updated
# the versions and Docker requirements in all places.
# the version in all places and generated the changelog.
# **when?**
# This is triggered either manually OR
# from the repository_dispatch event "version-bump" which is sent from
# the dbt-release repo Action
# This is triggered manually
name: Version Bump
@@ -20,35 +17,21 @@ on:
workflow_dispatch:
inputs:
version_number:
description: 'The version number to bump to'
description: 'The version number to bump to (ex. 1.2.0, 1.3.0b1)'
required: true
is_dry_run:
description: 'Creates a draft PR to allow testing instead of committing to a branch'
required: true
default: 'true'
repository_dispatch:
types: [version-bump]
jobs:
bump:
runs-on: ubuntu-latest
steps:
- name: "[DEBUG] Print Variables"
run: |
echo "all variables defined as inputs"
echo The version_number: ${{ github.event.inputs.version_number }}
- name: Check out the repository
uses: actions/checkout@v2
- name: Set version and dry run values
id: variables
env:
VERSION_NUMBER: "${{ github.event.client_payload.version_number == '' && github.event.inputs.version_number || github.event.client_payload.version_number }}"
IS_DRY_RUN: "${{ github.event.client_payload.is_dry_run == '' && github.event.inputs.is_dry_run || github.event.client_payload.is_dry_run }}"
run: |
echo Repository dispatch event version: ${{ github.event.client_payload.version_number }}
echo Repository dispatch event dry run: ${{ github.event.client_payload.is_dry_run }}
echo Workflow dispatch event version: ${{ github.event.inputs.version_number }}
echo Workflow dispatch event dry run: ${{ github.event.inputs.is_dry_run }}
echo ::set-output name=VERSION_NUMBER::$VERSION_NUMBER
echo ::set-output name=IS_DRY_RUN::$IS_DRY_RUN
- uses: actions/setup-python@v2
with:
python-version: "3.8"
@@ -59,51 +42,80 @@ jobs:
source env/bin/activate
pip install --upgrade pip
- name: Create PR branch
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
- name: Add Homebrew to PATH
run: |
git checkout -b bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
git push origin bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
git branch --set-upstream-to=origin/bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
echo "/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin" >> $GITHUB_PATH
# - name: Generate Docker requirements
# run: |
# source env/bin/activate
# pip install -r requirements.txt
# pip freeze -l > docker/requirements/requirements.txt
# git status
- name: Install Homebrew packages
run: |
brew install pre-commit
brew tap miniscruff/changie https://github.com/miniscruff/changie
brew install changie
- name: Audit Version and Parse Into Parts
id: semver
uses: dbt-labs/actions/parse-semver@v1
with:
version: ${{ github.event.inputs.version_number }}
- name: Set branch value
id: variables
run: |
echo "::set-output name=BRANCH_NAME::prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID"
- name: Create PR branch
run: |
git checkout -b ${{ steps.variables.outputs.BRANCH_NAME }}
git push origin ${{ steps.variables.outputs.BRANCH_NAME }}
git branch --set-upstream-to=origin/${{ steps.variables.outputs.BRANCH_NAME }} ${{ steps.variables.outputs.BRANCH_NAME }}
- name: Bump version
run: |
source env/bin/activate
pip install -r dev-requirements.txt
env/bin/bumpversion --allow-dirty --new-version ${{steps.variables.outputs.VERSION_NUMBER}} major
env/bin/bumpversion --allow-dirty --new-version ${{ github.event.inputs.version_number }} major
git status
- name: Commit version bump directly
uses: EndBug/add-and-commit@v7
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'false' }}
with:
author_name: 'Github Build Bot'
author_email: 'buildbot@fishtownanalytics.com'
message: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
- name: Run changie
run: |
if [[ ${{ steps.semver.outputs.is-pre-release }} -eq 1 ]]
then
changie batch ${{ steps.semver.outputs.base-version }} --move-dir '${{ steps.semver.outputs.base-version }}' --prerelease '${{ steps.semver.outputs.pre-release }}'
else
changie batch ${{ steps.semver.outputs.base-version }} --include '${{ steps.semver.outputs.base-version }}' --remove-prereleases
fi
changie merge
git status
# this step will fail on whitespace errors but also correct them
- name: Remove trailing whitespace
continue-on-error: true
run: |
pre-commit run trailing-whitespace --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
# this step will fail on newline errors but also correct them
- name: Removing extra newlines
continue-on-error: true
run: |
pre-commit run end-of-file-fixer --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
- name: Commit version bump to branch
uses: EndBug/add-and-commit@v7
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
with:
author_name: 'Github Build Bot'
author_email: 'buildbot@fishtownanalytics.com'
message: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
push: 'origin origin/bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
message: 'Bumping version to ${{ github.event.inputs.version_number }} and generate CHANGELOG'
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
push: 'origin origin/${{ steps.variables.outputs.BRANCH_NAME }}'
- name: Create Pull Request
uses: peter-evans/create-pull-request@v3
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
with:
author: 'Github Build Bot <buildbot@fishtownanalytics.com>'
draft: true
base: ${{github.ref}}
title: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
title: 'Bumping version to ${{ github.event.inputs.version_number }} and generate changelog'
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
labels: |
Skip Changelog

View File

@@ -35,7 +35,7 @@ repos:
- "--target-version=py38"
- "--check"
- "--diff"
- repo: https://gitlab.com/pycqa/flake8
- repo: https://github.com/pycqa/flake8
rev: 4.0.1
hooks:
- id: flake8

View File

@@ -5,21 +5,75 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.1.5 - April 19, 2023
## dbt-core 1.1.0-b1 - March 22, 2022
### Fixes
- Improved failed event serialization handling and associated tests ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))
### Contributors
- [@QMalcolm](https://github.com/QMalcolm) ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))
## dbt-core 1.1.4 - February 28, 2023
### Fixes
- add pytz dependency ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
### Contributors
- [@sdebruyn](https://github.com/sdebruyn) ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
## dbt-core 1.1.3 - January 05, 2023
### Fixes
- Bug when partial parsing with an empty schema file ([#4850](https://github.com/dbt-labs/dbt-core/issues/4850), [#<no value>](https://github.com/dbt-labs/dbt-core/pull/<no value>))
## dbt-core 1.1.2 - July 28, 2022
### Fixes
- Define compatibility for older manifest versions when using state: selection methods ([#5213](https://github.com/dbt-labs/dbt-core/issues/5213), [#5346](https://github.com/dbt-labs/dbt-core/pull/5346))
### Under the Hood
- Add annotation to render_value method reimplemented in #5334 ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5382](https://github.com/dbt-labs/dbt-core/pull/5382))
## dbt-core 1.1.1 - June 15, 2022
### Fixes
- Relax minimum supported version of MarkupSafe ([#4745](https://github.com/dbt-labs/dbt-core/issues/4745), [#5039](https://github.com/dbt-labs/dbt-core/pull/5039))
- When parsing 'all_sources' should be a list of unique dirs ([#5120](https://github.com/dbt-labs/dbt-core/issues/5120), [#5176](https://github.com/dbt-labs/dbt-core/pull/5176))
- Remove docs file from manifest when removing doc node ([#4146](https://github.com/dbt-labs/dbt-core/issues/4146), [#5270](https://github.com/dbt-labs/dbt-core/pull/5270))
- Fixing Windows color regression ([#5191](https://github.com/dbt-labs/dbt-core/issues/5191), [#5327](https://github.com/dbt-labs/dbt-core/pull/5327))
### Under the Hood
- Update context readme + clean up context code" ([#4796](https://github.com/dbt-labs/dbt-core/issues/4796), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Dependencies
- Bumping hologram version ([#5219](https://github.com/dbt-labs/dbt-core/issues/5219), [#5218](https://github.com/dbt-labs/dbt-core/pull/5218))
- Pin networkx to <2.8.4 for v1.1 patches ([#5286](https://github.com/dbt-labs/dbt-core/issues/5286), [#5334](https://github.com/dbt-labs/dbt-core/pull/5334))
### Contributors
- [@adamantike](https://github.com/adamantike) ([#5039](https://github.com/dbt-labs/dbt-core/pull/5039))
## dbt-core 1.1.0 - April 28, 2022
### Breaking Changes
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter ([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- **Relevant to maintainers of adapter plugins _only_:** The abstractmethods `get_response` and `execute` now only return a `connection.AdapterReponse` in type hints. (Previously, they could return a string.) We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter
([#4499](https://github.com/dbt-labs/dbt-core/issues/4499), [#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- For adapter plugin maintainers only: Internal adapter methods `set_relations_cache` + `_relations_cache_for_schemas` each take an additional argument, for use with experimental `CACHE_SELECTED_ONLY` config ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
### Features
- Change behaviour of `non_null` test so that it only `select`s all columns if `--store-failures` is enabled. ([#4769](https://github.com/dbt-labs/dbt-core/issues/4769), [#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
- Testing framework for dbt adapter testing ([#4730](https://github.com/dbt-labs/dbt-core/issues/4730), [#4846](https://github.com/dbt-labs/dbt-core/pull/4846))
- Allow unique key to take a list implementation for postgres/redshift ([#4738](https://github.com/dbt-labs/dbt-core/issues/4738), [#4858](https://github.com/dbt-labs/dbt-core/pull/4858))
- Add `--cache_selected_only` flag to cache schema object of selected models only. ([#4688](https://github.com/dbt-labs/dbt-core/issues/4688), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
- Support custom names for generic tests ([#3348](https://github.com/dbt-labs/dbt-core/issues/3348), [#4898](https://github.com/dbt-labs/dbt-core/pull/4898))
- Added Support for Semantic Versioning ([#4453](https://github.com/dbt-labs/dbt-core/issues/4453), [#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- New Dockerfile to support specific db adapters and platforms. See docker/README.md for details ([#4495](https://github.com/dbt-labs/dbt-core/issues/4495), [#4487](https://github.com/dbt-labs/dbt-core/pull/4487))
- Allow unique_key to take a list ([#2479](https://github.com/dbt-labs/dbt-core/issues/2479), [#4618](https://github.com/dbt-labs/dbt-core/pull/4618))
- Add `--quiet` global flag and `print` Jinja function ([#3451](https://github.com/dbt-labs/dbt-core/issues/3451), [#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- Add space before justification periods ([#4737](https://github.com/dbt-labs/dbt-core/issues/4737), [#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- Enable dbt jobs to run downstream models based on fresher sources. Compare the source freshness results between previous and current state. If any source is fresher and/or new in current vs. previous state, dbt will run and test the downstream models in scope. Example command: `dbt build --select source_status:fresher+` ([#4050](https://github.com/dbt-labs/dbt-core/issues/4050), [#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- converting unique key as list tests to new pytest format ([#4882](https://github.com/dbt-labs/dbt-core/issues/4882), [#4958](https://github.com/dbt-labs/dbt-core/pull/4958))
- Add a variable called selected_resources in the Jinja context containing a list of all the resources matching the nodes for the --select, --exclude and/or --selector parameters. ([#3471](https://github.com/dbt-labs/dbt-core/issues/3471), [#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
- Support the DO_NOT_TRACK environment variable from the consoledonottrack.com initiative ([#3540](https://github.com/dbt-labs/dbt-core/issues/3540), [#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
- Add `--no-print` global flag ([#4710](https://github.com/dbt-labs/dbt-core/issues/4710), [#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
- add enabled as a source config ([#3662](https://github.com/dbt-labs/dbt-core/issues/3662), [#5008](https://github.com/dbt-labs/dbt-core/pull/5008))
### Fixes
- Fix bug causing empty node level meta, snapshot config errors ([#4459](https://github.com/dbt-labs/dbt-core/issues/4459), [#4726](https://github.com/dbt-labs/dbt-core/pull/4726))
- Inconsistent timestamps between inserted/updated and deleted rows in snapshots ([#4347](https://github.com/dbt-labs/dbt-core/issues/4347), [#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
- Catch all Requests Exceptions on deps install to attempt retries. Also log the exceptions hit. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4865](https://github.com/dbt-labs/dbt-core/pull/4865))
- Use cli_vars instead of context to create package and selector renderers ([#4876](https://github.com/dbt-labs/dbt-core/issues/4876), [#4878](https://github.com/dbt-labs/dbt-core/pull/4878))
- depend on new dbt-extractor version with fixed github links ([#4891](https://github.com/dbt-labs/dbt-core/issues/4891), [#4890](https://github.com/dbt-labs/dbt-core/pull/4890))
@@ -30,11 +84,26 @@
- Allow override of string and numeric types for adapters. ([#4603](https://github.com/dbt-labs/dbt-core/issues/4603), [#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- A change in secret environment variables won't trigger a full reparse ([#4650](https://github.com/dbt-labs/dbt-core/issues/4650), [#4665](https://github.com/dbt-labs/dbt-core/pull/4665))
- Fix misspellings and typos in docstrings ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- Catch more cases to retry package retrieval for deps pointing to the hub. Also start to cache the package requests. ([#4849](https://github.com/dbt-labs/dbt-core/issues/4849), [#4982](https://github.com/dbt-labs/dbt-core/pull/4982))
- Make the warning message for a full event deque more descriptive ([#4962](https://github.com/dbt-labs/dbt-core/issues/4962), [#5011](https://github.com/dbt-labs/dbt-core/pull/5011))
- Fix hard delete snapshot test ([#4916](https://github.com/dbt-labs/dbt-core/issues/4916), [#5020](https://github.com/dbt-labs/dbt-core/pull/5020))
- Restore ability to utilize `updated_at` for check_cols snapshots ([#5076](https://github.com/dbt-labs/dbt-core/issues/5076), [#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- Use yaml renderer (with target context) for rendering selectors ([#5131](https://github.com/dbt-labs/dbt-core/issues/5131), [#5136](https://github.com/dbt-labs/dbt-core/pull/5136))
- Fix retry logic to return values after initial try ([#5023](https://github.com/dbt-labs/dbt-core/issues/5023), [#5137](https://github.com/dbt-labs/dbt-core/pull/5137))
- Scrub secret env vars from CommandError in exception stacktrace ([#5151](https://github.com/dbt-labs/dbt-core/issues/5151), [#5152](https://github.com/dbt-labs/dbt-core/pull/5152))
### Docs
- Resolve errors related to operations preventing DAG from generating in the docs. Also patch a spark issue to allow search to filter accurately past the missing columns. ([#4578](https://github.com/dbt-labs/dbt-core/issues/4578), [#4763](https://github.com/dbt-labs/dbt-core/pull/4763))
- Fixed capitalization in UI for exposures of `type: ml` ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- List packages and tags in alphabetical order ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- Bump jekyll from 3.8.7 to 3.9.0 ([#4984](https://github.com/dbt-labs/dbt-core/issues/4984), [#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- Updated docker README to reflect necessity of using BuildKit ([#4990](https://github.com/dbt-labs/dbt-core/issues/4990), [#5018](https://github.com/dbt-labs/dbt-core/pull/5018))
### Under the Hood
- Automate changelog generation with changie ([#4652](https://github.com/dbt-labs/dbt-core/issues/4652), [#4743](https://github.com/dbt-labs/dbt-core/pull/4743))
- add performance regression testing runner without orchestration ([#4021](https://github.com/dbt-labs/dbt-core/issues/4021), [#4602](https://github.com/dbt-labs/dbt-core/pull/4602))
- Fix broken links for changelog generation and tweak GHA to only post a comment once when changelog entry is missing. ([#4848](https://github.com/dbt-labs/dbt-core/issues/4848), [#4857](https://github.com/dbt-labs/dbt-core/pull/4857))
- Add support for Python 3.10 ([#4562](https://github.com/dbt-labs/dbt-core/issues/4562), [#4866](https://github.com/dbt-labs/dbt-core/pull/4866))
- Enable more dialects to snapshot sources with added columns, even those that don't support boolean datatypes ([#4488](https://github.com/dbt-labs/dbt-core/issues/4488), [#4871](https://github.com/dbt-labs/dbt-core/pull/4871))
- Add Graph Compilation and Adapter Cache tracking ([#4625](https://github.com/dbt-labs/dbt-core/issues/4625), [#4912](https://github.com/dbt-labs/dbt-core/pull/4912))
- Testing cleanup ([#3648](https://github.com/dbt-labs/dbt-core/issues/3648), [#4509](https://github.com/dbt-labs/dbt-core/pull/4509))
- Clean up test deprecation warnings ([#3988](https://github.com/dbt-labs/dbt-core/issues/3988), [#4556](https://github.com/dbt-labs/dbt-core/pull/4556))
- Use mashumaro for serialization in event logging ([#4504](https://github.com/dbt-labs/dbt-core/issues/4504), [#4505](https://github.com/dbt-labs/dbt-core/pull/4505))
@@ -44,19 +113,38 @@
- Add deps module README ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4686](https://github.com/dbt-labs/dbt-core/pull/4686))
- Initial conversion of tests to pytest ([#4690](https://github.com/dbt-labs/dbt-core/issues/4690), [#4691](https://github.com/dbt-labs/dbt-core/pull/4691))
- Fix errors in Windows for tests/functions ([#4782](https://github.com/dbt-labs/dbt-core/issues/4782), [#4767](https://github.com/dbt-labs/dbt-core/pull/4767))
### Docs
- Resolve errors related to operations preventing DAG from generating in the docs. Also patch a spark issue to allow search to filter accurately past the missing columns. ([#4578](https://github.com/dbt-labs/dbt-core/issues/4578), [#4763](https://github.com/dbt-labs/dbt-core/pull/4763))
### Contributors
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- [@alswang18](https://github.com/alswang18) ([#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- [@amirkdv](https://github.com/amirkdv) ([#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- [@ehmartens](https://github.com/ehmartens) ([#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- [@joellabes](https://github.com/joellabes) ([#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- [@kazanzhy](https://github.com/kazanzhy) ([#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- [@mdesmet](https://github.com/mdesmet) ([#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- [@versusfacit](https://github.com/versusfacit) ([#4869](https://github.com/dbt-labs/dbt-core/pull/4869))
- [@willbowditch](https://github.com/willbowditch) ([#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
- Create a dbt.tests.adapter release when releasing dbt and postgres ([#4812](https://github.com/dbt-labs/dbt-core/issues/4812), [#4948](https://github.com/dbt-labs/dbt-core/pull/4948))
- update docker image to use python 3.10.3 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4963](https://github.com/dbt-labs/dbt-core/pull/4963))
- updates black to 22.3.0 which fixes dependency incompatibility when running with precommit. ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#4972](https://github.com/dbt-labs/dbt-core/pull/4972))
- Adds config util for ad-hoc creation of project objs or dicts ([#4808](https://github.com/dbt-labs/dbt-core/issues/4808), [#4981](https://github.com/dbt-labs/dbt-core/pull/4981))
- Remove TableComparison and convert existing calls to use dbt.tests.util ([#4778](https://github.com/dbt-labs/dbt-core/issues/4778), [#4986](https://github.com/dbt-labs/dbt-core/pull/4986))
- Remove unneeded create_schema in snapshot materialization ([#4742](https://github.com/dbt-labs/dbt-core/issues/4742), [#4993](https://github.com/dbt-labs/dbt-core/pull/4993))
- Added .git-blame-ignore-revs file to mask re-formmating commits from git blame ([#5004](https://github.com/dbt-labs/dbt-core/issues/5004), [#5019](https://github.com/dbt-labs/dbt-core/pull/5019))
- Convert version tests to pytest ([#5024](https://github.com/dbt-labs/dbt-core/issues/5024), [#5026](https://github.com/dbt-labs/dbt-core/pull/5026))
- Updating tests and docs to show that we now support Python 3.10 ([#4974](https://github.com/dbt-labs/dbt-core/issues/4974), [#5025](https://github.com/dbt-labs/dbt-core/pull/5025))
- Update --version output and logic ([#4724](https://github.com/dbt-labs/dbt-core/issues/4724), [#5029](https://github.com/dbt-labs/dbt-core/pull/5029))
- ([#5033](https://github.com/dbt-labs/dbt-core/issues/5033), [#5032](https://github.com/dbt-labs/dbt-core/pull/5032))
### Contributors
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
- [@agoblet](https://github.com/agoblet) ([#5000](https://github.com/dbt-labs/dbt-core/pull/5000))
- [@alswang18](https://github.com/alswang18) ([#4644](https://github.com/dbt-labs/dbt-core/pull/4644))
- [@amirkdv](https://github.com/amirkdv) ([#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
- [@anaisvaillant](https://github.com/anaisvaillant) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@b-per](https://github.com/b-per) ([#5001](https://github.com/dbt-labs/dbt-core/pull/5001))
- [@dbeatty10](https://github.com/dbeatty10) ([#5077](https://github.com/dbt-labs/dbt-core/pull/5077))
- [@ehmartens](https://github.com/ehmartens) ([#4701](https://github.com/dbt-labs/dbt-core/pull/4701))
- [@joellabes](https://github.com/joellabes) ([#4744](https://github.com/dbt-labs/dbt-core/pull/4744))
- [@jonstacks](https://github.com/jonstacks) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- [@kadero](https://github.com/kadero) ([#4513](https://github.com/dbt-labs/dbt-core/pull/4513))
- [@karunpoudel](https://github.com/karunpoudel) ([#4860](https://github.com/dbt-labs/dbt-core/pull/4860), [#4860](https://github.com/dbt-labs/dbt-core/pull/4860))
- [@kazanzhy](https://github.com/kazanzhy) ([#4545](https://github.com/dbt-labs/dbt-core/pull/4545))
- [@matt-winkler](https://github.com/matt-winkler) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@mdesmet](https://github.com/mdesmet) ([#4604](https://github.com/dbt-labs/dbt-core/pull/4604))
- [@pgoslatara](https://github.com/pgoslatara) ([#4995](https://github.com/dbt-labs/dbt-core/pull/4995))
- [@poloaraujo](https://github.com/poloaraujo) ([#4854](https://github.com/dbt-labs/dbt-core/pull/4854))
- [@sungchun12](https://github.com/sungchun12) ([#4256](https://github.com/dbt-labs/dbt-core/pull/4256))
- [@willbowditch](https://github.com/willbowditch) ([#4777](https://github.com/dbt-labs/dbt-core/pull/4777))
## Previous Releases

View File

@@ -226,7 +226,15 @@ We use [changie](https://changie.dev) to generate `CHANGELOG` entries. Do not e
Follow the steps to [install `changie`](https://changie.dev/guide/installation/) for your system.
Once changie is installed and your PR is created, simply run `changie new` and changie will walk you through the process of creating a changelog entry. Commit the file that's created and your changelog entry is complete!
Once changie is installed and your PR is created for a new feature, simply run the following command and changie will walk you through the process of creating a changelog entry:
```shell
changie new
```
Commit the file that's created and your changelog entry is complete!
If you are contributing to a feature already in progress, you will modify the changie yaml file in dbt/.changes/unreleased/ related to your change. If you need help finding this file, please ask within the discussion for the pull request!
## Submitting a Pull Request

View File

@@ -28,7 +28,7 @@ def _is_commit(revision: str) -> bool:
def _raise_git_cloning_error(repo, revision, error):
stderr = error.stderr.decode("utf-8").strip()
stderr = error.stderr.strip()
if "usage: git" in stderr:
stderr = stderr.split("\nusage: git")[0]
if re.match("fatal: destination path '(.+)' already exists", stderr):
@@ -115,8 +115,8 @@ def checkout(cwd, repo, revision=None):
try:
return _checkout(cwd, repo, revision)
except CommandResultError as exc:
stderr = exc.stderr.decode("utf-8").strip()
bad_package_spec(repo, revision, stderr)
stderr = exc.stderr.strip()
bad_package_spec(repo, revision, stderr)
def get_current_sha(cwd):
@@ -142,7 +142,7 @@ def clone_and_checkout(
subdirectory=subdirectory,
)
except CommandResultError as exc:
err = exc.stderr.decode("utf-8")
err = exc.stderr
exists = re.match("fatal: destination path '(.+)' already exists", err)
if not exists:
raise_git_cloning_problem(repo)

View File

@@ -132,7 +132,11 @@ def _all_source_paths(
analysis_paths: List[str],
macro_paths: List[str],
) -> List[str]:
return list(chain(model_paths, seed_paths, snapshot_paths, analysis_paths, macro_paths))
# We need to turn a list of lists into just a list, then convert to a set to
# get only unique elements, then back to a list
return list(
set(list(chain(model_paths, seed_paths, snapshot_paths, analysis_paths, macro_paths)))
)
T = TypeVar("T")

View File

@@ -1,12 +1,15 @@
from typing import Dict, Any, Tuple, Optional, Union, Callable
import re
import os
from dbt.clients.jinja import get_rendered, catch_jinja
from dbt.context.target import TargetContext
from dbt.context.secret import SecretContext
from dbt.context.secret import SecretContext, SECRET_PLACEHOLDER
from dbt.context.base import BaseContext
from dbt.contracts.connection import HasCredentials
from dbt.exceptions import DbtProjectError, CompilationException, RecursionException
from dbt.utils import deep_map_render
from dbt.logger import SECRET_ENV_PREFIX
Keypath = Tuple[Union[str, int], ...]
@@ -114,12 +117,10 @@ class DbtProjectYamlRenderer(BaseRenderer):
def name(self):
"Project config"
# Uses SecretRenderer
def get_package_renderer(self) -> BaseRenderer:
return PackageRenderer(self.ctx_obj.cli_vars)
def get_selector_renderer(self) -> BaseRenderer:
return SelectorRenderer(self.ctx_obj.cli_vars)
def render_project(
self,
project: Dict[str, Any],
@@ -136,8 +137,7 @@ class DbtProjectYamlRenderer(BaseRenderer):
return package_renderer.render_data(packages)
def render_selectors(self, selectors: Dict[str, Any]):
selector_renderer = self.get_selector_renderer()
return selector_renderer.render_data(selectors)
return self.render_data(selectors)
def render_entry(self, value: Any, keypath: Keypath) -> Any:
result = super().render_entry(value, keypath)
@@ -165,18 +165,10 @@ class DbtProjectYamlRenderer(BaseRenderer):
return True
class SelectorRenderer(BaseRenderer):
@property
def name(self):
return "Selector config"
class SecretRenderer(BaseRenderer):
def __init__(self, cli_vars: Optional[Dict[str, Any]] = None) -> None:
def __init__(self, cli_vars: Dict[str, Any] = {}) -> None:
# Generate contexts here because we want to save the context
# object in order to retrieve the env_vars.
if cli_vars is None:
cli_vars = {}
self.ctx_obj = SecretContext(cli_vars)
context = self.ctx_obj.to_dict()
super().__init__(context)
@@ -185,6 +177,28 @@ class SecretRenderer(BaseRenderer):
def name(self):
return "Secret"
def render_value(self, value: Any, keypath: Optional[Keypath] = None) -> Any:
# First, standard Jinja rendering, with special handling for 'secret' environment variables
# "{{ env_var('DBT_SECRET_ENV_VAR') }}" -> "$$$DBT_SECRET_START$$$DBT_SECRET_ENV_{VARIABLE_NAME}$$$DBT_SECRET_END$$$"
# This prevents Jinja manipulation of secrets via macros/filters that might leak partial/modified values in logs
rendered = super().render_value(value, keypath)
# Now, detect instances of the placeholder value ($$$DBT_SECRET_START...DBT_SECRET_END$$$)
# and replace them with the actual secret value
if SECRET_ENV_PREFIX in str(rendered):
search_group = f"({SECRET_ENV_PREFIX}(.*))"
pattern = SECRET_PLACEHOLDER.format(search_group).replace("$", r"\$")
m = re.search(
pattern,
rendered,
)
if m:
found = m.group(1)
value = os.environ[found]
replace_this = SECRET_PLACEHOLDER.format(found)
return rendered.replace(replace_this, value)
else:
return rendered
class ProfileRenderer(SecretRenderer):
@property

View File

@@ -3,7 +3,7 @@ from typing import Dict, Any, Union
from dbt.clients.yaml_helper import yaml, Loader, Dumper, load_yaml_text # noqa: F401
from dbt.dataclass_schema import ValidationError
from .renderer import SelectorRenderer
from .renderer import BaseRenderer
from dbt.clients.system import (
load_file_contents,
@@ -57,7 +57,7 @@ class SelectorConfig(Dict[str, Dict[str, Union[SelectionSpec, bool]]]):
def render_from_dict(
cls,
data: Dict[str, Any],
renderer: SelectorRenderer,
renderer: BaseRenderer,
) -> "SelectorConfig":
try:
rendered = renderer.render_data(data)
@@ -72,7 +72,7 @@ class SelectorConfig(Dict[str, Dict[str, Union[SelectionSpec, bool]]]):
def from_path(
cls,
path: Path,
renderer: SelectorRenderer,
renderer: BaseRenderer,
) -> "SelectorConfig":
try:
data = load_yaml_text(load_file_contents(str(path)))

View File

@@ -62,6 +62,8 @@ def get_project_config(
user_config = read_user_config(flags.PROFILES_DIR)
# Update flags
flags.set_from_args(args, user_config)
if cli_vars is None:
cli_vars = {}
profile = Profile.render_from_args(args, ProfileRenderer(cli_vars), profile_name)
# Generate a project
project = Project.from_project_root(

View File

@@ -1 +1,51 @@
# Contexts and Jinja rendering
Contexts are used for Jinja rendering. They include context methods, executable macros, and various settings that are available in Jinja.
The most common entrypoint to Jinja rendering in dbt is a method named `get_rendered`, which takes two arguments: templated code (string), and a context used to render it (dictionary).
The context is the bundle of information that is in "scope" when rendering Jinja-templated code. For instance, imagine a simple Jinja template:
```
{% set new_value = some_macro(some_variable) %}
```
Both `some_macro()` and `some_variable` must be defined in that context. Otherwise, it will raise an error when rendering.
Different contexts are used in different places because we allow access to different methods and data in different places. Executable SQL, for example, includes all available macros and the model being run. The variables and macros in scope for Jinja defined in yaml files is much more limited.
### Implementation
The context that is passed to Jinja is always in a dictionary format, not an actual class, so a `to_dict()` is executed on a context class before it is used for rendering.
Each context has a `generate_<name>_context` function to create the context. `ProviderContext` subclasses have different generate functions for parsing and for execution, so that certain functions (notably `ref`, `source`, and `config`) can return different results
### Hierarchy
All contexts inherit from the `BaseContext`, which includes "pure" methods (e.g. `tojson`), `env_var()`, and `var()` (but only CLI values, passed via `--vars`).
Methods available in parent contexts are also available in child contexts.
```
BaseContext -- core/dbt/context/base.py
SecretContext -- core/dbt/context/secret.py
TargetContext -- core/dbt/context/target.py
ConfiguredContext -- core/dbt/context/configured.py
SchemaYamlContext -- core/dbt/context/configured.py
DocsRuntimeContext -- core/dbt/context/configured.py
MacroResolvingContext -- core/dbt/context/configured.py
ManifestContext -- core/dbt/context/manifest.py
QueryHeaderContext -- core/dbt/context/manifest.py
ProviderContext -- core/dbt/context/provider.py
MacroContext -- core/dbt/context/provider.py
ModelContext -- core/dbt/context/provider.py
TestContext -- core/dbt/context/provider.py
```
### Contexts for configuration
Contexts for rendering "special" `.yml` (configuration) files:
- `SecretContext`: Supports "secret" env vars, which are prefixed with `DBT_ENV_SECRET_`. Used for rendering in `profiles.yml` and `packages.yml` ONLY. Secrets defined elsewhere will raise explicit errors.
- `TargetContext`: The same as `Base`, plus `target` (connection profile). Used most notably in `dbt_project.yml` and `selectors.yml`.
Contexts for other `.yml` files in the project:
- `SchemaYamlContext`: Supports `vars` declared on the CLI and in `dbt_project.yml`. Does not support custom macros, beyond `var()` + `env_var()` methods. Used for all `.yml` files, to define properties and configuration.
- `DocsRuntimeContext`: Standard `.yml` file context, plus `doc()` method (with all `docs` blocks in scope). Used to resolve `description` properties.

View File

@@ -24,38 +24,7 @@ import pytz
import datetime
import re
# Contexts in dbt Core
# Contexts are used for Jinja rendering. They include context methods,
# executable macros, and various settings that are available in Jinja.
#
# Different contexts are used in different places because we allow access
# to different methods and data in different places. Executable SQL, for
# example, includes the available macros and the model, while Jinja in
# yaml files is more limited.
#
# The context that is passed to Jinja is always in a dictionary format,
# not an actual class, so a 'to_dict()' is executed on a context class
# before it is used for rendering.
#
# Each context has a generate_<name>_context function to create the context.
# ProviderContext subclasses have different generate functions for
# parsing and for execution.
#
# Context class hierarchy
#
# BaseContext -- core/dbt/context/base.py
# SecretContext -- core/dbt/context/secret.py
# TargetContext -- core/dbt/context/target.py
# ConfiguredContext -- core/dbt/context/configured.py
# SchemaYamlContext -- core/dbt/context/configured.py
# DocsRuntimeContext -- core/dbt/context/configured.py
# MacroResolvingContext -- core/dbt/context/configured.py
# ManifestContext -- core/dbt/context/manifest.py
# QueryHeaderContext -- core/dbt/context/manifest.py
# ProviderContext -- core/dbt/context/provider.py
# MacroContext -- core/dbt/context/provider.py
# ModelContext -- core/dbt/context/provider.py
# TestContext -- core/dbt/context/provider.py
# See the `contexts` module README for more information on how contexts work
def get_pytz_module_context() -> Dict[str, Any]:
@@ -472,6 +441,10 @@ class BaseContext(metaclass=ContextMeta):
{{ log("Running some_macro: " ~ arg1 ~ ", " ~ arg2) }}
{% endmacro %}"
"""
if not isinstance(msg, str):
msg = str(msg)
if info:
fire_event(MacroEventInfo(msg=msg))
else:
@@ -552,9 +525,8 @@ class BaseContext(metaclass=ContextMeta):
{% endif %}
This supports all flags defined in flags submodule (core/dbt/flags.py)
TODO: Replace with object that provides read-only access to flag values
"""
return flags
return flags.get_flag_obj()
@contextmember
@staticmethod

View File

@@ -7,6 +7,9 @@ from dbt.exceptions import raise_parsing_error
from dbt.logger import SECRET_ENV_PREFIX
SECRET_PLACEHOLDER = "$$$DBT_SECRET_START$$${}$$$DBT_SECRET_END$$$"
class SecretContext(BaseContext):
"""This context is used in profiles.yml + packages.yml. It can render secret
env vars that aren't usable elsewhere"""
@@ -18,21 +21,29 @@ class SecretContext(BaseContext):
If the default is None, raise an exception for an undefined variable.
In this context *only*, env_var will return the actual values of
env vars prefixed with DBT_ENV_SECRET_
In this context *only*, env_var will accept env vars prefixed with DBT_ENV_SECRET_.
It will return the name of the secret env var, wrapped in 'start' and 'end' identifiers.
The actual value will be subbed in later in SecretRenderer.render_value()
"""
return_value = None
if var in os.environ:
# if this is a 'secret' env var, just return the name of the env var
# instead of rendering the actual value here, to avoid any risk of
# Jinja manipulation. it will be subbed out later, in SecretRenderer.render_value
if var in os.environ and var.startswith(SECRET_ENV_PREFIX):
return SECRET_PLACEHOLDER.format(var)
elif var in os.environ:
return_value = os.environ[var]
elif default is not None:
return_value = default
if return_value is not None:
# do not save secret environment variables
# store env vars in the internal manifest to power partial parsing
# if it's a 'secret' env var, we shouldn't even get here
# but just to be safe — don't save secrets
if not var.startswith(SECRET_ENV_PREFIX):
self.env_vars[var] = return_value
# return the value even if its a secret
return return_value
else:
msg = f"Env var required but not provided: '{var}'"

View File

@@ -1135,6 +1135,10 @@ class WritableManifest(ArtifactMixin):
)
)
@classmethod
def compatible_previous_versions(self):
return [("manifest", 4)]
def _check_duplicates(value: HasUniqueID, src: Mapping[str, HasUniqueID]):
if value.unique_id in src:

View File

@@ -201,6 +201,14 @@ class VersionedSchema(dbtClassMixin):
result["$id"] = str(cls.dbt_schema_version)
return result
@classmethod
def is_compatible_version(cls, schema_version):
compatible_versions = [str(cls.dbt_schema_version)]
if hasattr(cls, "compatible_previous_versions"):
for name, version in cls.compatible_previous_versions():
compatible_versions.append(str(SchemaVersion(name, version)))
return str(schema_version) in compatible_versions
@classmethod
def read_and_check_versions(cls, path: str):
try:
@@ -217,7 +225,7 @@ class VersionedSchema(dbtClassMixin):
if "metadata" in data and "dbt_schema_version" in data["metadata"]:
previous_schema_version = data["metadata"]["dbt_schema_version"]
# cls.dbt_schema_version is a SchemaVersion object
if str(cls.dbt_schema_version) != previous_schema_version:
if not cls.is_compatible_version(previous_schema_version):
raise IncompatibleSchemaException(
expected=str(cls.dbt_schema_version), found=previous_schema_version
)

View File

@@ -1,4 +1,3 @@
import colorama
from colorama import Style
import dbt.events.functions as this # don't worry I hate it too.
from dbt.events.base_types import NoStdOut, Event, NoFile, ShowException, Cache
@@ -50,14 +49,6 @@ format_color = True
format_json = False
invocation_id: Optional[str] = None
# Colorama needs some help on windows because we're using logger.info
# intead of print(). If the Windows env doesn't have a TERM var set,
# then we should override the logging stream to use the colorama
# converter. If the TERM var is set (as with Git Bash), then it's safe
# to send escape characters and no log handler injection is needed.
if sys.platform == "win32":
colorama.init(wrap=False)
def setup_event_logger(log_path, level_override=None):
# flags have been resolved, and log_path is known

View File

@@ -383,10 +383,11 @@ class FailedToConnectException(DatabaseException):
class CommandError(RuntimeException):
def __init__(self, cwd, cmd, message="Error running command"):
cmd_scrubbed = list(scrub_secrets(cmd_txt, env_secrets()) for cmd_txt in cmd)
super().__init__(message)
self.cwd = cwd
self.cmd = cmd
self.args = (cwd, cmd, message)
self.cmd = cmd_scrubbed
self.args = (cwd, cmd_scrubbed, message)
def __str__(self):
if len(self.cmd) == 0:
@@ -411,9 +412,9 @@ class CommandResultError(CommandError):
def __init__(self, cwd, cmd, returncode, stdout, stderr, message="Got a non-zero returncode"):
super().__init__(cwd, cmd, message)
self.returncode = returncode
self.stdout = stdout
self.stderr = stderr
self.args = (cwd, cmd, returncode, stdout, stderr, message)
self.stdout = scrub_secrets(stdout.decode("utf-8"), env_secrets())
self.stderr = scrub_secrets(stderr.decode("utf-8"), env_secrets())
self.args = (cwd, self.cmd, returncode, self.stdout, self.stderr, message)
def __str__(self):
return "{} running: {}".format(self.msg, self.cmd)
@@ -704,7 +705,6 @@ def missing_materialization(model, adapter_type):
def bad_package_spec(repo, spec, error_message):
msg = "Error checking out spec='{}' for repo {}\n{}".format(spec, repo, error_message)
raise InternalException(scrub_secrets(msg, env_secrets()))

View File

@@ -1,7 +1,9 @@
import os
# Do not import the os package because we expose this package in jinja
from os import name as os_name, path as os_path, getenv as os_getenv
import multiprocessing
from argparse import Namespace
if os.name != "nt":
if os_name != "nt":
# https://bugs.python.org/issue41567
import multiprocessing.popen_spawn_posix # type: ignore
from pathlib import Path
@@ -10,8 +12,8 @@ from typing import Optional
# PROFILES_DIR must be set before the other flags
# It also gets set in main.py and in set_from_args because the rpc server
# doesn't go through exactly the same main arg processing.
DEFAULT_PROFILES_DIR = os.path.join(os.path.expanduser("~"), ".dbt")
PROFILES_DIR = os.path.expanduser(os.getenv("DBT_PROFILES_DIR", DEFAULT_PROFILES_DIR))
DEFAULT_PROFILES_DIR = os_path.join(os_path.expanduser("~"), ".dbt")
PROFILES_DIR = os_path.expanduser(os_getenv("DBT_PROFILES_DIR", DEFAULT_PROFILES_DIR))
STRICT_MODE = False # Only here for backwards compatibility
FULL_REFRESH = False # subcommand
@@ -78,7 +80,7 @@ def env_set_truthy(key: str) -> Optional[str]:
"""Return the value if it was set to a "truthy" string value or None
otherwise.
"""
value = os.getenv(key)
value = os_getenv(key)
if not value or value.lower() in ("0", "false", "f"):
return None
return value
@@ -91,7 +93,7 @@ def env_set_bool(env_value):
def env_set_path(key: str) -> Optional[Path]:
value = os.getenv(key)
value = os_getenv(key)
if value is None:
return value
else:
@@ -168,7 +170,7 @@ def get_flag_value(flag, args, user_config):
if flag in ["PRINTER_WIDTH", "EVENT_BUFFER_SIZE"]: # must be ints
flag_value = int(flag_value)
if flag == "PROFILES_DIR":
flag_value = os.path.abspath(flag_value)
flag_value = os_path.abspath(flag_value)
return flag_value
@@ -192,7 +194,7 @@ def _load_flag_value(flag, args, user_config):
def _get_flag_value_from_env(flag):
# Environment variables use pattern 'DBT_{flag name}'
env_flag = _get_env_flag(flag)
env_value = os.getenv(env_flag)
env_value = os_getenv(env_flag)
if env_value is None or env_value == "":
return None
@@ -229,4 +231,19 @@ def get_flag_dict():
"event_buffer_size": EVENT_BUFFER_SIZE,
"quiet": QUIET,
"no_print": NO_PRINT,
"cache_selected_only": CACHE_SELECTED_ONLY,
}
# This is used by core/dbt/context/base.py to return a flag object
# in Jinja.
def get_flag_obj():
new_flags = Namespace()
for k, v in get_flag_dict().items():
setattr(new_flags, k.upper(), v)
# The following 3 are CLI arguments only so they're not full-fledged flags,
# but we put in flags for users.
setattr(new_flags, "FULL_REFRESH", FULL_REFRESH)
setattr(new_flags, "STORE_FAILURES", STORE_FAILURES)
setattr(new_flags, "WHICH", WHICH)
return new_flags

View File

@@ -132,7 +132,7 @@
{% set check_cols_config = config['check_cols'] %}
{% set primary_key = config['unique_key'] %}
{% set invalidate_hard_deletes = config.get('invalidate_hard_deletes', false) %}
{% set updated_at = snapshot_get_time() %}
{% set updated_at = config.get('updated_at', snapshot_get_time()) %}
{% set column_added = false %}

View File

@@ -15,17 +15,16 @@ import colorama
import logbook
from dbt.dataclass_schema import dbtClassMixin
# Colorama needs some help on windows because we're using logger.info
# intead of print(). If the Windows env doesn't have a TERM var set,
# then we should override the logging stream to use the colorama
# converter. If the TERM var is set (as with Git Bash), then it's safe
# to send escape characters and no log handler injection is needed.
logging_stdout = sys.stdout
if sys.platform == "win32":
if not os.getenv("TERM"):
logging_stdout = colorama.AnsiToWin32(sys.stdout).stream
colorama.init(wrap=False)
# Colorama is needed for colored logs on Windows because we're using logger.info
# intead of print(). If the Windows env doesn't have a TERM var set or it is set to None
# (i.e. in the case of Git Bash on Windows- this emulates Unix), then it's safe to initialize
# Colorama with wrapping turned on which allows us to strip ANSI sequences from stdout.
# You can safely initialize Colorama for any OS and the coloring stays the same except
# when piped to anoter process for Linux and MacOS, then it loses the coloring. To combat
# that, we will just initialize Colorama when needed on Windows using a non-Unix terminal.
if sys.platform == "win32" and (not os.getenv("TERM") or os.getenv("TERM") == "None"):
colorama.init(wrap=True)
STDOUT_LOG_FORMAT = "{record.message}"
DEBUG_LOG_FORMAT = (
@@ -455,7 +454,7 @@ class DelayedFileHandler(logbook.RotatingFileHandler, FormatterMixin):
class LogManager(logbook.NestedSetup):
def __init__(self, stdout=logging_stdout, stderr=sys.stderr):
def __init__(self, stdout=sys.stdout, stderr=sys.stderr):
self.stdout = stdout
self.stderr = stderr
self._null_handler = logbook.NullHandler()

View File

@@ -573,6 +573,10 @@ class PartialParsing:
# doc source_file.nodes
self.schedule_nodes_for_parsing(source_file.nodes)
source_file.nodes = []
# Remove the file object
self.deleted_manifest.files[source_file.file_id] = self.saved_manifest.files.pop(
source_file.file_id
)
# Schema files -----------------------
# Changed schema files

View File

@@ -115,7 +115,8 @@ def yaml_from_file(source_file: SchemaSourceFile) -> Dict[str, Any]:
"""If loading the yaml fails, raise an exception."""
path = source_file.path.relative_path
try:
return load_yaml_text(source_file.contents)
# source_file.contents can sometimes be None
return load_yaml_text(source_file.contents or "")
except ValidationException as e:
reason = validator_error_message(e)
raise ParsingException(

View File

@@ -107,6 +107,9 @@ def test_data_dir(request):
# This contains the profile target information, for simplicity in setting
# up different profiles, particularly in the adapter repos.
# Note: because we load the profile to create the adapter, this
# fixture can't be used to test vars and env_vars or errors. The
# profile must be written out after the test starts.
@pytest.fixture(scope="class")
def dbt_profile_target():
return {
@@ -179,6 +182,7 @@ def dbt_project_yml(project_root, project_config_update, logs_dir):
if project_config_update:
project_config.update(project_config_update)
write_file(yaml.safe_dump(project_config), project_root, "dbt_project.yml")
return project_config
# Fixture to provide packages as either yaml or dictionary

Some files were not shown because too many files have changed in this diff Show More