Compare commits

...

65 Commits

Author SHA1 Message Date
Michelle Ark
83c40d7fb5 remove ipdb 2023-04-28 16:31:03 -04:00
FishtownBuildBot
6bfba6d116 [Automated] Merged prep-release/1.3.4_4745738796 into target 1.3.latest during release process 2023-04-19 12:06:00 -05:00
Github Build Bot
d23d414dfe Bumping version to 1.3.4 and generate changelog 2023-04-19 16:39:21 +00:00
Quigley Malcolm
6e7e572e86 [BACKPORT 1.3.latest] Improved event log msg stringification (#7341)
* Add tests for logging jinja2.Undefined objects

[CT-2259](https://github.com/dbt-labs/dbt-core/issues/7108) identifies
an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined
object is attempted to be logged. This generally happened in the form
of `{{ log(undefined_variable, info=True) }}`. This commit adding this
test exists for two reasons
1. Ensure we don't have a regression in this going forward
2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3

* Add tests for checking `DBT_ENV_SECRET_`s don't break logging

[CT-1783](https://github.com/dbt-labs/dbt-core/issues/6568) describes
a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all
`{{ log("logging stuff", info=True) }}` invocations break. This commit
adds a test for this for two reasons:
1. Ensure we don't regress to this behavior going forward
2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3

* Add changie info for changes

* Ensure `log()` calls from jinja ensure the msg is stringified

The simplest way to resolve [CT-2259](https://github.com/dbt-labs/dbt-core/issues/7108)
and [CT-1783](https://github.com/dbt-labs/dbt-core/issues/6568) in backports
to dbt-core < 1.4 is to ensure `msg` in `BaseContext.log()` is stringified.

---------

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-04-18 13:20:22 -07:00
Michelle Ark
d57542307b sqlparse <0.4.4 (#7400) 2023-04-18 12:47:56 -04:00
FishtownBuildBot
7f2e9f94c0 [Automated] Merged prep-release/1.3.3_4294171507 into target 1.3.latest during release process 2023-02-28 09:22:11 -06:00
Github Build Bot
27b2e9ac54 Bumping version to 1.3.3 and generate changelog 2023-02-28 14:48:16 +00:00
Sam Debruyn
a6890609e0 fix: add pytz dependency (#7079) 2023-02-28 09:36:39 -05:00
Alexander Smolyakov
3c30f96ab2 [CI/CD] Backport release workflow to 1.3.latest (#6792)
* [CI/CD] Update release workflow and introduce workflow for nightly releases (#6602)

* Add release workflows

* Update nightly-release.yml

* Set default `test_run` value to `true`

* Update .bumpversion.cfg

* Resolve review comment

- Update workflow docs
- Change workflow name
- Set `test_run` default value to `true`

* Update Slack secret

* PyPI

* Update release workflow (#6778)

- Update AWS secrets
- Rework condition for Slack notification

* update regex to match all iterations (#6839)

* update regex to match all iterations

* convert to num to match all adapters

* add comments, remove extra .

* clarify with more comments

* Update .bumpversion.cfg

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>

---------

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
# Conflicts:
#	.bumpversion.cfg

* put back correct version

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-02-03 14:24:11 -06:00
Emily Rockman
6fff2888d4 [BACKPORT] fix contributor list generation (#6799) (#6809)
* Remove PR from most changelog kinds (#6374)

* update changie to require issue or pr, and allow multiple

* remove extraneous data from changelog files.

* allow for multiple PR/issues to be entered

* update contributing guide

* remove issue number from bot changelogs

* update format of PR

* fix dependency changelogs

* remove extra line

* remove extra lines, tweak contributor wording

* Update CONTRIBUTING.md

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
# Conflicts:
#	.changes/unreleased/Dependency-20220923-000646.yaml
#	.changes/unreleased/Dependency-20221007-000848.yaml
#	.changes/unreleased/Dependency-20221020-000753.yaml
#	.changes/unreleased/Dependency-20221026-000910.yaml
#	.changes/unreleased/Docs-20220908-154157.yaml
#	.changes/unreleased/Docs-20221007-090656.yaml
#	.changes/unreleased/Docs-20221017-171411.yaml
#	.changes/unreleased/Docs-20221116-155743.yaml
#	.changes/unreleased/Docs-20221202-150523.yaml
#	.changes/unreleased/Features-20220408-165459.yaml
#	.changes/unreleased/Features-20220817-154857.yaml
#	.changes/unreleased/Features-20220912-125935.yaml
#	.changes/unreleased/Features-20220914-095625.yaml
#	.changes/unreleased/Features-20220925-211651.yaml
#	.changes/unreleased/Features-20221003-110705.yaml
#	.changes/unreleased/Features-20221102-150003.yaml
#	.changes/unreleased/Features-20221114-185207.yaml
#	.changes/unreleased/Features-20221130-112913.yaml
#	.changes/unreleased/Fixes-20220916-104854.yaml
#	.changes/unreleased/Fixes-20221010-113218.yaml
#	.changes/unreleased/Fixes-20221011-160715.yaml
#	.changes/unreleased/Fixes-20221016-173742.yaml
#	.changes/unreleased/Fixes-20221107-095314.yaml
#	.changes/unreleased/Fixes-20221115-081021.yaml
#	.changes/unreleased/Fixes-20221202-164859.yaml
#	.changes/unreleased/Under the Hood-20220927-194259.yaml
#	.changes/unreleased/Under the Hood-20220929-134406.yaml
#	.changes/unreleased/Under the Hood-20221005-120310.yaml
#	.changes/unreleased/Under the Hood-20221007-094627.yaml
#	.changes/unreleased/Under the Hood-20221007-140044.yaml
#	.changes/unreleased/Under the Hood-20221013-181912.yaml
#	.changes/unreleased/Under the Hood-20221017-151511.yaml
#	.changes/unreleased/Under the Hood-20221017-155844.yaml
#	.changes/unreleased/Under the Hood-20221028-104837.yaml
#	.changes/unreleased/Under the Hood-20221028-110344.yaml
#	.changes/unreleased/Under the Hood-20221108-074550.yaml
#	.changes/unreleased/Under the Hood-20221108-115633.yaml
#	.changes/unreleased/Under the Hood-20221108-133104.yaml
#	.changes/unreleased/Under the Hood-20221116-130037.yaml

* fix contributor list generation (#6799)

# Conflicts:
#	.changie.yaml
2023-01-31 21:39:05 -06:00
github-actions[bot]
3f17044383 Bumping version to 1.3.2 and generate CHANGELOG (#6513)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-04 15:31:34 -05:00
github-actions[bot]
e1345d87cd Bumping version to 1.3.2rc1 and generate CHANGELOG (#6508)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-04 10:03:42 -05:00
leahwicz
6e9ff280e2 Partial parsing bug with empty schema file - ensure None is not passed to load_yaml_text (#6494) (#6505)
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-01-04 09:28:30 -05:00
Jeremy Cohen
6defc86ef7 Adjust tox passenv to be multiline (#6405) (#6406) 2022-12-07 23:55:11 +01:00
leahwicz
3fbb0a38ea Reverting back to older ubuntu image (#6363) (#6370)
* Reverting back to older ubuntu image

* Updating the structured logging workflow as well
2022-12-02 15:44:12 -05:00
github-actions[bot]
d71385bdb6 Bumping version to 1.3.1 and generate changelog (#6270)
* Bumping version to 1.3.1rc1 and generate CHANGELOG

* Updating version to remove rc1

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-11-16 14:01:53 -05:00
Jeremy Cohen
c1925c67c2 Backport dbt-docs changes to 1.3.latest (#6267)
* Add new index.html and changelog yaml files from dbt-docs (#6112)

* Add new index.html and changelog yaml files from dbt-docs (#6265)

Co-authored-by: FishtownBuildBot <77737458+FishtownBuildBot@users.noreply.github.com>
2022-11-16 13:04:06 -05:00
github-actions[bot]
54d186583b Updates lib to use new profile name functionality (#6202) (#6221)
* Updates lib to use new profile name functionality

* Adds changie entry

* Fixes formatting

(cherry picked from commit d0543c9242)

Co-authored-by: Rachel <41338402+racheldaniel@users.noreply.github.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-11-15 22:09:59 -05:00
github-actions[bot]
9026a0598c s/gitlab/github for flake8 precommit repo (#6252) (#6254)
(cherry picked from commit eae98677b9)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2022-11-15 11:12:52 -05:00
github-actions[bot]
2d1215ed47 more ergonomic profile name handling (#6157) (#6199)
(cherry picked from commit 77dfec7214)

Co-authored-by: Ian Knox <81931810+iknox-fa@users.noreply.github.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-11-04 10:01:29 -07:00
github-actions[bot]
7f63c3d083 add python version and upgrade action (#6204) (#6205)
(cherry picked from commit c3ccbe3357)

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-11-03 12:44:05 -05:00
github-actions[bot]
19c48e285e Bumping version to 1.3.0 and generate changelog (#6046)
* Bumping version to 1.3.0 and generate CHANGELOG

* Update to correct release date

* Updating release date

* Updated changelog with new entry

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2022-10-11 20:04:30 -04:00
Chenyu Li
65f40b317e fix: check length of args of python model function before accessing it (#6042) (#6049)
* fix: check length of args of python model function before accessing it

* Add test

* changie

(cherry picked from commit dcd6ef733b)

Co-authored-by: Matteo Ferrando <matteo.ferrando2@gmail.com>
2022-10-11 19:18:24 -04:00
github-actions[bot]
c94c891f73 Bumping version to 1.3.0rc2 and generate CHANGELOG (#5988)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-10-03 11:25:41 -04:00
github-actions[bot]
fc45a51582 consolidate timestamp logic (#5979) (#5986)
* Consolidate date macros into dates.sql

* rename to timestamps.sql

* fix whitespace + add changie

* cleanup macros and add testing

* fix whitespace

* remove now macro

* fix functional test

* remove local config

* make snowflake backwards compat return utc

* move timestamps to adaptor base tests

* move backcompat macros to respective adapters

* change timestamp param to source_timestamp

* move timestamps.py to utils

* update changie.yaml

* make expected schema a fixture

* formatting

* add debug message to assert

* fix changie.yaml

* Update tests/adapter/dbt/tests/adapter/utils/test_timestamps.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Update plugins/postgres/dbt/include/postgres/macros/timestamps.sql

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Update .changie.yaml

* add backcompat utc

* remove current_timestamp_in_utc

* remove convert_timezone

* add _in_utc_backcompat

* fix macro_calls typo

* add expected sql validation to test_timestamps

* make expected_sql optional

* improve sql check string comparison test

* remove extraneous test file

* add timestamp casting back

* Update plugins/postgres/dbt/include/postgres/macros/adapters.sql

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* add check_relation_has_expected_schema to comments

* fix whitespace

* remove default impl of current_timestamp

* manual changie log fix

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
(cherry picked from commit 50fe25d230)

Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
2022-10-03 10:50:18 -04:00
github-actions[bot]
6042469c71 Consolidate date macros into timestamps.sql (#5838) (#5985)
* Consolidate date macros into dates.sql

* rename to timestamps.sql

* fix whitespace + add changie

* cleanup macros and add testing

* fix whitespace

* remove now macro

* fix functional test

* remove local config

* make snowflake backwards compat return utc

* move timestamps to adaptor base tests

* move backcompat macros to respective adapters

* change timestamp param to source_timestamp

* move timestamps.py to utils

* update changie.yaml

* make expected schema a fixture

* formatting

* add debug message to assert

* fix changie.yaml

* Update tests/adapter/dbt/tests/adapter/utils/test_timestamps.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Update plugins/postgres/dbt/include/postgres/macros/timestamps.sql

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Update .changie.yaml

* add backcompat utc

* remove current_timestamp_in_utc

* remove convert_timezone

* add _in_utc_backcompat

* fix macro_calls typo

* add expected sql validation to test_timestamps

* make expected_sql optional

* improve sql check string comparison test

* remove extraneous test file

* add timestamp casting back

* Update plugins/postgres/dbt/include/postgres/macros/adapters.sql

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* add check_relation_has_expected_schema to comments

* fix whitespace

* remove default impl of current_timestamp

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
(cherry picked from commit a79960fa64)

Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-10-03 09:40:00 -04:00
github-actions[bot]
2584465169 Disabled Models in schema files (#5868) (#5984)
* clean up debugging

* reword some comments

* changelog

* add more tests

* move around the manifest.node

* fix typos

* all tests passing

* move logic for moving around nodes

* add tests

* more cleanup

* fix failing pp test

* remove comments

* add more tests, patch all disabled nodes

* fix test for windows

* fix node processing to not overwrite enabled nodes

* add checking disabled in pp, fix error msg

* stop deepcopying all nodes when processing

* update error message

(cherry picked from commit fa4f9d3d97)

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-10-03 08:13:12 -05:00
github-actions[bot]
9a81a4dfe3 Bumping version to 1.3.0rc1 and generate CHANGELOG (#5968)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-09-28 15:34:28 -04:00
github-actions[bot]
b6a82446e5 Update Version Bump to include Homebrew in PATH (#5963) (#5964)
(cherry picked from commit 0e60fc1078)

Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-09-28 15:03:24 -04:00
Callum McCann
17aca39e1c Adding metric expression validation (#5873)
* adding validation

* changie

* code formatting

* updating for review

* updating tests
2022-09-27 12:38:03 -04:00
Yoshiaki Ishihara
59744f18bb Fix typos of comments in core/dbt/adapters/ (#5693) 2022-09-27 09:10:24 -07:00
Rachel
f1326f526c Runtime: Prevent introspective queries at compile (SL only) (#5926)
* Preliminary changes to keep compile from connecting to the warehouse for runtime calls

* Adds option to lib to skip connecting to warehouse for compile; adds prelim tests

* Removes unused imports

* Simplifies test and renames to SqlCompileRunnerNoIntrospection

* Updates name in tests

* Spacing

* Updates test to check for adapter connection call instead of compile and execute

* Removes commented line

* Fixes test names

* Updates plugin to postgres type as snowflake isn't available

* Fixes docstring

* Fixes formatting

* Moves conditional logic out of class

* Fixes formatting

* Removes commented line

* Moves import

* Unmoves import

* Updates changelog

* Adds further info to method docstring
2022-09-27 09:49:55 -04:00
Jeremy Cohen
834ac716fd Prefer internal macros when called explicitly (#5907)
* Add functional test

* Prefer internal macros when called explicitly

* Add changelog entry

* update tests format

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-09-27 08:50:55 -04:00
Doug Beatty
0487b96098 Array macros (#5823)
* Helper macro to cast from array to string

* Default implementations and tests for array macros

* Trim Trailing Whitespace

* Changelog entry

* Remove dependence upon `cast_array_to_string` macro

* pre-commit fixes

* Remove `cast_array_to_string` macro

* pre-commit fix

* Trivial direct test; array_concat/append test non-triviallly indirectly

* Remove vestigial `lstrip`
2022-09-26 13:40:15 -06:00
FishtownBuildBot
dbd36f06e4 Add new index.html and changelog yaml files from dbt-docs (#5925) 2022-09-26 14:00:04 -05:00
dave-connors-3
38ada8a68e merge exclude columns for incremental models (#5457)
* exlcude cols like in dbt_utils.star

* dispatch macro

* changelog entry

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-09-26 14:49:12 -04:00
Doug Beatty
e58edaab2d Test for Koalas DataFrames (#5928) 2022-09-26 12:41:56 -06:00
Doug Beatty
c202e005cd Tests for current_timestamp (#5935)
* Tests for `current_timestamp`

* Black formatting
2022-09-26 12:31:36 -06:00
Peter Webb
8129862b3c CT-1221 add handle to changie (#5923)
* Add 'peterallenwebb' to changie's core_team list.

* fix accidental line break
2022-09-26 11:44:20 -04:00
Drew Banin
4e8aa007cf Fix adapter reset race condition in lib.py (#5921)
* (#5919) Fix adapter reset race condition in lib.py

* run black

* changie
2022-09-26 10:26:20 -04:00
Doug Beatty
fe88bfabbf Click CLI profiles directory (#5896)
* Default directories for projects and profiles

* Re-write of get_nearest_project_dir()

* Trim Trailing Whitespace

* Functionally equivalent resolvers
2022-09-24 10:47:47 -06:00
Gerda Shank
5328a64df2 CT 815 partial parsing handling of deleted metrics (#5920)
* Update delete_schema_metric to schedule referencing nodes for reparsing

* Changie
2022-09-23 18:48:00 -04:00
Chenyu Li
87c9974be1 improve error message for parsing args (#5895)
* improve error message for parsing args

* update error message

* Update models.py

* skip stack_trace for all dbt Exceptions
2022-09-23 13:31:10 -07:00
Emily Rockman
f3f509da92 update disabled metrics/exposures to use add_disabled (#5909)
* update disabled metrics/exposures to use add_disabled

* put back func fo tests

* add pp logic

* switch elif

* fix test name

* return node
2022-09-23 13:24:37 -05:00
dave-connors-3
5e8dcec2c5 update flag to -f (#5908) 2022-09-23 20:20:29 +02:00
Chenyu Li
56783446db PySpark dataframe related tests (#5906)
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2022-09-22 16:27:27 -07:00
Chenyu Li
207cc0383d initial support for a .dbtignore (#5897) 2022-09-22 09:06:35 -07:00
leahwicz
49ecd6a6a4 Adding missing permissions on GHA (#5870)
* Adding missing permissions on GHA

* Adding read all permissions explicitly
2022-09-21 14:53:27 -04:00
dave-connors-3
c109f39d82 add optional shorthand to full refresh command (#5879)
* add optional shorthand to full refresh command

* changie

* replace full refresh with shorthand in functional test
2022-09-21 14:37:28 -04:00
Ian Knox
fd778dceb5 Profiling and Adapter management work with Click (#5892) 2022-09-21 11:23:26 -05:00
Emily Rockman
e402241e0e Validate exposure names and add label (#5844)
* first pass

* add label and name validation

* changelog

* fix tests

* convert ParsingError to Deprecation

* fix bug where label did not end up in parsed node

* update deprecation msg
2022-09-21 10:24:44 -05:00
Emily Rockman
a6c37c948d Add name validation for metrics (#5841)
* add tests, add name validation

* tweak test

* small update

* change to 250 char limit, tweak message
2022-09-21 09:31:13 -05:00
Daniel Messias
fd886cb7dd ConfigSelectorMethod should check for bools (#5889)
* ConfigSelectorMethod should check for bools

* Add changelog entry

* Add support for lists and test cases

* Typo and formatting in test

* pre-commit linting
2022-09-21 10:20:14 -04:00
James McNeill
b089a471b7 implement type_boolean macro (#5875)
* implement type_boolean macro

* changie result
2022-09-20 19:24:25 -06:00
Emily Rockman
ae294b643b Manifest deserialization error with disabled models (#5891)
* account for disabled nodes in renaming attributes

* tweak naming
2022-09-20 14:27:25 -05:00
colin-rogers-dbt
0bd6df0d1b [CT-1101] 024_custom_schema_tests (#5828)
* create functional custom schema tests

* delete old integration tests

* changie log

* formatting fixes

* delete changie entry and errant comment
2022-09-20 10:42:07 -07:00
Gerda Shank
7b1d61c956 Ct 1191 event history cleanup (#5858)
* Change Exceptions in events to strings. Refactor event_buffer_handling.

* Changie

* fix fire_event call MainEncounteredError

* Set EventBufferFull message when event buffer >= 10,000
2022-09-20 12:44:33 -04:00
Chenyu Li
646a0c704f restrict python submission (#5822)
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-09-20 09:39:20 -07:00
Doug Beatty
bbf4fc30a5 Default to current working directory for profiles.yml and fall back to ~/.dbt (#5717)
* Method for capturing standard out during testing (rather than logs)

* Allow dbt exit code assertion to be optional

* Verify priority order to search for profiles.yml configuration

* Updates after pre-commit checks

* Test searching for profiles.yml within the dbt project directory before `~/.dbt/`

* Refactor `dbt debug` to move to the project directory prior to looking up profiles directory

* Search the current working directory for profiles.yml

* Changelog

* Formatting with Black

* Move `run_dbt_and_capture_stdout` into the test case

* Update CLI help text

* Unify separate DEFAULT_PROFILES_DIR definitions

* Remove unused PROFILE_DIR_MESSAGE

* Remove unused DEFAULT_PROFILES_DIR

* Use shared definition of DEFAULT_PROFILES_DIR

* Define global vs. local profiles location and dynamically determine the default

* Restore original

* Remove function for determining the default profiles directory
2022-09-20 08:44:15 -04:00
jared-rimmer
6baaa2bcb0 Add metadata env method to provider context (#5794)
* Add dbt_metadata_envs contextproperty to ProviderContext

* Refactor helper methods into functions

* Fix code quality failures

* Add Changelog
2022-09-20 08:41:44 -04:00
Chenyu Li
13a595722a add support for custom file ending (#5845) 2022-09-19 13:58:38 -07:00
Sam Debruyn
3680b6ad0e remove source quoting setting in adapter tests (#5839)
* remove source quoting setting in adapter tests

* changelog entry

* formatting
2022-09-19 11:00:12 -07:00
Ian Knox
4c29d48d1c Flags work with Click (#5790) 2022-09-19 12:36:38 -05:00
Chenyu Li
e00eb9aa3a fix multiple dbt.config.get in python model (#5850)
* fix multiple dbt.config.get in python model

Co-authored-by: Stu Kilgore <stuart.kilgore@gmail.com>
2022-09-16 15:07:59 -07:00
Jeremy Cohen
f5a94fc774 Back/fwd compatibility for renamed metrics attributes (#5825)
* Naive handling for metric attr renames

* Add tests for bwd/fwd compatibility

* Add deprecation

* Add changelog entry

* PR feedback

* Small fixups

* emmyoop's suggestions

Co-authored-by: Callum McCann <cmccann51@gmail.com>
2022-09-16 11:21:35 +02:00
226 changed files with 4729 additions and 2318 deletions

View File

@@ -1,13 +1,15 @@
[bumpversion]
current_version = 1.3.0b2
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
((?P<prekind>a|b|rc)
(?P<pre>\d+) # pre-release version num
current_version = 1.3.4
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
(((?P<prekind>a|b|rc) # optional pre-release type
?(?P<num>[\d]+?)) # optional pre-release version number
\.?(?P<nightly>[a-z0-9]+\+[a-z]+)? # optional nightly release indicator
)?
serialize =
{major}.{minor}.{patch}{prekind}{pre}
{major}.{minor}.{patch}{prekind}{num}.{nightly}
{major}.{minor}.{patch}{prekind}{num}
{major}.{minor}.{patch}
commit = False
tag = False
@@ -21,9 +23,11 @@ values =
rc
final
[bumpversion:part:pre]
[bumpversion:part:num]
first_value = 1
[bumpversion:part:nightly]
[bumpversion:file:core/setup.py]
[bumpversion:file:core/dbt/version.py]

View File

@@ -1,38 +0,0 @@
## dbt-core 1.3.0-b1 - July 29, 2022
### Features
- Python model inital version ([#5261](https://github.com/dbt-labs/dbt-core/issues/5261), [#5421](https://github.com/dbt-labs/dbt-core/pull/5421))
- allows user to include the file extension for .py models in the dbt run -m command. ([#5289](https://github.com/dbt-labs/dbt-core/issues/5289), [#5295](https://github.com/dbt-labs/dbt-core/pull/5295))
- Incremental materialization refactor and cleanup ([#5245](https://github.com/dbt-labs/dbt-core/issues/5245), [#5359](https://github.com/dbt-labs/dbt-core/pull/5359))
- Python models can support incremental logic ([#0](https://github.com/dbt-labs/dbt-core/issues/0), [#35](https://github.com/dbt-labs/dbt-core/pull/35))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
- add exponential backoff to connection retries on Postgres (and Redshift) ([#5502](https://github.com/dbt-labs/dbt-core/issues/5502), [#5503](https://github.com/dbt-labs/dbt-core/pull/5503))
### Fixes
- Add context to compilation errors generated while rendering generic test configuration values. ([#5294](https://github.com/dbt-labs/dbt-core/issues/5294), [#5393](https://github.com/dbt-labs/dbt-core/pull/5393))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- Ignore empty strings passed in as secrets ([#5312](https://github.com/dbt-labs/dbt-core/issues/5312), [#5518](https://github.com/dbt-labs/dbt-core/pull/5518))
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
### Docs
- Update dependency inline-source from ^6.1.5 to ^7.2.0 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Update dependency jest from ^26.2.2 to ^28.1.3 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Update dependency underscore from ^1.9.0 to ^1.13.4 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Update dependency webpack-cli from ^3.3.12 to ^4.7.0 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Update dependency webpack-dev-server from ^3.1.11 to ^4.9.3 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Searches no longer require perfect matches, and instead consider each word individually. `my model` or `model my` will now find `my_model`, without the need for underscores ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Support the renaming of SQL to code happening in dbt-core ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
### Under the Hood
- Added language to tracked fields in run_model event ([#5571](https://github.com/dbt-labs/dbt-core/issues/5571), [#5469](https://github.com/dbt-labs/dbt-core/pull/5469))
- Update mashumaro to 3.0.3 ([#4940](https://github.com/dbt-labs/dbt-core/issues/4940), [#5118](https://github.com/dbt-labs/dbt-core/pull/5118))
- Add python incremental materialization test ([#0000](https://github.com/dbt-labs/dbt-core/issues/0000), [#5571](https://github.com/dbt-labs/dbt-core/pull/5571))
### Dependencies
- Upgrade to Jinja2==3.1.2 from Jinja2==2.11.3 ([#4748](https://github.com/dbt-labs/dbt-core/issues/4748), [#5465](https://github.com/dbt-labs/dbt-core/pull/5465))
- Bump mypy from 0.961 to 0.971 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5495](https://github.com/dbt-labs/dbt-core/pull/5495))
- Remove pin for MarkUpSafe from >=0.23,<2.1 ([#5506](https://github.com/dbt-labs/dbt-core/issues/5506), [#5507](https://github.com/dbt-labs/dbt-core/pull/5507))
### Contributors
- [@Goodkat](https://github.com/Goodkat) ([#5518](https://github.com/dbt-labs/dbt-core/pull/5518))
- [@drewbanin](https://github.com/drewbanin) ([#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@joellabes](https://github.com/joellabes) ([#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- [@leoebfolsom](https://github.com/leoebfolsom) ([#5295](https://github.com/dbt-labs/dbt-core/pull/5295))
- [@nicholasyager](https://github.com/nicholasyager) ([#5393](https://github.com/dbt-labs/dbt-core/pull/5393))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432))

View File

@@ -1,49 +0,0 @@
## dbt-core 1.3.0-b2 - August 29, 2022
### Features
- Add `--defer` flag to dbt compile & dbt docs generate ([#4110](https://github.com/dbt-labs/dbt-core/issues/4110), [#4514](https://github.com/dbt-labs/dbt-core/pull/4514))
- use MethodName.File when value ends with .csv ([#5578](https://github.com/dbt-labs/dbt-core/issues/5578), [#5581](https://github.com/dbt-labs/dbt-core/pull/5581))
- Make `docs` configurable in `dbt_project.yml` and add a `node_color` attribute to change the color of nodes in the DAG ([#5333](https://github.com/dbt-labs/dbt-core/issues/5333), [#5397](https://github.com/dbt-labs/dbt-core/pull/5397))
- Adding ResolvedMetricReference helper functions and tests ([#5567](https://github.com/dbt-labs/dbt-core/issues/5567), [#5607](https://github.com/dbt-labs/dbt-core/pull/5607))
- Check dbt-core version requirements when installing Hub packages ([#5648](https://github.com/dbt-labs/dbt-core/issues/5648), [#5651](https://github.com/dbt-labs/dbt-core/pull/5651))
### Fixes
- Remove the default 256 characters limit on postgres character varying type when no limitation is set ([#5238](https://github.com/dbt-labs/dbt-core/issues/5238), [#5292](https://github.com/dbt-labs/dbt-core/pull/5292))
- Include schema file config in unrendered_config ([#5338](https://github.com/dbt-labs/dbt-core/issues/5338), [#5344](https://github.com/dbt-labs/dbt-core/pull/5344))
- Resolves #5351 - Do not consider shorter varchar cols as schema changes ([#5351](https://github.com/dbt-labs/dbt-core/issues/5351), [#5395](https://github.com/dbt-labs/dbt-core/pull/5395))
- on_shchma_change fail verbosity enhancement ([#5504](https://github.com/dbt-labs/dbt-core/issues/5504), [#5505](https://github.com/dbt-labs/dbt-core/pull/5505))
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
- Extended validations for the project names ([#5379](https://github.com/dbt-labs/dbt-core/issues/5379), [#5620](https://github.com/dbt-labs/dbt-core/pull/5620))
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
- multiple args for ref and source ([#5634](https://github.com/dbt-labs/dbt-core/issues/5634), [#5635](https://github.com/dbt-labs/dbt-core/pull/5635))
- Fix Unexpected behavior when chaining methods on dbt-ref'ed/sourced dataframes ([#5646](https://github.com/dbt-labs/dbt-core/issues/5646), [#5677](https://github.com/dbt-labs/dbt-core/pull/5677))
### Docs
- Leverages `docs.node_color` from `dbt-core` to color nodes in the DAG ([dbt-docs/#44](https://github.com/dbt-labs/dbt-docs/issues/44), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
### Under the Hood
- Save use of default env vars to manifest to enable partial parsing in those cases. ([#5155](https://github.com/dbt-labs/dbt-core/issues/5155), [#5589](https://github.com/dbt-labs/dbt-core/pull/5589))
- add more information to log line interop test failures ([#5658](https://github.com/dbt-labs/dbt-core/issues/5658), [#5659](https://github.com/dbt-labs/dbt-core/pull/5659))
- Add supported languages to materializations ([#5569](https://github.com/dbt-labs/dbt-core/issues/5569), [#5695](https://github.com/dbt-labs/dbt-core/pull/5695))
### Dependency
- Bump python from 3.10.5-slim-bullseye to 3.10.6-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5623](https://github.com/dbt-labs/dbt-core/pull/5623))
- Bump mashumaro[msgpack] from 3.0.3 to 3.0.4 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5649](https://github.com/dbt-labs/dbt-core/pull/5649))
### Contributors
- [@Goodkat](https://github.com/Goodkat) ([#5581](https://github.com/dbt-labs/dbt-core/pull/5581), [#5620](https://github.com/dbt-labs/dbt-core/pull/5620))
- [@Ilanbenb](https://github.com/Ilanbenb) ([#5505](https://github.com/dbt-labs/dbt-core/pull/5505))
- [@b-per](https://github.com/b-per) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5607](https://github.com/dbt-labs/dbt-core/pull/5607))
- [@epapineau](https://github.com/epapineau) ([#5395](https://github.com/dbt-labs/dbt-core/pull/5395))
- [@kadero](https://github.com/kadero) ([#4514](https://github.com/dbt-labs/dbt-core/pull/4514))
- [@matt-winkler](https://github.com/matt-winkler) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@shrodingers](https://github.com/shrodingers) ([#5292](https://github.com/dbt-labs/dbt-core/pull/5292))
- [@sungchun12](https://github.com/sungchun12) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))

140
.changes/1.3.0.md Normal file
View File

@@ -0,0 +1,140 @@
## dbt-core 1.3.0 - October 12, 2022
### Breaking Changes
- Renaming Metric Spec Attributes ([#5774](https://github.com/dbt-labs/dbt-core/issues/5774), [#5775](https://github.com/dbt-labs/dbt-core/pull/5775))
### Features
- Add `--defer` flag to dbt compile & dbt docs generate ([#4110](https://github.com/dbt-labs/dbt-core/issues/4110), [#4514](https://github.com/dbt-labs/dbt-core/pull/4514))
- Python model inital version ([#5261](https://github.com/dbt-labs/dbt-core/issues/5261), [#5421](https://github.com/dbt-labs/dbt-core/pull/5421))
- allows user to include the file extension for .py models in the dbt run -m command. ([#5289](https://github.com/dbt-labs/dbt-core/issues/5289), [#5295](https://github.com/dbt-labs/dbt-core/pull/5295))
- Incremental materialization refactor and cleanup ([#5245](https://github.com/dbt-labs/dbt-core/issues/5245), [#5359](https://github.com/dbt-labs/dbt-core/pull/5359))
- Python models can support incremental logic ([#0](https://github.com/dbt-labs/dbt-core/issues/0), [#35](https://github.com/dbt-labs/dbt-core/pull/35))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
- merge_exclude_columns for incremental materialization ([#5260](https://github.com/dbt-labs/dbt-core/issues/5260), [#5457](https://github.com/dbt-labs/dbt-core/pull/5457))
- add exponential backoff to connection retries on Postgres (and Redshift) ([#5502](https://github.com/dbt-labs/dbt-core/issues/5502), [#5503](https://github.com/dbt-labs/dbt-core/pull/5503))
- use MethodName.File when value ends with .csv ([#5578](https://github.com/dbt-labs/dbt-core/issues/5578), [#5581](https://github.com/dbt-labs/dbt-core/pull/5581))
- Make `docs` configurable in `dbt_project.yml` and add a `node_color` attribute to change the color of nodes in the DAG ([#5333](https://github.com/dbt-labs/dbt-core/issues/5333), [#5397](https://github.com/dbt-labs/dbt-core/pull/5397))
- Adding ResolvedMetricReference helper functions and tests ([#5567](https://github.com/dbt-labs/dbt-core/issues/5567), [#5607](https://github.com/dbt-labs/dbt-core/pull/5607))
- Check dbt-core version requirements when installing Hub packages ([#5648](https://github.com/dbt-labs/dbt-core/issues/5648), [#5651](https://github.com/dbt-labs/dbt-core/pull/5651))
- Search current working directory for `profiles.yml` ([#5411](https://github.com/dbt-labs/dbt-core/issues/5411), [#5717](https://github.com/dbt-labs/dbt-core/pull/5717))
- Adding the `window` parameter to the metric spec. ([#5721](https://github.com/dbt-labs/dbt-core/issues/5721), [#5722](https://github.com/dbt-labs/dbt-core/pull/5722))
- Add invocation args dict to ProviderContext class ([#5524](https://github.com/dbt-labs/dbt-core/issues/5524), [#5782](https://github.com/dbt-labs/dbt-core/pull/5782))
- Adds new cli framework ([#5526](https://github.com/dbt-labs/dbt-core/issues/5526), [#5647](https://github.com/dbt-labs/dbt-core/pull/5647))
- Flags work with new Click CLI ([#5529](https://github.com/dbt-labs/dbt-core/issues/5529), [#5790](https://github.com/dbt-labs/dbt-core/pull/5790))
- Add metadata env method to ProviderContext class ([#5522](https://github.com/dbt-labs/dbt-core/issues/5522), [#5794](https://github.com/dbt-labs/dbt-core/pull/5794))
- Array macros ([#5520](https://github.com/dbt-labs/dbt-core/issues/5520), [#5823](https://github.com/dbt-labs/dbt-core/pull/5823))
- Add enabled config to exposures and metrics ([#5422](https://github.com/dbt-labs/dbt-core/issues/5422), [#5815](https://github.com/dbt-labs/dbt-core/pull/5815))
- Migrate dbt-utils current_timestamp macros into core + adapters ([#5521](https://github.com/dbt-labs/dbt-core/issues/5521), [#5838](https://github.com/dbt-labs/dbt-core/pull/5838))
- add -fr flag shorthand ([#5878](https://github.com/dbt-labs/dbt-core/issues/5878), [#5879](https://github.com/dbt-labs/dbt-core/pull/5879))
- add type_boolean as a data type macro ([#5739](https://github.com/dbt-labs/dbt-core/issues/5739), [#5875](https://github.com/dbt-labs/dbt-core/pull/5875))
- Support .dbtignore in project root to ignore certain files being read by dbt ([#5733](https://github.com/dbt-labs/dbt-core/issues/5733), [#5897](https://github.com/dbt-labs/dbt-core/pull/5897))
- This conditionally no-ops warehouse connection at compile depending on an env var, disabling introspection/queries during compilation only. This is a temporary solution to more complex permissions requirements for the semantic layer. ([#5936](https://github.com/dbt-labs/dbt-core/issues/5936), [#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
### Fixes
- Remove the default 256 characters limit on postgres character varying type when no limitation is set ([#5238](https://github.com/dbt-labs/dbt-core/issues/5238), [#5292](https://github.com/dbt-labs/dbt-core/pull/5292))
- Include schema file config in unrendered_config ([#5338](https://github.com/dbt-labs/dbt-core/issues/5338), [#5344](https://github.com/dbt-labs/dbt-core/pull/5344))
- Add context to compilation errors generated while rendering generic test configuration values. ([#5294](https://github.com/dbt-labs/dbt-core/issues/5294), [#5393](https://github.com/dbt-labs/dbt-core/pull/5393))
- Resolves #5351 - Do not consider shorter varchar cols as schema changes ([#5351](https://github.com/dbt-labs/dbt-core/issues/5351), [#5395](https://github.com/dbt-labs/dbt-core/pull/5395))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- on_shchma_change fail verbosity enhancement ([#5504](https://github.com/dbt-labs/dbt-core/issues/5504), [#5505](https://github.com/dbt-labs/dbt-core/pull/5505))
- Ignore empty strings passed in as secrets ([#5312](https://github.com/dbt-labs/dbt-core/issues/5312), [#5518](https://github.com/dbt-labs/dbt-core/pull/5518))
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
- Extended validations for the project names ([#5379](https://github.com/dbt-labs/dbt-core/issues/5379), [#5620](https://github.com/dbt-labs/dbt-core/pull/5620))
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
- multiple args for ref and source ([#5634](https://github.com/dbt-labs/dbt-core/issues/5634), [#5635](https://github.com/dbt-labs/dbt-core/pull/5635))
- Fix Unexpected behavior when chaining methods on dbt-ref'ed/sourced dataframes ([#5646](https://github.com/dbt-labs/dbt-core/issues/5646), [#5677](https://github.com/dbt-labs/dbt-core/pull/5677))
- Fix typos of comments in core/dbt/adapters/ ([#5690](https://github.com/dbt-labs/dbt-core/issues/5690), [#5693](https://github.com/dbt-labs/dbt-core/pull/5693))
- Include py.typed in MANIFEST.in. This enables packages that install dbt-core from pypi to use mypy. ([#5703](https://github.com/dbt-labs/dbt-core/issues/5703), [#5703](https://github.com/dbt-labs/dbt-core/pull/5703))
- Removal of all .coverage files when using make clean command ([#5633](https://github.com/dbt-labs/dbt-core/issues/5633), [#5759](https://github.com/dbt-labs/dbt-core/pull/5759))
- Remove temp files generated by unit tests ([#5631](https://github.com/dbt-labs/dbt-core/issues/5631), [#5749](https://github.com/dbt-labs/dbt-core/pull/5749))
- Fix warnings as errors during tests ([#5424](https://github.com/dbt-labs/dbt-core/issues/5424), [#5800](https://github.com/dbt-labs/dbt-core/pull/5800))
- Prevent event_history from holding references ([#5848](https://github.com/dbt-labs/dbt-core/issues/5848), [#5858](https://github.com/dbt-labs/dbt-core/pull/5858))
- Account for disabled flags on models in schema files more completely ([#3992](https://github.com/dbt-labs/dbt-core/issues/3992), [#5868](https://github.com/dbt-labs/dbt-core/pull/5868))
- ConfigSelectorMethod should check for bools ([#5890](https://github.com/dbt-labs/dbt-core/issues/5890), [#5889](https://github.com/dbt-labs/dbt-core/pull/5889))
- shorthand for full refresh should be one character ([#5878](https://github.com/dbt-labs/dbt-core/issues/5878), [#5908](https://github.com/dbt-labs/dbt-core/pull/5908))
- Fix macro resolution order during static analysis for custom generic tests ([#5720](https://github.com/dbt-labs/dbt-core/issues/5720), [#5907](https://github.com/dbt-labs/dbt-core/pull/5907))
- Fix race condition when invoking dbt via lib.py concurrently ([#5919](https://github.com/dbt-labs/dbt-core/issues/5919), [#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
- check length of args of python model function before accessing it ([#6041](https://github.com/dbt-labs/dbt-core/issues/6041), [#6042](https://github.com/dbt-labs/dbt-core/pull/6042))
### Docs
- Update dependency inline-source from ^6.1.5 to ^7.2.0 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Update dependency jest from ^26.2.2 to ^28.1.3 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Update dependency underscore from ^1.9.0 to ^1.13.4 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Update dependency webpack-cli from ^3.3.12 to ^4.7.0 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Update dependency webpack-dev-server from ^3.1.11 to ^4.9.3 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Searches no longer require perfect matches, and instead consider each word individually. `my model` or `model my` will now find `my_model`, without the need for underscores ([dbt-docs/#143](https://github.com/dbt-labs/dbt-docs/issues/143), [dbt-docs/#145](https://github.com/dbt-labs/dbt-docs/pull/145))
- Support the renaming of SQL to code happening in dbt-core ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#292](https://github.com/dbt-labs/dbt-docs/pull/292))
- Leverages `docs.node_color` from `dbt-core` to color nodes in the DAG ([dbt-docs/#44](https://github.com/dbt-labs/dbt-docs/issues/44), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- Refer to exposures by their label by default. ([dbt-docs/#306](https://github.com/dbt-labs/dbt-docs/issues/306), [dbt-docs/#307](https://github.com/dbt-labs/dbt-docs/pull/307))
### Under the Hood
- Added language to tracked fields in run_model event ([#5571](https://github.com/dbt-labs/dbt-core/issues/5571), [#5469](https://github.com/dbt-labs/dbt-core/pull/5469))
- Update mashumaro to 3.0.3 ([#4940](https://github.com/dbt-labs/dbt-core/issues/4940), [#5118](https://github.com/dbt-labs/dbt-core/pull/5118))
- Add python incremental materialization test ([#0000](https://github.com/dbt-labs/dbt-core/issues/0000), [#5571](https://github.com/dbt-labs/dbt-core/pull/5571))
- Save use of default env vars to manifest to enable partial parsing in those cases. ([#5155](https://github.com/dbt-labs/dbt-core/issues/5155), [#5589](https://github.com/dbt-labs/dbt-core/pull/5589))
- add more information to log line interop test failures ([#5658](https://github.com/dbt-labs/dbt-core/issues/5658), [#5659](https://github.com/dbt-labs/dbt-core/pull/5659))
- Add supported languages to materializations ([#5569](https://github.com/dbt-labs/dbt-core/issues/5569), [#5695](https://github.com/dbt-labs/dbt-core/pull/5695))
- Migrate integration test 014 but also fix the snapshot hard delete test's timezone logic and force all integration tests to run flags.set_from_args to force environment variables are accessible to all integration test threads. ([#5760](https://github.com/dbt-labs/dbt-core/issues/5760), [#5760](https://github.com/dbt-labs/dbt-core/pull/5760))
- Support dbt-metrics compilation by rebuilding flat_graph ([#5525](https://github.com/dbt-labs/dbt-core/issues/5525), [#5786](https://github.com/dbt-labs/dbt-core/pull/5786))
- Reworking the way we define the window attribute of metrics to match freshness tests ([#5722](https://github.com/dbt-labs/dbt-core/issues/5722), [#5793](https://github.com/dbt-labs/dbt-core/pull/5793))
- Add PythonJobHelper base class in core and add more type checking ([#5802](https://github.com/dbt-labs/dbt-core/issues/5802), [#5802](https://github.com/dbt-labs/dbt-core/pull/5802))
- The link did not go to the anchor directly, now it does ([#5813](https://github.com/dbt-labs/dbt-core/issues/5813), [#5814](https://github.com/dbt-labs/dbt-core/pull/5814))
- remove key as reserved keyword from test_bool_or ([#5817](https://github.com/dbt-labs/dbt-core/issues/5817), [#5818](https://github.com/dbt-labs/dbt-core/pull/5818))
- Convert default selector tests to pytest ([#5728](https://github.com/dbt-labs/dbt-core/issues/5728), [#5820](https://github.com/dbt-labs/dbt-core/pull/5820))
- Compatibiltiy for metric attribute renaming ([#5807](https://github.com/dbt-labs/dbt-core/issues/5807), [#5825](https://github.com/dbt-labs/dbt-core/pull/5825))
- remove source quoting setting in adapter tests ([#5836](https://github.com/dbt-labs/dbt-core/issues/5836), [#5839](https://github.com/dbt-labs/dbt-core/pull/5839))
- Add name validation for metrics ([#5456](https://github.com/dbt-labs/dbt-core/issues/5456), [#5841](https://github.com/dbt-labs/dbt-core/pull/5841))
- Validate exposure name and add label ([#5606](https://github.com/dbt-labs/dbt-core/issues/5606), [#5844](https://github.com/dbt-labs/dbt-core/pull/5844))
- Adding validation for metric expression attribute ([#5871](https://github.com/dbt-labs/dbt-core/issues/5871), [#5873](https://github.com/dbt-labs/dbt-core/pull/5873))
- Profiling and Adapter Management work with Click CLI ([#5531](https://github.com/dbt-labs/dbt-core/issues/5531), [#5892](https://github.com/dbt-labs/dbt-core/pull/5892))
- Reparse references to deleted metric ([#5444](https://github.com/dbt-labs/dbt-core/issues/5444), [#5920](https://github.com/dbt-labs/dbt-core/pull/5920))
### Dependencies
- Upgrade to Jinja2==3.1.2 from Jinja2==2.11.3 ([#4748](https://github.com/dbt-labs/dbt-core/issues/4748), [#5465](https://github.com/dbt-labs/dbt-core/pull/5465))
- Bump mypy from 0.961 to 0.971 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5495](https://github.com/dbt-labs/dbt-core/pull/5495))
- Remove pin for MarkUpSafe from >=0.23,<2.1 ([#5506](https://github.com/dbt-labs/dbt-core/issues/5506), [#5507](https://github.com/dbt-labs/dbt-core/pull/5507))
### Dependency
- Bump python from 3.10.5-slim-bullseye to 3.10.6-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5623](https://github.com/dbt-labs/dbt-core/pull/5623))
- Bump mashumaro[msgpack] from 3.0.3 to 3.0.4 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5649](https://github.com/dbt-labs/dbt-core/pull/5649))
- Bump black from 22.6.0 to 22.8.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5750](https://github.com/dbt-labs/dbt-core/pull/5750))
- Bump python from 3.10.6-slim-bullseye to 3.10.7-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5805](https://github.com/dbt-labs/dbt-core/pull/5805))
### Contributors
- [@Goodkat](https://github.com/Goodkat) ([#5581](https://github.com/dbt-labs/dbt-core/pull/5581), [#5518](https://github.com/dbt-labs/dbt-core/pull/5518), [#5620](https://github.com/dbt-labs/dbt-core/pull/5620))
- [@Ilanbenb](https://github.com/Ilanbenb) ([#5505](https://github.com/dbt-labs/dbt-core/pull/5505))
- [@b-per](https://github.com/b-per) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@bbroeksema](https://github.com/bbroeksema) ([#5749](https://github.com/dbt-labs/dbt-core/pull/5749))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5775](https://github.com/dbt-labs/dbt-core/pull/5775), [#5607](https://github.com/dbt-labs/dbt-core/pull/5607), [#5722](https://github.com/dbt-labs/dbt-core/pull/5722), [#5793](https://github.com/dbt-labs/dbt-core/pull/5793), [#5825](https://github.com/dbt-labs/dbt-core/pull/5825), [#5873](https://github.com/dbt-labs/dbt-core/pull/5873))
- [@danielcmessias](https://github.com/danielcmessias) ([#5889](https://github.com/dbt-labs/dbt-core/pull/5889))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#5457](https://github.com/dbt-labs/dbt-core/pull/5457), [#5879](https://github.com/dbt-labs/dbt-core/pull/5879), [#5908](https://github.com/dbt-labs/dbt-core/pull/5908))
- [@dbeatty10](https://github.com/dbeatty10) ([#5717](https://github.com/dbt-labs/dbt-core/pull/5717), [#5823](https://github.com/dbt-labs/dbt-core/pull/5823))
- [@drewbanin](https://github.com/drewbanin) ([#5921](https://github.com/dbt-labs/dbt-core/pull/5921), [dbt-docs/#292](https://github.com/dbt-labs/dbt-docs/pull/292))
- [@epapineau](https://github.com/epapineau) ([#5395](https://github.com/dbt-labs/dbt-core/pull/5395))
- [@graciegoheen](https://github.com/graciegoheen) ([#5823](https://github.com/dbt-labs/dbt-core/pull/5823))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5782](https://github.com/dbt-labs/dbt-core/pull/5782), [#5794](https://github.com/dbt-labs/dbt-core/pull/5794), [#5759](https://github.com/dbt-labs/dbt-core/pull/5759))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@joellabes](https://github.com/joellabes) ([dbt-docs/#145](https://github.com/dbt-labs/dbt-docs/pull/145))
- [@jpmmcneill](https://github.com/jpmmcneill) ([#5875](https://github.com/dbt-labs/dbt-core/pull/5875))
- [@kadero](https://github.com/kadero) ([#4514](https://github.com/dbt-labs/dbt-core/pull/4514))
- [@leoebfolsom](https://github.com/leoebfolsom) ([#5295](https://github.com/dbt-labs/dbt-core/pull/5295))
- [@matt-winkler](https://github.com/matt-winkler) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@nicholasyager](https://github.com/nicholasyager) ([#5393](https://github.com/dbt-labs/dbt-core/pull/5393))
- [@panasenco](https://github.com/panasenco) ([#5703](https://github.com/dbt-labs/dbt-core/pull/5703))
- [@racheldaniel](https://github.com/racheldaniel) ([#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
- [@sdebruyn](https://github.com/sdebruyn) ([#5814](https://github.com/dbt-labs/dbt-core/pull/5814), [#5818](https://github.com/dbt-labs/dbt-core/pull/5818), [#5839](https://github.com/dbt-labs/dbt-core/pull/5839))
- [@shrodingers](https://github.com/shrodingers) ([#5292](https://github.com/dbt-labs/dbt-core/pull/5292))
- [@sungchun12](https://github.com/sungchun12) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- [@yoiki](https://github.com/yoiki) ([#5693](https://github.com/dbt-labs/dbt-core/pull/5693))
- [@chamini2](https://github.com/chamini2) ([#6042](https://github.com/dbt-labs/dbt-core/pull/6042))

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: Upgrade to Jinja2==3.1.2 from Jinja2==2.11.3
time: 2022-07-19T14:16:46.665631-05:00
custom:
Author: emmyoop
Issue: "4748"
PR: "5465"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: "Bump mypy from 0.961 to 0.971"
time: 2022-07-20T00:07:41.000000-05:00
custom:
Author: dependabot[bot]
Issue: "4904"
PR: "5495"

View File

@@ -1,7 +0,0 @@
kind: Dependencies
body: Remove pin for MarkUpSafe from >=0.23,<2.1
time: 2022-07-21T09:32:33.494002-05:00
custom:
Author: emmyoop
Issue: "5506"
PR: "5507"

View File

@@ -1,7 +0,0 @@
kind: Dependency
body: "Bump python from 3.10.5-slim-bullseye to 3.10.6-slim-bullseye in /docker"
time: 2022-08-08T13:23:27.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 5623

View File

@@ -1,7 +0,0 @@
kind: Dependency
body: "Bump mashumaro[msgpack] from 3.0.3 to 3.0.4 in /core"
time: 2022-08-15T18:27:31.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 5649

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Update dependency inline-source from ^6.1.5 to ^7.2.0
time: 2022-07-28T14:02:58.441963-07:00
custom:
Author: emmyoop
Issue: "299"
PR: "291"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Update dependency jest from ^26.2.2 to ^28.1.3
time: 2022-07-28T14:03:29.837274-07:00
custom:
Author: emmyoop
Issue: "299"
PR: "291"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Update dependency underscore from ^1.9.0 to ^1.13.4
time: 2022-07-28T14:03:51.123441-07:00
custom:
Author: emmyoop
Issue: "299"
PR: "291"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Update dependency webpack-cli from ^3.3.12 to ^4.7.0
time: 2022-07-28T14:04:25.629638-07:00
custom:
Author: emmyoop
Issue: "299"
PR: "291"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Update dependency webpack-dev-server from ^3.1.11 to ^4.9.3
time: 2022-07-28T14:04:49.637369-07:00
custom:
Author: emmyoop
Issue: "299"
PR: "291"

View File

@@ -1,8 +0,0 @@
kind: Docs
body: Searches no longer require perfect matches, and instead consider each word individually.
`my model` or `model my` will now find `my_model`, without the need for underscores
time: 2022-07-28T14:06:20.371364-07:00
custom:
Author: joellabes
Issue: "143"
PR: "145"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Support the renaming of SQL to code happening in dbt-core
time: 2022-07-28T14:08:06.184934-07:00
custom:
Author: jtcohen6 stu-k drewbanin ChenyuLInx
Issue: "299"
PR: "292"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Leverages `docs.node_color` from `dbt-core` to color nodes in the DAG
time: 2022-08-04T13:41:38.669987-05:00
custom:
Author: matt-winkler sungchun12 b-per
Issue: "44"
PR: "281"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add `--defer` flag to dbt compile & dbt docs generate
time: 2022-03-14T18:11:27.783107864+01:00
custom:
Author: kadero
Issue: "4110"
PR: "4514"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Python model inital version
time: 2022-05-10T16:51:30.245589-07:00
custom:
Author: ChenyuLInx
Issue: "5261"
PR: "5421"

View File

@@ -1,7 +0,0 @@
kind: Features
body: allows user to include the file extension for .py models in the dbt run -m command.
time: 2022-05-26T16:53:23.389671-07:00
custom:
Author: leoebfolsom
Issue: "5289"
PR: "5295"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Incremental materialization refactor and cleanup
time: 2022-06-10T10:56:47.226887-04:00
custom:
Author: gshank
Issue: "5245"
PR: "5359"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Python models can support incremental logic
time: 2022-06-27T13:10:42.123303-05:00
custom:
Author: iknox-fa
Issue: "0"
PR: "35"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Add reusable function for retrying adapter connections. Utilize said function
to add retries for Postgres (and Redshift).
time: 2022-07-15T03:55:55.270637265+02:00
custom:
Author: tomasfarias
Issue: "5022"
PR: "5432"

View File

@@ -1,7 +0,0 @@
kind: Features
body: add exponential backoff to connection retries on Postgres (and Redshift)
time: 2022-07-20T17:12:57.486949-04:00
custom:
Author: nathaniel-may
Issue: "5502"
PR: "5503"

View File

@@ -1,7 +0,0 @@
kind: Features
body: use MethodName.File when value ends with .csv
time: 2022-07-29T17:32:31.395677157+02:00
custom:
Author: Goodkat
Issue: "5578"
PR: "5581"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Make `docs` configurable in `dbt_project.yml` and add a `node_color` attribute
to change the color of nodes in the DAG
time: 2022-08-03T10:42:30.60624+02:00
custom:
Author: matt-winkler sungchun12 b-per
Issue: "5333"
PR: "5397"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adding ResolvedMetricReference helper functions and tests
time: 2022-08-04T12:09:36.202919-04:00
custom:
Author: callum-mcdata
Issue: "5567"
PR: "5607"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Check dbt-core version requirements when installing Hub packages
time: 2022-08-15T13:43:12.965143+01:00
custom:
Author: jtcohen6
Issue: "5648"
PR: "5651"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Remove the default 256 characters limit on postgres character varying type when
no limitation is set
time: 2022-05-23T10:38:43.392232+02:00
custom:
Author: shrodingers
Issue: "5238"
PR: "5292"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Include schema file config in unrendered_config
time: 2022-06-07T12:30:58.535207-04:00
custom:
Author: gshank
Issue: "5338"
PR: "5344"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: 'Resolves #5351 - Do not consider shorter varchar cols as schema changes'
time: 2022-06-17T19:37:31.885484-07:00
custom:
Author: epapineau
Issue: "5351"
PR: "5395"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Rename try to strict for more intuitiveness
time: 2022-07-15T23:11:48.327928+12:00
custom:
Author: jeremyyeo
Issue: "5475"
PR: "5477"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: on_shchma_change fail verbosity enhancement
time: 2022-07-20T21:37:46.474241+03:00
custom:
Author: Ilanbenb
Issue: "5504"
PR: "5505"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Ignore empty strings passed in as secrets
time: 2022-07-23T21:53:30.907759094+02:00
custom:
Author: Goodkat
Issue: "5312"
PR: "5518"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix handling of top-level exceptions
time: 2022-07-26T11:36:36.824979-04:00
custom:
Author: gshank
Issue: "5564"
PR: "5560"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix error rendering docs block in metrics description
time: 2022-08-03T14:42:21.386265-04:00
custom:
Author: gshank
Issue: "5585"
PR: "5603"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Extended validations for the project names
time: 2022-08-05T22:10:22.746830854+02:00
custom:
Author: Goodkat
Issue: "5379"
PR: "5620"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Use sys.exit instead of exit
time: 2022-08-08T11:20:01.838171926-04:00
custom:
Author: varun-dc
Issue: "5621"
PR: "5627"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Finishing logic upgrade to Redshift for name truncation collisions.
time: 2022-08-15T23:04:09.173645-07:00
custom:
Author: versusfacit
Issue: "5586"
PR: "5656"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: multiple args for ref and source
time: 2022-08-16T15:34:01.348339-07:00
custom:
Author: ChenyuLInx
Issue: "5634"
PR: "5635"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix Unexpected behavior when chaining methods on dbt-ref'ed/sourced dataframes
time: 2022-08-17T16:36:42.678275-07:00
custom:
Author: ChenyuLInx
Issue: "5646"
PR: "5677"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Add context to compilation errors generated while rendering generic test configuration
values.
time: 2022-06-17T15:07:44.751037-04:00
custom:
Author: nicholasyager
Issue: "5294"
PR: "5393"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Added language to tracked fields in run_model event
time: 2022-07-13T12:49:25.362678-05:00
custom:
Author: stu-k
Issue: "5571"
PR: "5469"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Update mashumaro to 3.0.3
time: 2022-07-20T11:52:26.210876-04:00
custom:
Author: gshank
Issue: "4940"
PR: "5118"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add python incremental materialization test
time: 2022-07-28T09:45:36.13608-05:00
custom:
Author: stu-k
Issue: "0000"
PR: "5571"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Save use of default env vars to manifest to enable partial parsing in those
cases.
time: 2022-08-02T11:29:36.417589-05:00
custom:
Author: emmyoop
Issue: "5155"
PR: "5589"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: add more information to log line interop test failures
time: 2022-08-16T12:20:32.119588+01:00
custom:
Author: nathaniel-may
Issue: "5658"
PR: "5659"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add supported languages to materializations
time: 2022-08-22T10:37:39.50743-05:00
custom:
Author: stu-k
Issue: "5569"
PR: "5695"

10
.changes/1.3.1.md Normal file
View File

@@ -0,0 +1,10 @@
## dbt-core 1.3.1 - November 16, 2022
### Features
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201), [#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
### Docs
- ([dbt-docs/#5880](https://github.com/dbt-labs/dbt-docs/issues/5880), [dbt-docs/#324](https://github.com/dbt-labs/dbt-docs/pull/324))
- Fix rendering of sample code for metrics ([dbt-docs/#323](https://github.com/dbt-labs/dbt-docs/issues/323), [dbt-docs/#346](https://github.com/dbt-labs/dbt-docs/pull/346))
### Contributors
- [@paulbenschmidt](https://github.com/paulbenschmidt) ([dbt-docs/#324](https://github.com/dbt-labs/dbt-docs/pull/324))
- [@racheldaniel](https://github.com/racheldaniel) ([#6202](https://github.com/dbt-labs/dbt-core/pull/6202))

5
.changes/1.3.2.md Normal file
View File

@@ -0,0 +1,5 @@
## dbt-core 1.3.2 - January 04, 2023
### Fixes
- Bug when partial parsing with an empty schema file ([#4850](https://github.com/dbt-labs/dbt-core/issues/4850), [#<no value>](https://github.com/dbt-labs/dbt-core/pull/<no value>))

8
.changes/1.3.3.md Normal file
View File

@@ -0,0 +1,8 @@
## dbt-core 1.3.3 - February 28, 2023
### Fixes
- add pytz dependency ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
### Contributors
- [@sdebruyn](https://github.com/sdebruyn) ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))

8
.changes/1.3.4.md Normal file
View File

@@ -0,0 +1,8 @@
## dbt-core 1.3.4 - April 19, 2023
### Fixes
- Improved failed event serialization handling and associated tests ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))
### Contributors
- [@QMalcolm](https://github.com/QMalcolm) ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))

View File

@@ -1,7 +0,0 @@
kind: Breaking Changes
body: Renaming Metric Spec Attributes
time: 2022-09-06T15:45:21.2769-05:00
custom:
Author: callum-mcdata
Issue: "5774"
PR: "5775"

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Bump black from 22.6.0 to 22.8.0"
time: 2022-09-06T13:48:58.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 5750

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Bump python from 3.10.6-slim-bullseye to 3.10.7-slim-bullseye in /docker"
time: 2022-09-12T00:22:53.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 5805

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adding the `window` parameter to the metric spec.
time: 2022-08-31T12:13:19.48356-05:00
custom:
Author: callum-mcdata
Issue: "5721"
PR: "5722"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add invocation args dict to ProviderContext class
time: 2022-09-08T08:13:15.17337+01:00
custom:
Author: jared-rimmer
Issue: "5524"
PR: "5782"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Adds new cli framework
time: 2022-09-08T10:41:49.375734-05:00
custom:
Author: iknox-fa
Issue: "5526"
PR: "5647"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add enabled config to exposures and metrics
time: 2022-09-13T09:59:24.445918-05:00
custom:
Author: emmyoop
Issue: "5422"
PR: "5815"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Include py.typed in MANIFEST.in. This enables packages that install dbt-core
from pypi to use mypy.
time: 2022-08-23T11:26:33.8415455-07:00
custom:
Author: panasenco
Issue: "5703"
PR: "5703"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Removal of all .coverage files when using make clean command
time: 2022-09-03T15:54:09.741554082+01:00
custom:
Author: jared-rimmer
Issue: "5633"
PR: "5759"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Remove temp files generated by unit tests
time: 2022-09-09T09:28:32.590208+02:00
custom:
Author: bbroeksema
Issue: "5631"
PR: "5749"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix warnings as errors during tests
time: 2022-09-09T09:56:27.90654-05:00
custom:
Author: stu-k
Issue: "5424"
PR: "5800"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Migrate integration test 014 but also fix the snapshot hard delete test's timezone logic and force all integration tests to run flags.set_from_args to force environment variables are accessible to all integration test threads.
time: 2022-09-05T00:17:49.564534-07:00
custom:
Author: versusfacit
Issue: "5760"
PR: "5760"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Support dbt-metrics compilation by rebuilding flat_graph
time: 2022-09-08T14:56:44.173322-04:00
custom:
Author: gshank
Issue: "5525"
PR: "5786"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Reworking the way we define the window attribute of metrics to match freshness
tests
time: 2022-09-08T18:07:31.532608-05:00
custom:
Author: callum-mcdata
Issue: "5722"
PR: "5793"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add PythonJobHelper base class in core and add more type checking
time: 2022-09-09T11:52:20.419364-07:00
custom:
Author: ChenyuLInx
Issue: "5802"
PR: "5802"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert default selector tests to pytest
time: 2022-09-12T13:40:00.625912-05:00
custom:
Author: stu-k
Issue: "5728"
PR: "5820"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: The link did not go to the anchor directly, now it does
time: 2022-09-12T14:00:35.899828+02:00
custom:
Author: sdebruyn
Issue: "5813"
PR: "5814"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: remove key as reserved keyword from test_bool_or
time: 2022-09-12T19:03:41.481601+02:00
custom:
Author: sdebruyn
Issue: "5817"
PR: "5818"

113
.changie.yaml Executable file → Normal file
View File

@@ -6,19 +6,67 @@ changelogPath: CHANGELOG.md
versionExt: md
versionFormat: '## dbt-core {{.Version}} - {{.Time.Format "January 02, 2006"}}'
kindFormat: '### {{.Kind}}'
changeFormat: '- {{.Body}} ([#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), [#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
kinds:
- label: Breaking Changes
- label: Features
- label: Fixes
- label: Docs
changeFormat: '- {{.Body}} ([dbt-docs/#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-docs/issues/{{.Custom.Issue}}), [dbt-docs/#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-docs/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[dbt-docs/#nbr](https://github.com/dbt-labs/dbt-docs/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: '- {{.Body}} ({{if ne .Custom.Issue ""}}[#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), {{end}}[#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: '- {{.Body}} ({{if ne .Custom.Issue ""}}[#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), {{end}}[#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
@@ -33,42 +81,47 @@ custom:
type: string
minLength: 3
- key: Issue
label: GitHub Issue Number
type: int
minInt: 1
- key: PR
label: GitHub Pull Request Number
type: int
minInt: 1
label: GitHub Issue Number (separated by a single space if multiple)
type: string
minLength: 1
footerFormat: |
{{- $contributorDict := dict }}
{{- /* any names added to this list should be all lowercase for later matching purposes */}}
{{- $core_team := list "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "dependabot[bot]" "snyk-bot" }}
{{- $core_team := list "michelleark" "peterallenwebb" "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "aranke" "dependabot[bot]" "snyk-bot" "colin-rogers-dbt" }}
{{- range $change := .Changes }}
{{- $authorList := splitList " " $change.Custom.Author }}
{{- /* loop through all authors for a PR */}}
{{- /* loop through all authors for a single changelog */}}
{{- range $author := $authorList }}
{{- $authorLower := lower $author }}
{{- /* we only want to include non-core team contributors */}}
{{- if not (has $authorLower $core_team)}}
{{- /* Docs kind link back to dbt-docs instead of dbt-core PRs */}}
{{- $prLink := $change.Kind }}
{{- if eq $change.Kind "Docs" }}
{{- $prLink = "[dbt-docs/#pr](https://github.com/dbt-labs/dbt-docs/pull/pr)" | replace "pr" $change.Custom.PR }}
{{- else }}
{{- $prLink = "[#pr](https://github.com/dbt-labs/dbt-core/pull/pr)" | replace "pr" $change.Custom.PR }}
{{- end }}
{{- /* check if this contributor has other PRs associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $prList := get $contributorDict $author }}
{{- $prList = append $prList $prLink }}
{{- $contributorDict := set $contributorDict $author $prList }}
{{- else }}
{{- $prList := list $prLink }}
{{- $contributorDict := set $contributorDict $author $prList }}
{{- end }}
{{- end}}
{{- $changeList := splitList " " $change.Custom.Author }}
{{- $IssueList := list }}
{{- $changeLink := $change.Kind }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }}
{{- $changes := splitList " " $change.Custom.PR }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- else }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- end }}
{{- /* check if this contributor has other changes associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $contributionList := get $contributorDict $author }}
{{- $contributionList = concat $contributionList $IssueList }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- else }}
{{- $contributionList := $IssueList }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- end }}
{{- end}}
{{- end}}
{{- end }}
{{- /* no indentation here for formatting so the final markdown doesn't have unneeded indentations */}}

View File

@@ -40,7 +40,7 @@ jobs:
matrix:
include:
- label: "dependencies"
changie_kind: "Dependency"
changie_kind: "Dependencies"
- label: "snyk"
changie_kind: "Security"
runs-on: ubuntu-latest
@@ -58,4 +58,4 @@ jobs:
commit_message: "Add automated changelog yaml from template for bot PR"
changie_kind: ${{ matrix.changie_kind }}
label: ${{ matrix.label }}
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n Issue: 4904\n PR: ${{ github.event.pull_request.number }}"
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n PR: ${{ github.event.pull_request.number }}"

View File

@@ -15,6 +15,9 @@ on:
issues:
types: [closed, deleted, reopened]
# no special access is needed
permissions: read-all
jobs:
call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-transition.yml@main

View File

@@ -45,7 +45,9 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: '3.8'
- name: Install python dependencies
run: |
@@ -82,7 +84,7 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: ${{ matrix.python-version }}
@@ -117,7 +119,7 @@ jobs:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10"]
os: [ubuntu-latest]
os: [ubuntu-20.04]
include:
- python-version: 3.8
os: windows-latest
@@ -137,7 +139,7 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: ${{ matrix.python-version }}
@@ -190,9 +192,9 @@ jobs:
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4.3.0
with:
python-version: 3.8
python-version: '3.8'
- name: Install python dependencies
run: |

109
.github/workflows/nightly-release.yml vendored Normal file
View File

@@ -0,0 +1,109 @@
# **what?**
# Nightly releases to GitHub and PyPI. This workflow produces the following outcome:
# - generate and validate data for night release (commit SHA, version number, release branch);
# - pass data to release workflow;
# - night release will be pushed to GitHub as a draft release;
# - night build will be pushed to test PyPI;
#
# **why?**
# Ensure an automated and tested release process for nightly builds
#
# **when?**
# This workflow runs on schedule or can be run manually on demand.
name: Nightly Test Release to GitHub and PyPI
on:
workflow_dispatch: # for manual triggering
schedule:
- cron: 0 9 * * *
permissions:
contents: write # this is the permission that allows creating a new release
defaults:
run:
shell: bash
env:
RELEASE_BRANCH: "main"
jobs:
aggregate-release-data:
runs-on: ubuntu-latest
outputs:
commit_sha: ${{ steps.resolve-commit-sha.outputs.release_commit }}
version_number: ${{ steps.nightly-release-version.outputs.number }}
release_branch: ${{ steps.release-branch.outputs.name }}
steps:
- name: "Checkout ${{ github.repository }} Branch ${{ env.RELEASE_BRANCH }}"
uses: actions/checkout@v3
with:
ref: ${{ env.RELEASE_BRANCH }}
- name: "Resolve Commit To Release"
id: resolve-commit-sha
run: |
commit_sha=$(git rev-parse HEAD)
echo "release_commit=$commit_sha" >> $GITHUB_OUTPUT
- name: "Get Current Version Number"
id: version-number-sources
run: |
current_version=`awk -F"current_version = " '{print $2}' .bumpversion.cfg | tr '\n' ' '`
echo "current_version=$current_version" >> $GITHUB_OUTPUT
- name: "Audit Version And Parse Into Parts"
id: semver
uses: dbt-labs/actions/parse-semver@v1.1.0
with:
version: ${{ steps.version-number-sources.outputs.current_version }}
- name: "Get Current Date"
id: current-date
run: echo "date=$(date +'%m%d%Y')" >> $GITHUB_OUTPUT
- name: "Generate Nightly Release Version Number"
id: nightly-release-version
run: |
number="${{ steps.semver.outputs.version }}.dev${{ steps.current-date.outputs.date }}+nightly"
echo "number=$number" >> $GITHUB_OUTPUT
- name: "Audit Nightly Release Version And Parse Into Parts"
uses: dbt-labs/actions/parse-semver@v1.1.0
with:
version: ${{ steps.nightly-release-version.outputs.number }}
- name: "Set Release Branch"
id: release-branch
run: |
echo "name=${{ env.RELEASE_BRANCH }}" >> $GITHUB_OUTPUT
log-outputs-aggregate-release-data:
runs-on: ubuntu-latest
needs: [aggregate-release-data]
steps:
- name: "[DEBUG] Log Outputs"
run: |
echo commit_sha : ${{ needs.aggregate-release-data.outputs.commit_sha }}
echo version_number: ${{ needs.aggregate-release-data.outputs.version_number }}
echo release_branch: ${{ needs.aggregate-release-data.outputs.release_branch }}
release-github-pypi:
needs: [aggregate-release-data]
uses: ./.github/workflows/release.yml
with:
sha: ${{ needs.aggregate-release-data.outputs.commit_sha }}
target_branch: ${{ needs.aggregate-release-data.outputs.release-branch }}
version_number: ${{ needs.aggregate-release-data.outputs.version_number }}
build_script_path: "scripts/build-dist.sh"
env_setup_script_path: "scripts/env-setup.sh"
s3_bucket_name: "core-team-artifacts"
package_test_command: "dbt --version"
test_run: true
nightly_release: true
secrets: inherit

View File

@@ -1,199 +1,230 @@
# **what?**
# Take the given commit, run unit tests specifically on that sha, build and
# package it, and then release to GitHub and PyPi with that specific build
# Release workflow provides the following steps:
# - checkout the given commit;
# - validate version in sources and changelog file for given version;
# - bump the version and generate a changelog if needed;
# - merge all changes to the target branch if needed;
# - run unit and integration tests against given commit;
# - build and package that SHA;
# - release it to GitHub and PyPI with that specific build;
#
# **why?**
# Ensure an automated and tested release process
#
# **when?**
# This will only run manually with a given sha and version
# This workflow can be run manually on demand or can be called by other workflows
name: Release to GitHub and PyPi
name: Release to GitHub and PyPI
on:
workflow_dispatch:
inputs:
sha:
description: 'The last commit sha in the release'
required: true
description: "The last commit sha in the release"
type: string
required: true
target_branch:
description: "The branch to release from"
type: string
required: true
version_number:
description: 'The release version number (i.e. 1.0.0b1)'
required: true
description: "The release version number (i.e. 1.0.0b1)"
type: string
required: true
build_script_path:
description: "Build script path"
type: string
default: "scripts/build-dist.sh"
required: true
env_setup_script_path:
description: "Environment setup script path"
type: string
default: "scripts/env-setup.sh"
required: false
s3_bucket_name:
description: "AWS S3 bucket name"
type: string
default: "core-team-artifacts"
required: true
package_test_command:
description: "Package test command"
type: string
default: "dbt --version"
required: true
test_run:
description: "Test run (Publish release as draft)"
type: boolean
default: true
required: false
nightly_release:
description: "Nightly release to dev environment"
type: boolean
default: false
required: false
workflow_call:
inputs:
sha:
description: "The last commit sha in the release"
type: string
required: true
target_branch:
description: "The branch to release from"
type: string
required: true
version_number:
description: "The release version number (i.e. 1.0.0b1)"
type: string
required: true
build_script_path:
description: "Build script path"
type: string
default: "scripts/build-dist.sh"
required: true
env_setup_script_path:
description: "Environment setup script path"
type: string
default: "scripts/env-setup.sh"
required: false
s3_bucket_name:
description: "AWS S3 bucket name"
type: string
default: "core-team-artifacts"
required: true
package_test_command:
description: "Package test command"
type: string
default: "dbt --version"
required: true
test_run:
description: "Test run (Publish release as draft)"
type: boolean
default: true
required: false
nightly_release:
description: "Nightly release to dev environment"
type: boolean
default: false
required: false
permissions:
contents: write # this is the permission that allows creating a new release
defaults:
run:
shell: bash
jobs:
unit:
name: Unit test
log-inputs:
name: Log Inputs
runs-on: ubuntu-latest
env:
TOXENV: "unit"
steps:
- name: Check out the repository
uses: actions/checkout@v2
with:
persist-credentials: false
ref: ${{ github.event.inputs.sha }}
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
- name: "[DEBUG] Print Variables"
run: |
pip install --user --upgrade pip
pip install tox
pip --version
tox --version
echo The last commit sha in the release: ${{ inputs.sha }}
echo The branch to release from: ${{ inputs.target_branch }}
echo The release version number: ${{ inputs.version_number }}
echo Build script path: ${{ inputs.build_script_path }}
echo Environment setup script path: ${{ inputs.env_setup_script_path }}
echo AWS S3 bucket name: ${{ inputs.s3_bucket_name }}
echo Package test command: ${{ inputs.package_test_command }}
echo Test run: ${{ inputs.test_run }}
echo Nightly release: ${{ inputs.nightly_release }}
- name: Run tox
run: tox
bump-version-generate-changelog:
name: Bump package version, Generate changelog
build:
name: build packages
uses: dbt-labs/dbt-release/.github/workflows/release-prep.yml@main
with:
sha: ${{ inputs.sha }}
version_number: ${{ inputs.version_number }}
target_branch: ${{ inputs.target_branch }}
env_setup_script_path: ${{ inputs.env_setup_script_path }}
test_run: ${{ inputs.test_run }}
nightly_release: ${{ inputs.nightly_release }}
secrets:
FISHTOWN_BOT_PAT: ${{ secrets.FISHTOWN_BOT_PAT }}
log-outputs-bump-version-generate-changelog:
name: "[Log output] Bump package version, Generate changelog"
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog]
runs-on: ubuntu-latest
steps:
- name: Check out the repository
uses: actions/checkout@v2
with:
persist-credentials: false
ref: ${{ github.event.inputs.sha }}
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
- name: Print variables
run: |
pip install --user --upgrade pip
pip install --upgrade setuptools wheel twine check-wheel-contents
pip --version
echo Final SHA : ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
echo Changelog path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
- name: Build distributions
run: ./scripts/build-dist.sh
build-test-package:
name: Build, Test, Package
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog]
- name: Show distributions
run: ls -lh dist/
uses: dbt-labs/dbt-release/.github/workflows/build.yml@main
- name: Check distribution descriptions
run: |
twine check dist/*
with:
sha: ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
changelog_path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
build_script_path: ${{ inputs.build_script_path }}
s3_bucket_name: ${{ inputs.s3_bucket_name }}
package_test_command: ${{ inputs.package_test_command }}
test_run: ${{ inputs.test_run }}
nightly_release: ${{ inputs.nightly_release }}
- name: Check wheel contents
run: |
check-wheel-contents dist/*.whl --ignore W007,W008
- uses: actions/upload-artifact@v2
with:
name: dist
path: |
dist/
!dist/dbt-${{github.event.inputs.version_number}}.tar.gz
test-build:
name: verify packages
needs: [build, unit]
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
run: |
pip install --user --upgrade pip
pip install --upgrade wheel
pip --version
- uses: actions/download-artifact@v2
with:
name: dist
path: dist/
- name: Show distributions
run: ls -lh dist/
- name: Install wheel distributions
run: |
find ./dist/*.whl -maxdepth 1 -type f | xargs pip install --force-reinstall --find-links=dist/
- name: Check wheel distributions
run: |
dbt --version
- name: Install source distributions
run: |
find ./dist/*.gz -maxdepth 1 -type f | xargs pip install --force-reinstall --find-links=dist/
- name: Check source distributions
run: |
dbt --version
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
github-release:
name: GitHub Release
if: ${{ !failure() && !cancelled() }}
needs: test-build
needs: [bump-version-generate-changelog, build-test-package]
runs-on: ubuntu-latest
uses: dbt-labs/dbt-release/.github/workflows/github-release.yml@main
steps:
- uses: actions/download-artifact@v2
with:
name: dist
path: '.'
# Need to set an output variable because env variables can't be taken as input
# This is needed for the next step with releasing to GitHub
- name: Find release type
id: release_type
env:
IS_PRERELEASE: ${{ contains(github.event.inputs.version_number, 'rc') || contains(github.event.inputs.version_number, 'b') }}
run: |
echo ::set-output name=isPrerelease::$IS_PRERELEASE
- name: Creating GitHub Release
uses: softprops/action-gh-release@v1
with:
name: dbt-core v${{github.event.inputs.version_number}}
tag_name: v${{github.event.inputs.version_number}}
prerelease: ${{ steps.release_type.outputs.isPrerelease }}
target_commitish: ${{github.event.inputs.sha}}
body: |
[Release notes](https://github.com/dbt-labs/dbt-core/blob/main/CHANGELOG.md)
files: |
dbt_postgres-${{github.event.inputs.version_number}}-py3-none-any.whl
dbt_core-${{github.event.inputs.version_number}}-py3-none-any.whl
dbt-postgres-${{github.event.inputs.version_number}}.tar.gz
dbt-core-${{github.event.inputs.version_number}}.tar.gz
with:
sha: ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
changelog_path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
test_run: ${{ inputs.test_run }}
pypi-release:
name: Pypi release
name: PyPI Release
runs-on: ubuntu-latest
needs: [github-release]
needs: github-release
uses: dbt-labs/dbt-release/.github/workflows/pypi-release.yml@main
environment: PypiProd
steps:
- uses: actions/download-artifact@v2
with:
name: dist
path: 'dist'
with:
version_number: ${{ inputs.version_number }}
test_run: ${{ inputs.test_run }}
- name: Publish distribution to PyPI
uses: pypa/gh-action-pypi-publish@v1.4.2
with:
password: ${{ secrets.PYPI_API_TOKEN }}
secrets:
PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
TEST_PYPI_API_TOKEN: ${{ secrets.TEST_PYPI_API_TOKEN }}
slack-notification:
name: Slack Notification
if: ${{ failure() && (!inputs.test_run || inputs.nightly_release) }}
needs:
[
bump-version-generate-changelog,
build-test-package,
github-release,
pypi-release,
]
uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}

View File

@@ -21,6 +21,9 @@ on:
- "*.latest"
- "releases/*"
# no special access is needed
permissions: read-all
env:
LATEST_SCHEMA_PATH: ${{ github.workspace }}/new_schemas
SCHEMA_DIFF_ARTIFACT: ${{ github.workspace }}//schema_schanges.txt

View File

@@ -3,6 +3,10 @@ on:
schedule:
- cron: "30 1 * * *"
permissions:
issues: write
pull-requests: write
jobs:
stale:
runs-on: ubuntu-latest

View File

@@ -22,7 +22,7 @@ jobs:
# run the performance measurements on the current or default branch
test-schema:
name: Test Log Schema
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
env:
# turns warnings into errors
RUSTFLAGS: "-D warnings"

View File

@@ -20,6 +20,10 @@ on:
description: 'The version number to bump to (ex. 1.2.0, 1.3.0b1)'
required: true
permissions:
contents: write
pull-requests: write
jobs:
bump:
runs-on: ubuntu-latest
@@ -42,6 +46,16 @@ jobs:
source env/bin/activate
pip install --upgrade pip
- name: Add Homebrew to PATH
run: |
echo "/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin" >> $GITHUB_PATH
- name: Install Homebrew packages
run: |
brew install pre-commit
brew tap miniscruff/changie https://github.com/miniscruff/changie
brew install changie
- name: Audit Version and Parse Into Parts
id: semver
uses: dbt-labs/actions/parse-semver@v1
@@ -66,18 +80,8 @@ jobs:
env/bin/bumpversion --allow-dirty --new-version ${{ github.event.inputs.version_number }} major
git status
# this step will fail on whitespace errors but also correct them
- name: Format bumpversion file
continue-on-error: true
run: |
brew install pre-commit
pre-commit run trailing-whitespace --files .bumpversion.cfg
git status
- name: Run changie
run: |
brew tap miniscruff/changie https://github.com/miniscruff/changie
brew install changie
if [[ ${{ steps.semver.outputs.is-pre-release }} -eq 1 ]]
then
changie batch ${{ steps.semver.outputs.base-version }} --move-dir '${{ steps.semver.outputs.base-version }}' --prerelease '${{ steps.semver.outputs.pre-release }}'
@@ -87,6 +91,20 @@ jobs:
changie merge
git status
# this step will fail on whitespace errors but also correct them
- name: Remove trailing whitespace
continue-on-error: true
run: |
pre-commit run trailing-whitespace --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
# this step will fail on newline errors but also correct them
- name: Removing extra newlines
continue-on-error: true
run: |
pre-commit run end-of-file-fixer --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
- name: Commit version bump to branch
uses: EndBug/add-and-commit@v7
with:

1
.gitignore vendored
View File

@@ -95,6 +95,7 @@ venv/
# vscode
.vscode/
*.code-workspace
# poetry
pyproject.toml

View File

@@ -35,7 +35,7 @@ repos:
- "--target-version=py38"
- "--check"
- "--diff"
- repo: https://gitlab.com/pycqa/flake8
- repo: https://github.com/pycqa/flake8
rev: 4.0.1
hooks:
- id: flake8

View File

@@ -5,95 +5,182 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.3.0-b2 - August 29, 2022
## dbt-core 1.3.4 - April 19, 2023
### Fixes
- Improved failed event serialization handling and associated tests ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))
### Contributors
- [@QMalcolm](https://github.com/QMalcolm) ([#7113](https://github.com/dbt-labs/dbt-core/issues/7113), [#7108](https://github.com/dbt-labs/dbt-core/issues/7108), [#6568](https://github.com/dbt-labs/dbt-core/issues/6568))
## dbt-core 1.3.3 - February 28, 2023
### Fixes
- add pytz dependency ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
### Contributors
- [@sdebruyn](https://github.com/sdebruyn) ([#7077](https://github.com/dbt-labs/dbt-core/issues/7077))
## dbt-core 1.3.2 - January 04, 2023
### Fixes
- Bug when partial parsing with an empty schema file ([#4850](https://github.com/dbt-labs/dbt-core/issues/4850), [#<no value>](https://github.com/dbt-labs/dbt-core/pull/<no value>))
## dbt-core 1.3.1 - November 16, 2022
### Features
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201), [#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
### Docs
- ([dbt-docs/#5880](https://github.com/dbt-labs/dbt-docs/issues/5880), [dbt-docs/#324](https://github.com/dbt-labs/dbt-docs/pull/324))
- Fix rendering of sample code for metrics ([dbt-docs/#323](https://github.com/dbt-labs/dbt-docs/issues/323), [dbt-docs/#346](https://github.com/dbt-labs/dbt-docs/pull/346))
### Contributors
- [@paulbenschmidt](https://github.com/paulbenschmidt) ([dbt-docs/#324](https://github.com/dbt-labs/dbt-docs/pull/324))
- [@racheldaniel](https://github.com/racheldaniel) ([#6202](https://github.com/dbt-labs/dbt-core/pull/6202))
## dbt-core 1.3.0 - October 12, 2022
### Breaking Changes
- Renaming Metric Spec Attributes ([#5774](https://github.com/dbt-labs/dbt-core/issues/5774), [#5775](https://github.com/dbt-labs/dbt-core/pull/5775))
### Features
- Add `--defer` flag to dbt compile & dbt docs generate ([#4110](https://github.com/dbt-labs/dbt-core/issues/4110), [#4514](https://github.com/dbt-labs/dbt-core/pull/4514))
- Python model inital version ([#5261](https://github.com/dbt-labs/dbt-core/issues/5261), [#5421](https://github.com/dbt-labs/dbt-core/pull/5421))
- allows user to include the file extension for .py models in the dbt run -m command. ([#5289](https://github.com/dbt-labs/dbt-core/issues/5289), [#5295](https://github.com/dbt-labs/dbt-core/pull/5295))
- Incremental materialization refactor and cleanup ([#5245](https://github.com/dbt-labs/dbt-core/issues/5245), [#5359](https://github.com/dbt-labs/dbt-core/pull/5359))
- Python models can support incremental logic ([#0](https://github.com/dbt-labs/dbt-core/issues/0), [#35](https://github.com/dbt-labs/dbt-core/pull/35))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
- merge_exclude_columns for incremental materialization ([#5260](https://github.com/dbt-labs/dbt-core/issues/5260), [#5457](https://github.com/dbt-labs/dbt-core/pull/5457))
- add exponential backoff to connection retries on Postgres (and Redshift) ([#5502](https://github.com/dbt-labs/dbt-core/issues/5502), [#5503](https://github.com/dbt-labs/dbt-core/pull/5503))
- use MethodName.File when value ends with .csv ([#5578](https://github.com/dbt-labs/dbt-core/issues/5578), [#5581](https://github.com/dbt-labs/dbt-core/pull/5581))
- Make `docs` configurable in `dbt_project.yml` and add a `node_color` attribute to change the color of nodes in the DAG ([#5333](https://github.com/dbt-labs/dbt-core/issues/5333), [#5397](https://github.com/dbt-labs/dbt-core/pull/5397))
- Adding ResolvedMetricReference helper functions and tests ([#5567](https://github.com/dbt-labs/dbt-core/issues/5567), [#5607](https://github.com/dbt-labs/dbt-core/pull/5607))
- Check dbt-core version requirements when installing Hub packages ([#5648](https://github.com/dbt-labs/dbt-core/issues/5648), [#5651](https://github.com/dbt-labs/dbt-core/pull/5651))
- Search current working directory for `profiles.yml` ([#5411](https://github.com/dbt-labs/dbt-core/issues/5411), [#5717](https://github.com/dbt-labs/dbt-core/pull/5717))
- Adding the `window` parameter to the metric spec. ([#5721](https://github.com/dbt-labs/dbt-core/issues/5721), [#5722](https://github.com/dbt-labs/dbt-core/pull/5722))
- Add invocation args dict to ProviderContext class ([#5524](https://github.com/dbt-labs/dbt-core/issues/5524), [#5782](https://github.com/dbt-labs/dbt-core/pull/5782))
- Adds new cli framework ([#5526](https://github.com/dbt-labs/dbt-core/issues/5526), [#5647](https://github.com/dbt-labs/dbt-core/pull/5647))
- Flags work with new Click CLI ([#5529](https://github.com/dbt-labs/dbt-core/issues/5529), [#5790](https://github.com/dbt-labs/dbt-core/pull/5790))
- Add metadata env method to ProviderContext class ([#5522](https://github.com/dbt-labs/dbt-core/issues/5522), [#5794](https://github.com/dbt-labs/dbt-core/pull/5794))
- Array macros ([#5520](https://github.com/dbt-labs/dbt-core/issues/5520), [#5823](https://github.com/dbt-labs/dbt-core/pull/5823))
- Add enabled config to exposures and metrics ([#5422](https://github.com/dbt-labs/dbt-core/issues/5422), [#5815](https://github.com/dbt-labs/dbt-core/pull/5815))
- Migrate dbt-utils current_timestamp macros into core + adapters ([#5521](https://github.com/dbt-labs/dbt-core/issues/5521), [#5838](https://github.com/dbt-labs/dbt-core/pull/5838))
- add -fr flag shorthand ([#5878](https://github.com/dbt-labs/dbt-core/issues/5878), [#5879](https://github.com/dbt-labs/dbt-core/pull/5879))
- add type_boolean as a data type macro ([#5739](https://github.com/dbt-labs/dbt-core/issues/5739), [#5875](https://github.com/dbt-labs/dbt-core/pull/5875))
- Support .dbtignore in project root to ignore certain files being read by dbt ([#5733](https://github.com/dbt-labs/dbt-core/issues/5733), [#5897](https://github.com/dbt-labs/dbt-core/pull/5897))
- This conditionally no-ops warehouse connection at compile depending on an env var, disabling introspection/queries during compilation only. This is a temporary solution to more complex permissions requirements for the semantic layer. ([#5936](https://github.com/dbt-labs/dbt-core/issues/5936), [#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
### Fixes
- Remove the default 256 characters limit on postgres character varying type when no limitation is set ([#5238](https://github.com/dbt-labs/dbt-core/issues/5238), [#5292](https://github.com/dbt-labs/dbt-core/pull/5292))
- Include schema file config in unrendered_config ([#5338](https://github.com/dbt-labs/dbt-core/issues/5338), [#5344](https://github.com/dbt-labs/dbt-core/pull/5344))
- Add context to compilation errors generated while rendering generic test configuration values. ([#5294](https://github.com/dbt-labs/dbt-core/issues/5294), [#5393](https://github.com/dbt-labs/dbt-core/pull/5393))
- Resolves #5351 - Do not consider shorter varchar cols as schema changes ([#5351](https://github.com/dbt-labs/dbt-core/issues/5351), [#5395](https://github.com/dbt-labs/dbt-core/pull/5395))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- on_shchma_change fail verbosity enhancement ([#5504](https://github.com/dbt-labs/dbt-core/issues/5504), [#5505](https://github.com/dbt-labs/dbt-core/pull/5505))
- Ignore empty strings passed in as secrets ([#5312](https://github.com/dbt-labs/dbt-core/issues/5312), [#5518](https://github.com/dbt-labs/dbt-core/pull/5518))
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
- Fix error rendering docs block in metrics description ([#5585](https://github.com/dbt-labs/dbt-core/issues/5585), [#5603](https://github.com/dbt-labs/dbt-core/pull/5603))
- Extended validations for the project names ([#5379](https://github.com/dbt-labs/dbt-core/issues/5379), [#5620](https://github.com/dbt-labs/dbt-core/pull/5620))
- Use sys.exit instead of exit ([#5621](https://github.com/dbt-labs/dbt-core/issues/5621), [#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- Finishing logic upgrade to Redshift for name truncation collisions. ([#5586](https://github.com/dbt-labs/dbt-core/issues/5586), [#5656](https://github.com/dbt-labs/dbt-core/pull/5656))
- multiple args for ref and source ([#5634](https://github.com/dbt-labs/dbt-core/issues/5634), [#5635](https://github.com/dbt-labs/dbt-core/pull/5635))
- Fix Unexpected behavior when chaining methods on dbt-ref'ed/sourced dataframes ([#5646](https://github.com/dbt-labs/dbt-core/issues/5646), [#5677](https://github.com/dbt-labs/dbt-core/pull/5677))
- Fix typos of comments in core/dbt/adapters/ ([#5690](https://github.com/dbt-labs/dbt-core/issues/5690), [#5693](https://github.com/dbt-labs/dbt-core/pull/5693))
- Include py.typed in MANIFEST.in. This enables packages that install dbt-core from pypi to use mypy. ([#5703](https://github.com/dbt-labs/dbt-core/issues/5703), [#5703](https://github.com/dbt-labs/dbt-core/pull/5703))
- Removal of all .coverage files when using make clean command ([#5633](https://github.com/dbt-labs/dbt-core/issues/5633), [#5759](https://github.com/dbt-labs/dbt-core/pull/5759))
- Remove temp files generated by unit tests ([#5631](https://github.com/dbt-labs/dbt-core/issues/5631), [#5749](https://github.com/dbt-labs/dbt-core/pull/5749))
- Fix warnings as errors during tests ([#5424](https://github.com/dbt-labs/dbt-core/issues/5424), [#5800](https://github.com/dbt-labs/dbt-core/pull/5800))
- Prevent event_history from holding references ([#5848](https://github.com/dbt-labs/dbt-core/issues/5848), [#5858](https://github.com/dbt-labs/dbt-core/pull/5858))
- Account for disabled flags on models in schema files more completely ([#3992](https://github.com/dbt-labs/dbt-core/issues/3992), [#5868](https://github.com/dbt-labs/dbt-core/pull/5868))
- ConfigSelectorMethod should check for bools ([#5890](https://github.com/dbt-labs/dbt-core/issues/5890), [#5889](https://github.com/dbt-labs/dbt-core/pull/5889))
- shorthand for full refresh should be one character ([#5878](https://github.com/dbt-labs/dbt-core/issues/5878), [#5908](https://github.com/dbt-labs/dbt-core/pull/5908))
- Fix macro resolution order during static analysis for custom generic tests ([#5720](https://github.com/dbt-labs/dbt-core/issues/5720), [#5907](https://github.com/dbt-labs/dbt-core/pull/5907))
- Fix race condition when invoking dbt via lib.py concurrently ([#5919](https://github.com/dbt-labs/dbt-core/issues/5919), [#5921](https://github.com/dbt-labs/dbt-core/pull/5921))
- check length of args of python model function before accessing it ([#6041](https://github.com/dbt-labs/dbt-core/issues/6041), [#6042](https://github.com/dbt-labs/dbt-core/pull/6042))
### Docs
- Update dependency inline-source from ^6.1.5 to ^7.2.0 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Update dependency jest from ^26.2.2 to ^28.1.3 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Update dependency underscore from ^1.9.0 to ^1.13.4 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Update dependency webpack-cli from ^3.3.12 to ^4.7.0 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Update dependency webpack-dev-server from ^3.1.11 to ^4.9.3 ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#291](https://github.com/dbt-labs/dbt-docs/pull/291))
- Searches no longer require perfect matches, and instead consider each word individually. `my model` or `model my` will now find `my_model`, without the need for underscores ([dbt-docs/#143](https://github.com/dbt-labs/dbt-docs/issues/143), [dbt-docs/#145](https://github.com/dbt-labs/dbt-docs/pull/145))
- Support the renaming of SQL to code happening in dbt-core ([dbt-docs/#299](https://github.com/dbt-labs/dbt-docs/issues/299), [dbt-docs/#292](https://github.com/dbt-labs/dbt-docs/pull/292))
- Leverages `docs.node_color` from `dbt-core` to color nodes in the DAG ([dbt-docs/#44](https://github.com/dbt-labs/dbt-docs/issues/44), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- Refer to exposures by their label by default. ([dbt-docs/#306](https://github.com/dbt-labs/dbt-docs/issues/306), [dbt-docs/#307](https://github.com/dbt-labs/dbt-docs/pull/307))
### Under the Hood
- Added language to tracked fields in run_model event ([#5571](https://github.com/dbt-labs/dbt-core/issues/5571), [#5469](https://github.com/dbt-labs/dbt-core/pull/5469))
- Update mashumaro to 3.0.3 ([#4940](https://github.com/dbt-labs/dbt-core/issues/4940), [#5118](https://github.com/dbt-labs/dbt-core/pull/5118))
- Add python incremental materialization test ([#0000](https://github.com/dbt-labs/dbt-core/issues/0000), [#5571](https://github.com/dbt-labs/dbt-core/pull/5571))
- Save use of default env vars to manifest to enable partial parsing in those cases. ([#5155](https://github.com/dbt-labs/dbt-core/issues/5155), [#5589](https://github.com/dbt-labs/dbt-core/pull/5589))
- add more information to log line interop test failures ([#5658](https://github.com/dbt-labs/dbt-core/issues/5658), [#5659](https://github.com/dbt-labs/dbt-core/pull/5659))
- Add supported languages to materializations ([#5569](https://github.com/dbt-labs/dbt-core/issues/5569), [#5695](https://github.com/dbt-labs/dbt-core/pull/5695))
- Migrate integration test 014 but also fix the snapshot hard delete test's timezone logic and force all integration tests to run flags.set_from_args to force environment variables are accessible to all integration test threads. ([#5760](https://github.com/dbt-labs/dbt-core/issues/5760), [#5760](https://github.com/dbt-labs/dbt-core/pull/5760))
- Support dbt-metrics compilation by rebuilding flat_graph ([#5525](https://github.com/dbt-labs/dbt-core/issues/5525), [#5786](https://github.com/dbt-labs/dbt-core/pull/5786))
- Reworking the way we define the window attribute of metrics to match freshness tests ([#5722](https://github.com/dbt-labs/dbt-core/issues/5722), [#5793](https://github.com/dbt-labs/dbt-core/pull/5793))
- Add PythonJobHelper base class in core and add more type checking ([#5802](https://github.com/dbt-labs/dbt-core/issues/5802), [#5802](https://github.com/dbt-labs/dbt-core/pull/5802))
- The link did not go to the anchor directly, now it does ([#5813](https://github.com/dbt-labs/dbt-core/issues/5813), [#5814](https://github.com/dbt-labs/dbt-core/pull/5814))
- remove key as reserved keyword from test_bool_or ([#5817](https://github.com/dbt-labs/dbt-core/issues/5817), [#5818](https://github.com/dbt-labs/dbt-core/pull/5818))
- Convert default selector tests to pytest ([#5728](https://github.com/dbt-labs/dbt-core/issues/5728), [#5820](https://github.com/dbt-labs/dbt-core/pull/5820))
- Compatibiltiy for metric attribute renaming ([#5807](https://github.com/dbt-labs/dbt-core/issues/5807), [#5825](https://github.com/dbt-labs/dbt-core/pull/5825))
- remove source quoting setting in adapter tests ([#5836](https://github.com/dbt-labs/dbt-core/issues/5836), [#5839](https://github.com/dbt-labs/dbt-core/pull/5839))
- Add name validation for metrics ([#5456](https://github.com/dbt-labs/dbt-core/issues/5456), [#5841](https://github.com/dbt-labs/dbt-core/pull/5841))
- Validate exposure name and add label ([#5606](https://github.com/dbt-labs/dbt-core/issues/5606), [#5844](https://github.com/dbt-labs/dbt-core/pull/5844))
- Adding validation for metric expression attribute ([#5871](https://github.com/dbt-labs/dbt-core/issues/5871), [#5873](https://github.com/dbt-labs/dbt-core/pull/5873))
- Profiling and Adapter Management work with Click CLI ([#5531](https://github.com/dbt-labs/dbt-core/issues/5531), [#5892](https://github.com/dbt-labs/dbt-core/pull/5892))
- Reparse references to deleted metric ([#5444](https://github.com/dbt-labs/dbt-core/issues/5444), [#5920](https://github.com/dbt-labs/dbt-core/pull/5920))
### Dependencies
- Upgrade to Jinja2==3.1.2 from Jinja2==2.11.3 ([#4748](https://github.com/dbt-labs/dbt-core/issues/4748), [#5465](https://github.com/dbt-labs/dbt-core/pull/5465))
- Bump mypy from 0.961 to 0.971 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5495](https://github.com/dbt-labs/dbt-core/pull/5495))
- Remove pin for MarkUpSafe from >=0.23,<2.1 ([#5506](https://github.com/dbt-labs/dbt-core/issues/5506), [#5507](https://github.com/dbt-labs/dbt-core/pull/5507))
### Dependency
- Bump python from 3.10.5-slim-bullseye to 3.10.6-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5623](https://github.com/dbt-labs/dbt-core/pull/5623))
- Bump mashumaro[msgpack] from 3.0.3 to 3.0.4 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5649](https://github.com/dbt-labs/dbt-core/pull/5649))
- Bump black from 22.6.0 to 22.8.0 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5750](https://github.com/dbt-labs/dbt-core/pull/5750))
- Bump python from 3.10.6-slim-bullseye to 3.10.7-slim-bullseye in /docker ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5805](https://github.com/dbt-labs/dbt-core/pull/5805))
### Contributors
- [@Goodkat](https://github.com/Goodkat) ([#5581](https://github.com/dbt-labs/dbt-core/pull/5581), [#5620](https://github.com/dbt-labs/dbt-core/pull/5620))
- [@Goodkat](https://github.com/Goodkat) ([#5581](https://github.com/dbt-labs/dbt-core/pull/5581), [#5518](https://github.com/dbt-labs/dbt-core/pull/5518), [#5620](https://github.com/dbt-labs/dbt-core/pull/5620))
- [@Ilanbenb](https://github.com/Ilanbenb) ([#5505](https://github.com/dbt-labs/dbt-core/pull/5505))
- [@b-per](https://github.com/b-per) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5607](https://github.com/dbt-labs/dbt-core/pull/5607))
- [@bbroeksema](https://github.com/bbroeksema) ([#5749](https://github.com/dbt-labs/dbt-core/pull/5749))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#5775](https://github.com/dbt-labs/dbt-core/pull/5775), [#5607](https://github.com/dbt-labs/dbt-core/pull/5607), [#5722](https://github.com/dbt-labs/dbt-core/pull/5722), [#5793](https://github.com/dbt-labs/dbt-core/pull/5793), [#5825](https://github.com/dbt-labs/dbt-core/pull/5825), [#5873](https://github.com/dbt-labs/dbt-core/pull/5873))
- [@danielcmessias](https://github.com/danielcmessias) ([#5889](https://github.com/dbt-labs/dbt-core/pull/5889))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#5457](https://github.com/dbt-labs/dbt-core/pull/5457), [#5879](https://github.com/dbt-labs/dbt-core/pull/5879), [#5908](https://github.com/dbt-labs/dbt-core/pull/5908))
- [@dbeatty10](https://github.com/dbeatty10) ([#5717](https://github.com/dbt-labs/dbt-core/pull/5717), [#5823](https://github.com/dbt-labs/dbt-core/pull/5823))
- [@drewbanin](https://github.com/drewbanin) ([#5921](https://github.com/dbt-labs/dbt-core/pull/5921), [dbt-docs/#292](https://github.com/dbt-labs/dbt-docs/pull/292))
- [@epapineau](https://github.com/epapineau) ([#5395](https://github.com/dbt-labs/dbt-core/pull/5395))
- [@graciegoheen](https://github.com/graciegoheen) ([#5823](https://github.com/dbt-labs/dbt-core/pull/5823))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5782](https://github.com/dbt-labs/dbt-core/pull/5782), [#5794](https://github.com/dbt-labs/dbt-core/pull/5794), [#5759](https://github.com/dbt-labs/dbt-core/pull/5759))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@joellabes](https://github.com/joellabes) ([dbt-docs/#145](https://github.com/dbt-labs/dbt-docs/pull/145))
- [@jpmmcneill](https://github.com/jpmmcneill) ([#5875](https://github.com/dbt-labs/dbt-core/pull/5875))
- [@kadero](https://github.com/kadero) ([#4514](https://github.com/dbt-labs/dbt-core/pull/4514))
- [@leoebfolsom](https://github.com/leoebfolsom) ([#5295](https://github.com/dbt-labs/dbt-core/pull/5295))
- [@matt-winkler](https://github.com/matt-winkler) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@nicholasyager](https://github.com/nicholasyager) ([#5393](https://github.com/dbt-labs/dbt-core/pull/5393))
- [@panasenco](https://github.com/panasenco) ([#5703](https://github.com/dbt-labs/dbt-core/pull/5703))
- [@racheldaniel](https://github.com/racheldaniel) ([#5926](https://github.com/dbt-labs/dbt-core/pull/5926))
- [@sdebruyn](https://github.com/sdebruyn) ([#5814](https://github.com/dbt-labs/dbt-core/pull/5814), [#5818](https://github.com/dbt-labs/dbt-core/pull/5818), [#5839](https://github.com/dbt-labs/dbt-core/pull/5839))
- [@shrodingers](https://github.com/shrodingers) ([#5292](https://github.com/dbt-labs/dbt-core/pull/5292))
- [@sungchun12](https://github.com/sungchun12) ([#5397](https://github.com/dbt-labs/dbt-core/pull/5397), [dbt-docs/#281](https://github.com/dbt-labs/dbt-docs/pull/281))
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
## dbt-core 1.3.0-b1 - July 29, 2022
### Features
- Python model inital version ([#5261](https://github.com/dbt-labs/dbt-core/issues/5261), [#5421](https://github.com/dbt-labs/dbt-core/pull/5421))
- allows user to include the file extension for .py models in the dbt run -m command. ([#5289](https://github.com/dbt-labs/dbt-core/issues/5289), [#5295](https://github.com/dbt-labs/dbt-core/pull/5295))
- Incremental materialization refactor and cleanup ([#5245](https://github.com/dbt-labs/dbt-core/issues/5245), [#5359](https://github.com/dbt-labs/dbt-core/pull/5359))
- Python models can support incremental logic ([#0](https://github.com/dbt-labs/dbt-core/issues/0), [#35](https://github.com/dbt-labs/dbt-core/pull/35))
- Add reusable function for retrying adapter connections. Utilize said function to add retries for Postgres (and Redshift). ([#5022](https://github.com/dbt-labs/dbt-core/issues/5022), [#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
- add exponential backoff to connection retries on Postgres (and Redshift) ([#5502](https://github.com/dbt-labs/dbt-core/issues/5502), [#5503](https://github.com/dbt-labs/dbt-core/pull/5503))
### Fixes
- Add context to compilation errors generated while rendering generic test configuration values. ([#5294](https://github.com/dbt-labs/dbt-core/issues/5294), [#5393](https://github.com/dbt-labs/dbt-core/pull/5393))
- Rename try to strict for more intuitiveness ([#5475](https://github.com/dbt-labs/dbt-core/issues/5475), [#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- Ignore empty strings passed in as secrets ([#5312](https://github.com/dbt-labs/dbt-core/issues/5312), [#5518](https://github.com/dbt-labs/dbt-core/pull/5518))
- Fix handling of top-level exceptions ([#5564](https://github.com/dbt-labs/dbt-core/issues/5564), [#5560](https://github.com/dbt-labs/dbt-core/pull/5560))
### Docs
- Update dependency inline-source from ^6.1.5 to ^7.2.0 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Update dependency jest from ^26.2.2 to ^28.1.3 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Update dependency underscore from ^1.9.0 to ^1.13.4 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Update dependency webpack-cli from ^3.3.12 to ^4.7.0 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Update dependency webpack-dev-server from ^3.1.11 to ^4.9.3 ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Searches no longer require perfect matches, and instead consider each word individually. `my model` or `model my` will now find `my_model`, without the need for underscores ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- Support the renaming of SQL to code happening in dbt-core ([#5574](https://github.com/dbt-labs/dbt-core/issues/5574), [#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
### Under the Hood
- Added language to tracked fields in run_model event ([#5571](https://github.com/dbt-labs/dbt-core/issues/5571), [#5469](https://github.com/dbt-labs/dbt-core/pull/5469))
- Update mashumaro to 3.0.3 ([#4940](https://github.com/dbt-labs/dbt-core/issues/4940), [#5118](https://github.com/dbt-labs/dbt-core/pull/5118))
- Add python incremental materialization test ([#0000](https://github.com/dbt-labs/dbt-core/issues/0000), [#5571](https://github.com/dbt-labs/dbt-core/pull/5571))
### Dependencies
- Upgrade to Jinja2==3.1.2 from Jinja2==2.11.3 ([#4748](https://github.com/dbt-labs/dbt-core/issues/4748), [#5465](https://github.com/dbt-labs/dbt-core/pull/5465))
- Bump mypy from 0.961 to 0.971 ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904), [#5495](https://github.com/dbt-labs/dbt-core/pull/5495))
- Remove pin for MarkUpSafe from >=0.23,<2.1 ([#5506](https://github.com/dbt-labs/dbt-core/issues/5506), [#5507](https://github.com/dbt-labs/dbt-core/pull/5507))
### Contributors
- [@Goodkat](https://github.com/Goodkat) ([#5518](https://github.com/dbt-labs/dbt-core/pull/5518))
- [@drewbanin](https://github.com/drewbanin) ([#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- [@jeremyyeo](https://github.com/jeremyyeo) ([#5477](https://github.com/dbt-labs/dbt-core/pull/5477))
- [@joellabes](https://github.com/joellabes) ([#5577](https://github.com/dbt-labs/dbt-core/pull/5577))
- [@leoebfolsom](https://github.com/leoebfolsom) ([#5295](https://github.com/dbt-labs/dbt-core/pull/5295))
- [@nicholasyager](https://github.com/nicholasyager) ([#5393](https://github.com/dbt-labs/dbt-core/pull/5393))
- [@tomasfarias](https://github.com/tomasfarias) ([#5432](https://github.com/dbt-labs/dbt-core/pull/5432))
- [@varun-dc](https://github.com/varun-dc) ([#5627](https://github.com/dbt-labs/dbt-core/pull/5627))
- [@yoiki](https://github.com/yoiki) ([#5693](https://github.com/dbt-labs/dbt-core/pull/5693))
- [@chamini2](https://github.com/chamini2) ([#6042](https://github.com/dbt-labs/dbt-core/pull/6042))
## Previous Releases

View File

@@ -201,13 +201,21 @@ Here are some general rules for adding tests:
* Sometimes flake8 complains about lines that are actually fine, in which case you can put a comment on the line such as: # noqa or # noqa: ANNN, where ANNN is the error code that flake8 issues.
* To collect output for `CProfile`, run dbt with the `-r` option and the name of an output file, i.e. `dbt -r dbt.cprof run`. If you just want to profile parsing, you can do: `dbt -r dbt.cprof parse`. `pip` install `snakeviz` to view the output. Run `snakeviz dbt.cprof` and output will be rendered in a browser window.
## Adding a CHANGELOG Entry
## Adding or modifying a CHANGELOG Entry
We use [changie](https://changie.dev) to generate `CHANGELOG` entries. **Note:** Do not edit the `CHANGELOG.md` directly. Your modifications will be lost.
Follow the steps to [install `changie`](https://changie.dev/guide/installation/) for your system.
Once changie is installed and your PR is created, simply run `changie new` and changie will walk you through the process of creating a changelog entry. Commit the file that's created and your changelog entry is complete!
Once changie is installed and your PR is created for a new feature, simply run the following command and changie will walk you through the process of creating a changelog entry:
```shell
changie new
```
Commit the file that's created and your changelog entry is complete!
If you are contributing to a feature already in progress, you will modify the changie yaml file in dbt/.changes/unreleased/ related to your change. If you need help finding this file, please ask within the discussion for the pull request!
You don't need to worry about which `dbt-core` version your change will go into. Just create the changelog entry with `changie`, and open your PR against the `main` branch. All merged changes will be included in the next minor version of `dbt-core`. The Core maintainers _may_ choose to "backport" specific changes in order to patch older minor versions. In that case, a maintainer will take care of that backport after merging your PR, before releasing the new version of `dbt-core`.

View File

@@ -12,6 +12,7 @@ class Column:
"TIMESTAMP": "TIMESTAMP",
"FLOAT": "FLOAT",
"INTEGER": "INT",
"BOOLEAN": "BOOLEAN",
}
column: str
dtype: str

View File

@@ -1220,7 +1220,6 @@ class BaseAdapter(metaclass=AdapterMeta):
def default_python_submission_method(self) -> str:
raise NotImplementedError("default_python_submission_method is not specified")
@available.parse_none
@log_code_execution
def submit_python_job(self, parsed_model: dict, compiled_code: str) -> AdapterResponse:
submission_method = parsed_model["config"].get(

View File

@@ -384,7 +384,7 @@ class RelationsCache:
relation = self.relations.pop(old_key)
new_key = new_relation.key()
# relaton has to rename its innards, so it needs the _CachedRelation.
# relation has to rename its innards, so it needs the _CachedRelation.
relation.rename(new_relation)
# update all the relations that refer to it
for cached in self.relations.values():

View File

@@ -1,23 +1,17 @@
import threading
from pathlib import Path
from contextlib import contextmanager
from importlib import import_module
from typing import Type, Dict, Any, List, Optional, Set
from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Type
from dbt.exceptions import RuntimeException, InternalException
from dbt.include.global_project import (
PACKAGE_PATH as GLOBAL_PROJECT_PATH,
PROJECT_NAME as GLOBAL_PROJECT_NAME,
)
from dbt.adapters.base.plugin import AdapterPlugin
from dbt.adapters.protocol import AdapterConfig, AdapterProtocol, RelationProtocol
from dbt.contracts.connection import AdapterRequiredConfig, Credentials
from dbt.events.functions import fire_event
from dbt.events.types import AdapterImportError, PluginLoadError
from dbt.contracts.connection import Credentials, AdapterRequiredConfig
from dbt.adapters.protocol import (
AdapterProtocol,
AdapterConfig,
RelationProtocol,
)
from dbt.adapters.base.plugin import AdapterPlugin
from dbt.exceptions import InternalException, RuntimeException
from dbt.include.global_project import PACKAGE_PATH as GLOBAL_PROJECT_PATH
from dbt.include.global_project import PROJECT_NAME as GLOBAL_PROJECT_NAME
Adapter = AdapterProtocol
@@ -64,7 +58,7 @@ class AdapterContainer:
# if we failed to import the target module in particular, inform
# the user about it via a runtime error
if exc.name == "dbt.adapters." + name:
fire_event(AdapterImportError(exc=exc))
fire_event(AdapterImportError(exc=str(exc)))
raise RuntimeException(f"Could not find adapter type {name}!")
# otherwise, the error had to have come from some underlying
# library. Log the stack trace.
@@ -217,3 +211,12 @@ def get_adapter_package_names(name: Optional[str]) -> List[str]:
def get_adapter_type_names(name: Optional[str]) -> List[str]:
return FACTORY.get_adapter_type_names(name)
@contextmanager
def adapter_management():
reset_adapters()
try:
yield
finally:
cleanup_connections()

View File

@@ -88,7 +88,7 @@ class AdapterProtocol( # type: ignore[misc]
],
):
# N.B. Technically these are ClassVars, but mypy doesn't support putting type vars in a
# ClassVar due to the restirctiveness of PEP-526
# ClassVar due to the restrictiveness of PEP-526
# See: https://github.com/python/mypy/issues/5144
AdapterSpecificConfigs: Type[AdapterConfig_T]
Column: Type[Column_T]

44
core/dbt/cli/flags.py Normal file
View File

@@ -0,0 +1,44 @@
# TODO Move this to /core/dbt/flags.py when we're ready to break things
import os
from dataclasses import dataclass
from multiprocessing import get_context
from pprint import pformat as pf
from click import get_current_context
if os.name != "nt":
# https://bugs.python.org/issue41567
import multiprocessing.popen_spawn_posix # type: ignore # noqa: F401
@dataclass(frozen=True)
class Flags:
def __init__(self, ctx=None) -> None:
if ctx is None:
ctx = get_current_context()
def assign_params(ctx):
"""Recursively adds all click params to flag object"""
for param_name, param_value in ctx.params.items():
# N.B. You have to use the base MRO method (object.__setattr__) to set attributes
# when using frozen dataclasses.
# https://docs.python.org/3/library/dataclasses.html#frozen-instances
if hasattr(self, param_name):
raise Exception(f"Duplicate flag names found in click command: {param_name}")
object.__setattr__(self, param_name.upper(), param_value)
if ctx.parent:
assign_params(ctx.parent)
assign_params(ctx)
# Hard coded flags
object.__setattr__(self, "WHICH", ctx.info_name)
object.__setattr__(self, "MP_CONTEXT", get_context("spawn"))
# Support console DO NOT TRACK initiave
if os.getenv("DO_NOT_TRACK", "").lower() in (1, "t", "true", "y", "yes"):
object.__setattr__(self, "ANONYMOUS_USAGE_STATS", False)
def __str__(self) -> str:
return str(pf(self.__dict__))

View File

@@ -1,44 +1,70 @@
import click
from dbt.cli import params as p
import sys
import inspect # This is temporary for RAT-ing
from copy import copy
from pprint import pformat as pf # This is temporary for RAT-ing
# This is temporary for RAT-ing
import inspect
from pprint import pformat as pf
import click
from dbt.adapters.factory import adapter_management
from dbt.cli import params as p
from dbt.cli.flags import Flags
from dbt.profiler import profiler
def cli_runner():
# Alias "list" to "ls"
ls = copy(cli.commands["list"])
ls.hidden = True
cli.add_command(ls, "ls")
# Run the cli
cli()
# dbt
@click.group(
context_settings={"help_option_names": ["-h", "--help"]},
invoke_without_command=True,
no_args_is_help=True,
epilog="Specify one of these sub-commands and you can find more help from there.",
)
@click.pass_context
@p.version
@p.anonymous_usage_stats
@p.cache_selected_only
@p.debug
@p.enable_legacy_logger
@p.event_buffer_size
@p.fail_fast
@p.log_cache_events
@p.log_format
@p.macro_debugging
@p.partial_parse
@p.print
@p.printer_width
@p.quiet
@p.send_anonymous_usage_stats
@p.record_timing_info
@p.static_parser
@p.use_colors
@p.use_experimental_parser
@p.version
@p.version_check
@p.warn_error
@p.write_json
@p.event_buffer_size
@p.record_timing
def cli(ctx, **kwargs):
"""An ELT tool for managing your SQL transformations and data models.
For more documentation on these commands, visit: docs.getdbt.com
"""
if kwargs.get("version", False):
incomplete_flags = Flags()
# Profiling
if incomplete_flags.RECORD_TIMING_INFO:
ctx.with_resource(profiler(enable=True, outfile=incomplete_flags.RECORD_TIMING_INFO))
# Adapter management
ctx.with_resource(adapter_management())
# Version info
if incomplete_flags.VERSION:
click.echo(f"`version` called\n ctx.params: {pf(ctx.params)}")
sys.exit()
return
else:
del ctx.params["version"]
@@ -46,26 +72,43 @@ def cli(ctx, **kwargs):
# dbt build
@cli.command("build")
@click.pass_context
@p.defer
@p.exclude
@p.fail_fast
@p.full_refresh
@p.indirect_selection
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.show
@p.state
@p.store_failures
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def build(ctx, **kwargs):
"""Run all Seeds, Models, Snapshots, and tests in DAG order"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt clean
@cli.command("clean")
@click.pass_context
@p.project_dir
@p.profiles_dir
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
def clean(ctx, **kwargs):
"""Delete all folders in the clean-targets list (usually the dbt_packages and target directories.)"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt docs
@@ -78,86 +121,82 @@ def docs(ctx, **kwargs):
# dbt docs generate
@docs.command("generate")
@click.pass_context
@p.version_check
@p.project_dir
@p.profiles_dir
@p.profile
@p.target
@p.vars
@p.compile_docs
@p.defer
@p.threads
@p.target_path
@p.exclude
@p.log_path
@p.models
@p.exclude
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.state
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def docs_generate(ctx, **kwargs):
"""Generate the documentation website for your project"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt docs serve
@docs.command("serve")
@click.pass_context
@p.project_dir
@p.profiles_dir
@p.browser
@p.port
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.port
@p.browser
def docs_serve(ctx, **kwargs):
"""Serve the documentation website for your project"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt compile
@cli.command("compile")
@click.pass_context
@p.version_check
@p.project_dir
@p.profiles_dir
@p.profile
@p.target
@p.vars
@p.parse_only
@p.threads
@p.target_path
@p.defer
@p.exclude
@p.full_refresh
@p.log_path
@p.models
@p.exclude
@p.parse_only
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.state
@p.defer
@p.full_refresh
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def compile(ctx, **kwargs):
"""Generates executable SQL from source, model, test, and analysis files. Compiled SQL files are written to the target/ directory."""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt debug
@cli.command("debug")
@click.pass_context
@p.version_check
@p.project_dir
@p.profiles_dir
@p.config_dir
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.config_dir
@p.version_check
def debug(ctx, **kwargs):
"""Show some helpful information about dbt for debugging. Not to be confused with the --debug option which increases verbosity."""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt deps
@@ -170,9 +209,8 @@ def debug(ctx, **kwargs):
@p.vars
def deps(ctx, **kwargs):
"""Pull the most recent version of the dependencies listed in packages.yml"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt init
@@ -181,147 +219,139 @@ def deps(ctx, **kwargs):
@p.profile
@p.profiles_dir
@p.project_dir
@p.skip_profile_setup
@p.target
@p.vars
@p.skip_profile_setup
def init(ctx, **kwargs):
"""Initialize a new DBT project."""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt list
# dbt TODO: Figure out aliasing for ls (or just c/p?)
@cli.command("list")
@click.pass_context
@p.exclude
@p.indirect_selection
@p.models
@p.output
@p.output_keys
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.output
@p.ouptut_keys
@p.resource_type
@p.models
@p.indirect_selection
@p.exclude
@p.selector
@p.state
@p.target
@p.vars
def list(ctx, **kwargs):
"""List the resources in your project"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt parse
@cli.command("parse")
@click.pass_context
@p.compile_parse
@p.log_path
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.write_manifest
@p.compile_parse
@p.threads
@p.target_path
@p.log_path
@p.threads
@p.vars
@p.version_check
@p.write_manifest
def parse(ctx, **kwargs):
"""Parses the project and provides information on performance"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt run
@cli.command("run")
@click.pass_context
@p.defer
@p.exclude
@p.fail_fast
@p.version_check
@p.full_refresh
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.log_path
@p.target_path
@p.threads
@p.models
@p.exclude
@p.selector
@p.state
@p.defer
@p.full_refresh
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def run(ctx, **kwargs):
"""Compile SQL and execute against the current target database."""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt run operation
@cli.command("run-operation")
@click.pass_context
@p.args
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.args
def run_operation(ctx, **kwargs):
"""Run the named macro with any supplied arguments."""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt seed
@cli.command("seed")
@click.pass_context
@p.version_check
@p.exclude
@p.full_refresh
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.selector
@p.show
@p.state
@p.target
@p.vars
@p.full_refresh
@p.log_path
@p.target_path
@p.threads
@p.models
@p.exclude
@p.selector
@p.state
@p.show
@p.vars
@p.version_check
def seed(ctx, **kwargs):
"""Load data from csv files into your data warehouse."""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt snapshot
@cli.command("snapshot")
@click.pass_context
@p.defer
@p.exclude
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.threads
@p.models
@p.exclude
@p.selector
@p.state
@p.defer
@p.target
@p.threads
@p.vars
def snapshot(ctx, **kwargs):
"""Execute snapshots defined in your project"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt source
@@ -334,51 +364,49 @@ def source(ctx, **kwargs):
# dbt source freshness
@source.command("freshness")
@click.pass_context
@p.exclude
@p.models
@p.output_path # TODO: Is this ok to re-use? We have three different output params, how much can we consolidate?
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.threads
@p.models
@p.exclude
@p.selector
@p.state
@p.output_path # TODO: Is this ok to re-use? We have three different output params, how much can we consolidate?
@p.target
@p.threads
@p.vars
def freshness(ctx, **kwargs):
"""Snapshots the current freshness of the project's sources"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# dbt test
@cli.command("test")
@click.pass_context
@p.defer
@p.exclude
@p.fail_fast
@p.version_check
@p.store_failures
@p.indirect_selection
@p.log_path
@p.models
@p.profile
@p.profiles_dir
@p.project_dir
@p.target
@p.vars
@p.indirect_selection
@p.log_path
@p.target_path
@p.threads
@p.models
@p.exclude
@p.selector
@p.state
@p.defer
@p.store_failures
@p.target
@p.target_path
@p.threads
@p.vars
@p.version_check
def test(ctx, **kwargs):
"""Runs tests on data in deployed models. Run this after `dbt run`"""
click.echo(
f"`{inspect.stack()[0][3]}` called\n kwargs: {kwargs}\n ctx: {pf(ctx.parent.params)}"
)
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
# Support running as a module
if __name__ == "__main__":
cli()
cli_runner()

View File

@@ -0,0 +1,33 @@
from click import ParamType
import yaml
class YAML(ParamType):
"""The Click YAML type. Converts YAML strings into objects."""
name = "YAML"
def convert(self, value, param, ctx):
# assume non-string values are a problem
if not isinstance(value, str):
self.fail(f"Cannot load YAML from type {type(value)}", param, ctx)
try:
return yaml.load(value, Loader=yaml.Loader)
except yaml.parser.ParserError:
self.fail(f"String '{value}' is not valid YAML", param, ctx)
class Truthy(ParamType):
"""The Click Truthy type. Converts strings into a "truthy" type"""
name = "TRUTHY"
def convert(self, value, param, ctx):
# assume non-string / non-None values are a problem
if not isinstance(value, (str, None)):
self.fail(f"Cannot load TRUTHY from type {type(value)}", param, ctx)
if value is None or value.lower() in ("0", "false", "f"):
return None
else:
return value

View File

@@ -1,56 +1,58 @@
import click
import yaml
from pathlib import Path, PurePath
from click import ParamType
import click
from dbt.cli.option_types import YAML
from dbt.cli.resolvers import default_project_dir, default_profiles_dir
class YAML(ParamType):
"""The Click YAML type. Converts YAML strings into objects."""
name = "YAML"
def convert(self, value, param, ctx):
# assume non-string values are a problem
if not isinstance(value, str):
self.fail(f"Cannot load YAML from type {type(value)}", param, ctx)
try:
return yaml.load(value, Loader=yaml.Loader)
except yaml.parser.ParserError:
self.fail(f"String '{value}' is not valid YAML", param, ctx)
# TODO: The name (reflected in flags) is a correction!
# The original name was `SEND_ANONYMOUS_USAGE_STATS` and used an env var called "DBT_SEND_ANONYMOUS_USAGE_STATS"
# Both of which break existing naming conventions (doesn't match param flag).
# This will need to be fixed before use in the main codebase and communicated as a change to the community!
anonymous_usage_stats = click.option(
"--anonymous-usage-stats/--no-anonymous-usage-stats",
envvar="DBT_ANONYMOUS_USAGE_STATS",
help="Send anonymous usage stats to dbt Labs.",
default=True,
)
args = click.option(
"--args",
envvar=None,
help="Supply arguments to the macro. This dictionary will be mapped to the keyword arguments defined in the selected macro. This argument should be a YAML string, eg. '{my_variable: my_value}'",
type=YAML(),
)
browser = click.option(
"--browser/--no-browser",
envvar=None,
help="Wether or not to open a local web browser after starting the server",
default=True,
)
cache_selected_only = click.option(
"--cache-selected-only/--no-cache-selected-only",
envvar="DBT_CACHE_SELECTED_ONLY",
help="Pre cache database objects relevant to selected resource only.",
default=False,
)
compile_docs = click.option(
"--compile/--no-compile",
envvar=None,
help="Wether or not to run 'dbt compile' as part of docs generation",
default=True,
)
compile_parse = click.option(
"--compile/--no-compile",
envvar=None,
help="TODO: No help text currently available",
default=True,
)
config_dir = click.option(
"--config-dir",
envvar=None,
help="If specified, DBT will show path information for this project",
type=click.STRING,
)
@@ -58,44 +60,68 @@ config_dir = click.option(
debug = click.option(
"--debug/--no-debug",
"-d/ ",
envvar="DBT_DEBUG",
help="Display debug logging during dbt execution. Useful for debugging and making bug reports.",
default=False,
)
# TODO: The env var and name (reflected in flags) are corrections!
# The original name was `DEFER_MODE` and used an env var called "DBT_DEFER_TO_STATE"
# Both of which break existing naming conventions.
# This will need to be fixed before use in the main codebase and communicated as a change to the community!
defer = click.option(
"--defer/--no-defer",
envvar="DBT_DEFER",
help="If set, defer to the state variable for resolving unselected nodes.",
default=True,
)
enable_legacy_logger = click.option(
"--enable-legacy-logger/--no-enable-legacy-logger",
envvar="DBT_ENABLE_LEGACY_LOGGER",
hidden=True,
)
event_buffer_size = click.option(
"--event-buffer-size",
envvar="DBT_EVENT_BUFFER_SIZE",
help="Sets the max number of events to buffer in EVENT_HISTORY.",
default=100000,
type=click.INT,
)
exclude = click.option("--exclude", help="Specify the nodes to exclude.")
exclude = click.option("--exclude", envvar=None, help="Specify the nodes to exclude.")
fail_fast = click.option(
"--fail-fast/--no-fail-fast", "-x/ ", help="Stop execution on first failure.", default=False
"--fail-fast/--no-fail-fast",
"-x/ ",
envvar="DBT_FAIL_FAST",
help="Stop execution on first failure.",
)
full_refresh = click.option(
"--full-refresh",
"-f",
envvar="DBT_FULL_REFRESH",
help="If specified, dbt will drop incremental models and fully-recalculate the incremental table from the model definition.",
is_flag=True,
)
indirect_selection = click.option(
"--indirect_selection",
"--indirect-selection",
envvar="DBT_INDIRECT_SELECTION",
help="Select all tests that are adjacent to selected resources, even if they those resources have been explicitly selected.",
type=click.Choice(["eager", "cautious"], case_sensitive=False),
default="eager",
)
log_cache_events = click.option(
"--log-cache-events/--no-log-cache-events",
help="Enable verbose adapter cache logging.",
envvar="DBT_LOG_CACHE_EVENTS",
)
log_format = click.option(
"--log-format",
envvar="DBT_LOG_FORMAT",
help="Specify the log format, overriding the command's default.",
type=click.Choice(["text", "json", "default"], case_sensitive=False),
default="default",
@@ -103,28 +129,42 @@ log_format = click.option(
log_path = click.option(
"--log-path",
envvar="DBT_LOG_PATH",
help="Configure the 'log-path'. Only applies this setting for the current run. Overrides the 'DBT_LOG_PATH' if it is set.",
type=click.Path(),
)
models = click.option("-m", "-s", help="Specify the nodes to include.", multiple=True)
macro_debugging = click.option(
"--macro-debugging/--no-macro-debugging",
envvar="DBT_MACRO_DEBUGGING",
hidden=True,
)
models = click.option(
"-m",
"-s",
"models",
envvar=None,
help="Specify the nodes to include.",
multiple=True,
)
output = click.option(
"--output",
envvar=None,
help="TODO: No current help text",
type=click.Choice(["json", "name", "path", "selector"], case_sensitive=False),
default="name",
)
ouptut_keys = click.option(
"--output-keys",
help="TODO: No current help text",
default=False,
output_keys = click.option(
"--output-keys", envvar=None, help="TODO: No current help text", type=click.STRING
)
output_path = click.option(
"--output",
"-o",
envvar=None,
help="Specify the output path for the json report. By default, outputs to 'target/sources.json'",
type=click.Path(file_okay=True, dir_okay=False, writable=True),
default=PurePath.joinpath(Path.cwd(), "target/sources.json"),
@@ -132,65 +172,84 @@ output_path = click.option(
parse_only = click.option(
"--parse-only",
envvar=None,
help="TODO: No help text currently available",
is_flag=True,
)
partial_parse = click.option(
"--partial-parse/--no-partial-parse",
envvar="DBT_PARTIAL_PARSE",
help="Allow for partial parsing by looking for and writing to a pickle file in the target directory. This overrides the user configuration file.",
default=True,
)
port = click.option(
"--port", help="Specify the port number for the docs server", default=8080, type=click.INT
"--port",
envvar=None,
help="Specify the port number for the docs server",
default=8080,
type=click.INT,
)
# TODO: The env var and name (reflected in flags) are corrections!
# The original name was `NO_PRINT` and used the env var `DBT_NO_PRINT`.
# Both of which break existing naming conventions.
# This will need to be fixed before use in the main codebase and communicated as a change to the community!
print = click.option(
"--print/--no-print", help="Output all {{ print() }} macro calls.", default=True
"--print/--no-print",
envvar="DBT_PRINT",
help="Output all {{ print() }} macro calls.",
default=True,
)
printer_width = click.option(
"--printer_width", help="Sets the width of terminal output", type=click.INT, default=80
"--printer-width",
envvar="DBT_PRINTER_WIDTH",
help="Sets the width of terminal output",
type=click.INT,
default=80,
)
profile = click.option(
"--profile",
envvar=None,
help="Which profile to load. Overrides setting in dbt_project.yml.",
)
profiles_dir = click.option(
"--profiles-dir",
help=f"Which directory to look in for the profiles.yml file. Default = {PurePath.joinpath(Path.home(), '.dbt')}",
default=PurePath.joinpath(Path.home(), ".dbt"),
type=click.Path(
exists=True,
),
envvar="DBT_PROFILES_DIR",
help="Which directory to look in for the profiles.yml file. If not set, dbt will look in the current working directory first, then HOME/.dbt/",
default=default_profiles_dir(),
type=click.Path(exists=True),
)
project_dir = click.option(
"--project-dir",
envvar=None,
help="Which directory to look in for the dbt_project.yml file. Default is the current working directory and its parents.",
default=Path.cwd(),
default=default_project_dir(),
type=click.Path(exists=True),
)
quiet = click.option(
"--quiet/--no-quiet",
envvar="DBT_QUIET",
help="Suppress all non-error logging to stdout. Does not affect {{ print() }} macro calls.",
default=False,
)
record_timing = click.option(
"-r",
record_timing_info = click.option(
"--record-timing-info",
"-r",
envvar=None,
help="When this option is passed, dbt will output low-level timing stats to the specified file. Example: `--record-timing-info output.profile`",
is_flag=True,
default=False,
type=click.Path(exists=False),
)
resource_type = click.option(
"--resource-type",
envvar=None,
help="TODO: No current help text",
type=click.Choice(
[
@@ -210,50 +269,63 @@ resource_type = click.option(
default="default",
)
selector = click.option("--selector", help="The selector name to use, as defined in selectors.yml")
send_anonymous_usage_stats = click.option(
"--anonymous-usage-stats/--no-anonymous-usage-stats",
help="Send anonymous usage stats to dbt Labs.",
default=True,
selector = click.option(
"--selector", envvar=None, help="The selector name to use, as defined in selectors.yml"
)
show = click.option(
"--show",
help="Show a sample of the loaded data in the terminal",
default=False,
"--show", envvar=None, help="Show a sample of the loaded data in the terminal", is_flag=True
)
skip_profile_setup = click.option(
"--skip-profile-setup",
"-s",
help="Skip interative profile setup.",
default=False,
"--skip-profile-setup", "-s", envvar=None, help="Skip interative profile setup.", is_flag=True
)
# TODO: The env var and name (reflected in flags) are corrections!
# The original name was `ARTIFACT_STATE_PATH` and used the env var `DBT_ARTIFACT_STATE_PATH`.
# Both of which break existing naming conventions.
# This will need to be fixed before use in the main codebase and communicated as a change to the community!
state = click.option(
"--state",
envvar="DBT_STATE",
help="If set, use the given directory as the source for json files to compare with this project.",
type=click.Path(
dir_okay=True,
exists=True,
file_okay=False,
readable=True,
resolve_path=True,
),
)
static_parser = click.option(
"--static-parser/--no-static-parser", help="Use the static parser.", default=True
"--static-parser/--no-static-parser",
envvar="DBT_STATIC_PARSER",
help="Use the static parser.",
default=True,
)
store_failures = click.option(
"--store-failures", help="Store test results (failing rows) in the database", default=False
"--store-failures",
envvar="DBT_STORE_FAILURES",
help="Store test results (failing rows) in the database",
is_flag=True,
)
target = click.option("-t", "--target", help="Which target to load for the given profile")
target = click.option(
"--target", "-t", envvar=None, help="Which target to load for the given profile"
)
target_path = click.option(
"--target-path",
envvar="DBT_TARGET_PATH",
help="Configure the 'target-path'. Only applies this setting for the current run. Overrides the 'DBT_TARGET_PATH' if it is set.",
type=click.Path(),
)
threads = click.option(
"--threads",
envvar=None,
help="Specify number of threads to use while executing models. Overrides settings in profiles.yml.",
default=1,
type=click.INT,
@@ -261,44 +333,54 @@ threads = click.option(
use_colors = click.option(
"--use-colors/--no-use-colors",
envvar="DBT_USE_COLORS",
help="Output is colorized by default and may also be set in a profile or at the command line.",
default=True,
)
use_experimental_parser = click.option(
"--use-experimental-parser/--no-use-experimental-parser",
envvar="DBT_USE_EXPERIMENTAL_PARSER",
help="Enable experimental parsing features.",
default=False,
)
vars = click.option(
"--vars",
envvar=None,
help="Supply variables to the project. This argument overrides variables defined in your dbt_project.yml file. This argument should be a YAML string, eg. '{my_variable: my_value}'",
type=YAML(),
)
version = click.option("--version", help="Show version information", is_flag=True, default=False)
version = click.option(
"--version",
envvar=None,
help="Show version information",
is_flag=True,
)
version_check = click.option(
"--version-check/--no-version-check",
envvar="DBT_VERSION_CHECK",
help="Ensure dbt's version matches the one specified in the dbt_project.yml file ('require-dbt-version')",
default=True,
)
warn_error = click.option(
"--warn-error/--no-warn-error",
envvar="DBT_WARN_ERROR",
help="If dbt would normally warn, instead raise an exception. Examples include --models that selects nothing, deprecations, configurations with no associated models, invalid test configurations, and missing sources/refs in tests.",
default=False,
)
write_json = click.option(
"--write-json/--no-write-json",
envvar="DBT_WRITE_JSON",
help="Writing the manifest and run_results.json files to disk",
default=True,
)
write_manifest = click.option(
"--write-manifest/--no-write-manifest",
envvar=None,
help="TODO: No help text currently available",
default=True,
)

11
core/dbt/cli/resolvers.py Normal file
View File

@@ -0,0 +1,11 @@
from pathlib import Path
def default_project_dir():
paths = list(Path.cwd().parents)
paths.insert(0, Path.cwd())
return next((x for x in paths if (x / "dbt_project.yml").exists()), Path.cwd())
def default_profiles_dir():
return Path.cwd() if (Path.cwd() / "profiles.yml").exists() else Path.home() / ".dbt"

View File

@@ -27,6 +27,7 @@ from dbt.utils import (
from dbt.clients._jinja_blocks import BlockIterator, BlockData, BlockTag
from dbt.contracts.graph.compiled import CompiledGenericTestNode
from dbt.contracts.graph.parsed import ParsedGenericTestNode
from dbt.exceptions import (
InternalException,
raise_compiler_error,
@@ -305,13 +306,13 @@ class MacroGenerator(BaseMacroGenerator):
@contextmanager
def track_call(self):
# This is only called from __call__
if self.stack is None or self.node is None:
if self.stack is None:
yield
else:
unique_id = self.macro.unique_id
depth = self.stack.depth
# only mark depth=0 as a dependency
if depth == 0:
# only mark depth=0 as a dependency, when creating this dependency we don't pass in stack
if depth == 0 and self.node:
self.node.depends_on.add_macro(unique_id)
self.stack.push(unique_id)
try:

View File

@@ -15,7 +15,7 @@ def statically_extract_macro_calls(string, ctx, db_wrapper=None):
if hasattr(func_call, "node") and hasattr(func_call.node, "name"):
func_name = func_call.node.name
else:
# func_call for dbt_utils.current_timestamp macro
# func_call for dbt.current_timestamp macro
# Call(
# node=Getattr(
# node=Name(

View File

@@ -12,6 +12,7 @@ import tarfile
import requests
import stat
from typing import Type, NoReturn, List, Optional, Dict, Any, Tuple, Callable, Union
from pathspec import PathSpec # type: ignore
from dbt.events.functions import fire_event
from dbt.events.types import (
@@ -36,6 +37,7 @@ def find_matching(
root_path: str,
relative_paths_to_search: List[str],
file_pattern: str,
ignore_spec: Optional[PathSpec] = None,
) -> List[Dict[str, Any]]:
"""
Given an absolute `root_path`, a list of relative paths to that
@@ -57,19 +59,30 @@ def find_matching(
reobj = re.compile(regex, re.IGNORECASE)
for relative_path_to_search in relative_paths_to_search:
# potential speedup for ignore_spec
# if ignore_spec.matches(relative_path_to_search):
# continue
absolute_path_to_search = os.path.join(root_path, relative_path_to_search)
walk_results = os.walk(absolute_path_to_search)
for current_path, subdirectories, local_files in walk_results:
# potential speedup for ignore_spec
# relative_dir = os.path.relpath(current_path, root_path) + os.sep
# if ignore_spec.match(relative_dir):
# continue
for local_file in local_files:
absolute_path = os.path.join(current_path, local_file)
relative_path = os.path.relpath(absolute_path, absolute_path_to_search)
relative_path_to_root = os.path.join(relative_path_to_search, relative_path)
modification_time = 0.0
try:
modification_time = os.path.getmtime(absolute_path)
except OSError:
fire_event(SystemErrorRetrievingModTime(path=absolute_path))
if reobj.match(local_file):
if reobj.match(local_file) and (
not ignore_spec or not ignore_spec.match_file(relative_path_to_root)
):
matching.append(
{
"searched_path": relative_path_to_search,
@@ -164,7 +177,7 @@ def write_file(path: str, contents: str = "") -> bool:
reason = "Path was possibly too long"
# all our hard work and the path was still too long. Log and
# continue.
fire_event(SystemCouldNotWrite(path=path, reason=reason, exc=exc))
fire_event(SystemCouldNotWrite(path=path, reason=reason, exc=str(exc)))
else:
raise
return True

View File

@@ -183,6 +183,7 @@ class Compiler:
context = generate_runtime_model_context(node, self.config, manifest)
context.update(extra_context)
if isinstance(node, CompiledGenericTestNode):
# for test nodes, add a special keyword args value to the context
jinja.add_rendered_test_kwargs(context, node)

View File

@@ -23,8 +23,6 @@ from .renderer import ProfileRenderer
DEFAULT_THREADS = 1
DEFAULT_PROFILES_DIR = os.path.join(os.path.expanduser("~"), ".dbt")
INVALID_PROFILE_MESSAGE = """
dbt encountered an error while trying to read your profiles.yml file.
@@ -44,7 +42,7 @@ defined in your profiles.yml file. You can find profiles.yml here:
{profiles_file}/profiles.yml
""".format(
profiles_file=DEFAULT_PROFILES_DIR
profiles_file=flags.DEFAULT_PROFILES_DIR
)

View File

@@ -3,31 +3,42 @@ import os
from copy import deepcopy
from dataclasses import dataclass, field
from pathlib import Path
from typing import Dict, Any, Optional, Mapping, Iterator, Iterable, Tuple, List, MutableSet, Type
from typing import (
Any,
Dict,
Iterable,
Iterator,
List,
Mapping,
MutableSet,
Optional,
Tuple,
Type,
Union,
)
from .profile import Profile
from .project import Project
from .renderer import DbtProjectYamlRenderer, ProfileRenderer
from .utils import parse_cli_vars
from dbt import flags
from dbt.adapters.factory import get_relation_class_by_name, get_include_paths
from dbt.helper_types import FQNPath, PathSet, DictDefaultEmptyStr
from dbt.adapters.factory import get_include_paths, get_relation_class_by_name
from dbt.config.profile import read_user_config
from dbt.contracts.connection import AdapterRequiredConfig, Credentials
from dbt.contracts.graph.manifest import ManifestMetadata
from dbt.contracts.relation import ComponentName
from dbt.ui import warning_tag
from dbt.contracts.project import Configuration, UserConfig
from dbt.contracts.relation import ComponentName
from dbt.dataclass_schema import ValidationError
from dbt.exceptions import (
RuntimeException,
DbtProjectError,
RuntimeException,
raise_compiler_error,
validator_error_message,
warn_or_error,
raise_compiler_error,
)
from dbt.helper_types import DictDefaultEmptyStr, FQNPath, PathSet
from dbt.ui import warning_tag
from dbt.dataclass_schema import ValidationError
from .profile import Profile
from .project import Project, PartialProject
from .renderer import DbtProjectYamlRenderer, ProfileRenderer
from .utils import parse_cli_vars
def _project_quoting_dict(proj: Project, profile: Profile) -> Dict[ComponentName, bool]:
@@ -190,28 +201,52 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
@classmethod
def collect_parts(cls: Type["RuntimeConfig"], args: Any) -> Tuple[Project, Profile]:
# profile_name from the project
project_root = args.project_dir if args.project_dir else os.getcwd()
version_check = bool(flags.VERSION_CHECK)
partial = Project.partial_load(project_root, verify_version=version_check)
# build the profile using the base renderer and the one fact we know
# Note: only the named profile section is rendered. The rest of the
# profile is ignored.
cli_vars: Dict[str, Any] = parse_cli_vars(getattr(args, "vars", "{}"))
profile = cls.collect_profile(args=args)
project_renderer = DbtProjectYamlRenderer(profile, cli_vars)
project = cls.collect_project(args=args, project_renderer=project_renderer)
assert type(project) is Project
return (project, profile)
@classmethod
def collect_profile(
cls: Type["RuntimeConfig"], args: Any, profile_name: Optional[str] = None
) -> Profile:
cli_vars: Dict[str, Any] = parse_cli_vars(getattr(args, "vars", "{}"))
profile_renderer = ProfileRenderer(cli_vars)
profile_name = partial.render_profile_name(profile_renderer)
# build the profile using the base renderer and the one fact we know
if profile_name is None:
# Note: only the named profile section is rendered here. The rest of the
# profile is ignored.
partial = cls.collect_project(args)
assert type(partial) is PartialProject
profile_name = partial.render_profile_name(profile_renderer)
profile = cls._get_rendered_profile(args, profile_renderer, profile_name)
# Save env_vars encountered in rendering for partial parsing
profile.profile_env_vars = profile_renderer.ctx_obj.env_vars
return profile
# get a new renderer using our target information and render the
# project
project_renderer = DbtProjectYamlRenderer(profile, cli_vars)
project = partial.render(project_renderer)
# Save env_vars encountered in rendering for partial parsing
project.project_env_vars = project_renderer.ctx_obj.env_vars
return (project, profile)
@classmethod
def collect_project(
cls: Type["RuntimeConfig"],
args: Any,
project_renderer: Optional[DbtProjectYamlRenderer] = None,
) -> Union[Project, PartialProject]:
project_root = args.project_dir if args.project_dir else os.getcwd()
version_check = bool(flags.VERSION_CHECK)
partial = Project.partial_load(project_root, verify_version=version_check)
if project_renderer is None:
return partial
else:
project = partial.render(project_renderer)
project.project_env_vars = project_renderer.ctx_obj.env_vars
return project
# Called in main.py, lib.py, task/base.py
@classmethod

Some files were not shown because too many files have changed in this diff Show More