Compare commits

...

198 Commits

Author SHA1 Message Date
Github Build Bot
275232f15d Add generated CLI API docs 2023-02-15 03:44:23 +00:00
Michelle Ark
bb1d22ee3a get_with_fallback 2023-02-14 22:42:51 -05:00
Michelle Ark
6926e7071b Merge branch 'main' into arky/default-flags 2023-02-14 11:33:32 -05:00
Michelle Ark
e5c468bb93 dbt_project.yml log-path respects project directory (#6908)
dbt_project.yml log-path respects project directory
2023-02-14 11:30:11 -05:00
Github Build Bot
e2e38da1f4 Add generated CLI API docs 2023-02-14 15:57:01 +00:00
Michelle Ark
7ff090330d Merge branch 'main' into arky/default-flags 2023-02-14 10:55:34 -05:00
Emily Rockman
605c72e86e Automate cutting new .latest branches (#6854)
* first pass

* WIP

* add notes/stubs on more pieces

* more work

* more cleanup

* cleanup

* add more cleanup and generalization

* update to use reusable workflow

* add TODO
2023-02-14 08:09:16 -06:00
Emily Rockman
aad46ac5a8 Changelog Cleanup from feature branch merge (#6935)
* remove changelogs from 1.4

* remove yaml files for already released changes
2023-02-14 08:08:50 -06:00
Emily Rockman
d85618ef26 remove +nightly from version parsing (#6966)
* remove  from version parsing

* update comment

* undo partial version bump
2023-02-14 08:04:26 -06:00
Jeremy Cohen
1250f23c44 Fix petits logging regressions from feature/click-cli merge (#6940)
* Add back initialization events

* Fix log_cache_events. Default stdout logger knows less than it used to

* Add back exception handling events

* Revert "Add back exception handling events"

This reverts commit 26f22d91b660f51f0df6a59f9e5cae16b0ee6fe5.

* Add changelog entry

* Fix test by stringifying dict values

* Add generated CLI API docs

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-14 10:48:56 +01:00
Gerda Shank
daea7d59a7 CT 1921 add missing nodes to previous state test (#6961)
* Update WritableManifest.disabled to include metrics and exposures in disabled dictionary
2023-02-13 21:02:20 -05:00
Emily Rockman
4575757c2a remove '+nightly' from release test version (#6907)
* remove '+nightly' from release test version

* fix release branch variable name

* remove comment
2023-02-13 19:41:34 -06:00
Emily Rockman
d7a2f77705 Infer core team membership in changie (#6420)
* part 1 of env var for core team

* add logic to use env vars to generate changelog

* modify version bump to add members via env var

* pull in main and tweak

* add token

* changes for testing

* split step

* remove leading slash

* add version check

* more debugging

* try curl

* try more things

* try more things

* chnage auth

* put back token

* update permissions

* add back fishtown pat

* use new pat

* fix typo

* swap token

* comment out list teams

* change url

* debug path

* add continue

* change core case

* more tweaks

* send output to file

* add file view

* make array

* tweak

* remove []

* add quotes

* add tojson

* add quotes to set

* tweak

* fix id

* tweaks

* more

* more

* remove new lines

* more tweaks

* update to generate changelog

* remove debugging bits

* use central version-bump

* use correct author list

* testing with changelog team automation

* add new token to input

* move secret

* remove testing aspects from workflow

* clean up team logic

* explicitly send secret

* move bumpversion comment

* move comments

* point workflow back tp main

* point to branch for testing

* point back to main

* inherit secrets
2023-02-13 14:08:49 -06:00
Kshitij Aranke
4a4b89606b Hoist dbt.cli.main into dbt.main namespace (#6939) 2023-02-13 12:06:28 -08:00
Michelle Ark
1ebe2e7118 ExposureOwner: Rename to Owner, accept additional fields, require one of 'name' or 'email' (#6912)
update ExposureOwner to require one of email/name, allow additional fields
2023-02-13 10:13:15 -05:00
Gerda Shank
f1087e57bf Fix types.proto and test_events.py (#6947) 2023-02-10 16:31:34 -05:00
Michelle Ark
250537ba58 handle_and_check in main.py (#6941)
handle_and_check in cli/main.py
2023-02-10 15:39:31 -05:00
Peter Webb
ccc7222868 CT-2079: Re-apply accidentally reverted logging fix (#6937) 2023-02-10 14:40:57 -05:00
Chenyu Li
311a57a21e Merge pull request #6931 from dbt-labs/merge_feature_click_cli_main
Merge feature/click-cli into main

Co-authored-by: Ian Knox <ian.knox@dbtlabs.com >
Co-authored-by: Stu Kilgore <stu.kilgore@dbtlabs.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com >
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com >
Co-authored-by: Jeremy Cohen <jeremey@dbtlabs.com >
2023-02-09 18:03:04 -08:00
Chenyu Li
b7c45de6b1 remove old test 2023-02-09 15:55:24 -08:00
Chenyu Li
c53c3cf181 remove old main 2023-02-09 15:53:37 -08:00
Chenyu Li
a77d325c8a fix all tests 2023-02-09 15:51:42 -08:00
Kshitij Aranke
dd41384d82 revert core.dbt.contracts back to code from main 2023-02-09 15:32:34 -08:00
Github Build Bot
aa55fb2d30 Add generated CLI API docs 2023-02-09 22:59:54 +00:00
Kshitij Aranke
864f4efb8b Merge feature/click-cli into main 2023-02-09 14:57:10 -08:00
dependabot[bot]
83c5a8c24b Bump ubuntu from 22.04 to 23.04 (#6865)
* Bump ubuntu from 22.04 to 23.04

Bumps ubuntu from 22.04 to 23.04.

---
updated-dependencies:
- dependency-name: ubuntu
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-09 16:59:25 -05:00
Kshitij Aranke
57aef33fb3 Click cli main merge (#6926) 2023-02-09 12:21:39 -08:00
FishtownBuildBot
6d78e5e640 Add most recent dbt-docs changes (#6923)
* Add new index.html and changelog yaml files from dbt-docs

* Update .changes/unreleased/Docs-20230209-082901.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-02-09 13:23:12 -06:00
Ryan Harris
f54a876f65 Update link to dbt install docs (#6883)
* update dbt install link

* add changelog entry
2023-02-09 09:35:33 -08:00
Kshitij Aranke
8bbae7926b Fix Project Env Var Tests (#6916)
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2023-02-09 08:53:39 -08:00
Jeremy Cohen
db2b12021e Fix test_builtin_invocation_args_dict_function (#6898)
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2023-02-09 09:53:42 -06:00
Peter Webb
8b2c9bf39d Ensure flush() after logging write() (#6909)
* ct-2063: Ensure flush after logging, by using Python's logging subsystem directly

* ct-2063: Add changelog entry
2023-02-09 09:37:37 -05:00
Jeremy Cohen
298bf8a1d4 Add back depends_on for seeds - only macros, never nodes (#6851)
* Extend functional tests for seeds w hooks

* Add MacroDependsOn to seeds, raise exception for other deps

* Add changelog entry

* Fix unit tests

* Update upgrade_seed_content

* Cleanup

* Regen manifest v8 schema. Fix tests

* Be less magical

* PR feedback
2023-02-09 10:56:12 +01:00
Michelle Ark
77748571b4 profiles dir exists=False for dbt debug (#6910)
profiles dir exists=False for dbt debug
2023-02-08 18:32:06 -05:00
Stu Kilgore
8ce4c289c5 Docs generate doesn't write manifest (#6905) 2023-02-08 16:16:31 -06:00
Emily Rockman
abbece8876 1.4 regression: Check if status has node attribute (#6899)
* check for node

* add changelog

* add test for regression
2023-02-08 13:49:24 -06:00
dave-connors-3
3ad40372e6 add base class for merge exclude tests (#6700)
* add base class for merge exclude tests

* changie <33

* remove comments

* add comments to sql, remove and clarify contents of resultholder
2023-02-08 10:20:13 -08:00
Kshitij Aranke
c6d0e7c926 Fix Click CLI test DB Name (#6895) 2023-02-08 09:58:16 -08:00
Github Build Bot
78ceb55cd9 Add generated CLI API docs 2023-02-08 17:53:59 +00:00
Michelle Ark
c176287d17 Merge branch 'feature/click-cli' into arky/default-flags 2023-02-08 12:52:01 -05:00
Michelle Ark
a156aabb10 support empty Flags constructor 2023-02-08 12:46:14 -05:00
Michelle Ark
bc015843d4 use UnsetProfile in deps and clean commands (#6890)
use unset_profile in deps and clean commands
2023-02-08 12:26:58 -05:00
Emily Rockman
df64511feb Dynamically list all .latest branches for scheduled testing (#6682)
* first pass at automating latest branches

* checkout repo first

* fetch all history

* reorg

* debugging

* update test id

* swap lines

* incorporate new branch aciton

* tweak vars
2023-02-08 08:02:21 -06:00
Github Build Bot
177b244d82 Add generated CLI API docs 2023-02-08 03:12:26 +00:00
Michelle Ark
140a3e8007 profit! 2023-02-07 22:10:56 -05:00
Michelle Ark
1454dfdc97 Merge branch 'feature/click-cli' into arky/default-flags 2023-02-07 22:05:01 -05:00
Michelle Ark
b83786f31e Merge branch 'feature/click-cli' into arky/default-flags 2023-02-07 22:02:08 -05:00
Michelle Ark
db0981afe7 NoneConfig for DebugTask (#6893)
NoneConfig for DebugTask
2023-02-07 21:33:15 -05:00
Michelle Ark
dcf6544f93 flags.THREADS defaults to None (#6887)
flags.THREADS defaults to None
2023-02-07 17:15:26 -05:00
Ian Knox
c2c8959fee Merge main to feature (#6817)
merge main to feature/click-cli
2023-02-07 16:32:31 -05:00
Peter Webb
ccb4fa26cd CT-1917: Fix a regression in the behavior of the -q/--quiet cli parameter (#6886) 2023-02-07 16:15:42 -05:00
Chenyu Li
d0b5d752df consolidate flags (#6788)
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-07 08:35:35 -08:00
Neelesh Salian
4c63b630de [CT-1959]: moving simple_seed tests to adapter zone (#6859)
* Formatting

* Changelog entry

* Rename to BaseSimpleSeedColumnOverride

* Better error handling

* Update test to include the BOM test

* Cleanup and formating

* Unused import remove

* nit line

* Pr comments
2023-02-06 19:51:32 -08:00
Kshitij Aranke
9c0b62b4f5 Fix CLI vars test to check for object instead of string (#6850) 2023-02-06 15:13:40 -08:00
Ian Knox
e08eede5e2 Remove unused cli_runner (#6877) 2023-02-06 15:24:09 -06:00
Ian Knox
05e53d4143 Test fix: TestProfileEnvVars::test_profile_env_vars (#6856) 2023-02-06 10:49:40 -06:00
Github Build Bot
f0b026ae72 Add generated CLI API docs 2023-02-04 01:33:03 +00:00
Michelle Ark
19f47d16f7 default flags + test 2023-02-03 20:31:32 -05:00
Michelle Ark
414e98198d load default click param value 2023-02-03 19:51:34 -05:00
colin-rogers-dbt
b2ea2b8b25 move test_store_test_failures.py to adapter zone (#6816) 2023-02-02 10:55:09 -08:00
Emily Rockman
2245d8d710 update regex to match all iterations (#6839)
* update regex to match all iterations

* convert to num to match all adapters

* add comments, remove extra .

* clarify with more comments

* Update .bumpversion.cfg

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>

---------

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
2023-02-02 12:16:06 -06:00
Gerda Shank
d9424cc710 CT 2000 fix semver prerelease comparisons (#6838)
* Modify semver.py to not use packaging.version.parse

* Changie
2023-02-02 12:22:48 -05:00
Stu Kilgore
0503c141b7 Lazily call --version (#6813)
* Lazily call --version

* Add generated CLI API docs

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-31 17:20:18 -08:00
Mila Page
1a6e4a00c7 Add clearer directions for custom test suite vars in Makefile. (#6764)
* Add clearer directions for custom test suite vars in Makefile.

* Fix up PR for review

* Fix erroneous whitespace.

* Fix a spelling error.

* Add documentation to discourage makefile edits but provide override tooling.

* Fix quotation marks. Very strange behavior

* Compact code and verify quotations happy inside bash and python.

* Fold comments into Makefile.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-31 13:53:55 -08:00
Mila Page
42b7caae19 Ct 1827/064 column comments tests conversion (#6766)
* Convert test and make it a bit more pytest-onic

* Ax old integration test.

* Run black on test conversion

* I didn't like how pytest was running the fixture so wrapped it into a closure.

* Merge converted test into persist docs.

* Move persist docs tests to the adapter zone. Prep for adapter tests.

* Fix up test names

* Fix name to be less confusing.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-31 12:58:19 -08:00
Emily Rockman
622e5fd71d fix contributor list generation (#6799) 2023-01-31 13:05:13 -06:00
Stu Kilgore
59d773ea7e Implement --version in click (#6802) 2023-01-31 12:40:01 -06:00
Kshitij Aranke
84bf5b4620 [CT-1947] Alias --models to --select for all commands except dbt ls (#6787)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-31 10:18:10 -08:00
Ian Knox
726c4d6c58 Enable the new Click Cli (#6785) 2023-01-30 19:30:01 -06:00
Michelle Ark
acc88d47a3 mutually exclusive handling for warn_error_options and warn_error params in Click CLI (#6771)
warn_error_options, warn_error mutual exclusivity with click
2023-01-30 18:38:36 -05:00
Chenyu Li
0a74594d09 move favor state arg to click (#6774) 2023-01-30 11:55:31 -08:00
Kshitij Aranke
d2f3cdd6de [CT-1841] Convert custom target test to Pytest (#6765) 2023-01-30 07:55:28 -08:00
Alexander Smolyakov
92d1ef8482 Update release workflow (#6778)
- Update AWS secrets
- Rework condition for Slack notification
2023-01-30 09:19:10 -06:00
Neelesh Salian
a8abc49632 [CT-1940] Stand-alone Python module for PostgresColumn (#6773) 2023-01-27 17:00:39 -08:00
Ian Knox
d6ac340df0 Merge main into feature/click-cli (#6761) 2023-01-27 15:07:30 -06:00
Neelesh Salian
c653330911 Adding nssalian to committers list (#6769) 2023-01-27 11:16:59 -08:00
Alexander Smolyakov
82d9b2fa87 [CI/CD] Update release workflow and introduce workflow for nightly releases (#6602)
* Add release workflows

* Update nightly-release.yml

* Set default `test_run` value to `true`

* Update .bumpversion.cfg

* Resolve review comment

- Update workflow docs
- Change workflow name
- Set `test_run` default value to `true`

* Update Slack secret

* PyPI
2023-01-27 09:04:31 -06:00
Mila Page
3f96fad4f9 Ct 1629/052 column quoting tests conversion (#6652)
* Test converted and reformatted for pytest.

* Ax old versions of 052 test

* Nix the 'os' import and black format

* Change names of models to be more PEP like

* cleanup code

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-26 12:23:02 -08:00
Peter Webb
c2c4757a2b Graph Analysis Optimization for Large Dags (#6720)
* Optimization to remove graph analysis bottleneck in large dags.

* Add changelog entry.
2023-01-26 14:27:42 -05:00
Kshitij Aranke
08b2d94ccd [CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs … (#6723)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-26 10:42:49 -08:00
Michelle Ark
7fa61f0816 dbt init works with click (#6698)
dbt init works with click
2023-01-26 12:45:15 -05:00
Mila Page
c65ba11ae6 Ct 1827/064 column comments tests conversion (#6654)
* Convert test and make it a bit more pytest-onic

* Ax old integration test.

* Run black on test conversion

* I didn't like how pytest was running the fixture so wrapped it into a closure.

* Merge converted test into persist docs.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-26 00:54:00 -08:00
Matthew Beall
b0651b13b5 change exposure_content to source_content (#6739)
* change `exposure_content` to `source_content`

* Adding changelog

Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2023-01-25 19:51:34 -05:00
Gerda Shank
a34521ec07 CT 1894 log partial parsing var changes and sort cli vars before hashing (#6713)
* Log information about vars_hash, normalize cli_vars before hashing

* Changie

* Add to test_events.py

* Update core/dbt/events/types.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-01-25 17:47:45 -05:00
Matthew McKnight
da47b90503 [CT-1630] Convert Column_types tests (#6690)
* init commit for column_types test conversion

* init start of test_column_types.py

* pass tes macros into both tests

* remove alt tests, remove old tests, push up working conversion

* rename base class, move to adapter zone so adapters can use

* typo fix
2023-01-25 14:57:16 -06:00
Stu Kilgore
d27016a4e7 Migrate debug task to click (#6728) 2023-01-25 14:52:57 -06:00
Peter Webb
db99e2f68d Event Clean-Up (#6716)
* CT-1857: Event cleanup

* Add changelog entry.
2023-01-25 13:51:52 -05:00
Michelle Ark
cbb9117ab9 test_init conversion (#6610)
* convert 044_init_tests
2023-01-24 16:22:26 -05:00
Gerda Shank
e2ccf011d9 CT 1886 include adapter_response in NodeFinished log message (#6709)
* Include adapter_response in run_result in NodeFinished log event

* Changie
2023-01-24 14:25:32 -05:00
Aezo
17014bfad3 add adapter_response for test (#6645)
resolves https://github.com/dbt-labs/dbt-core/issues/2964
2023-01-24 09:58:08 -08:00
Stu Kilgore
92b7166c10 Abstract manifest generation from tasks (#6565) 2023-01-24 11:05:57 -06:00
Peter Webb
7b464b8a49 CT-1718: Add Note and Formatting event types (#6691)
* CT-1718: Add Note and Formatting event types

* CT-1718: Add changelog entry
2023-01-23 16:39:29 -05:00
Sean McIntyre
5c765bf3e2 Cheeky performance improvement on big DAGs (#6694)
* Short-circuit set operations for nice speed boost

* Add changelog

* Fix issue

* Update .changes/unreleased/Under the Hood-20230122-215235.yaml

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-01-23 09:09:09 -07:00
Mila Page
93619a9a37 Ct 738/dbt debug log fix (#6541)
* Code cleanup and adding stderr to capture dbt

* Debug with --log-format json now prints structured logs.

* Add changelog.

* Move logs into miscellaneous and add values to test.

* nix whitespace and fix log levels

* List will now do structured logging when log format set to json.

* Add a quick None check.

* Add a get guard to class check.

* Better null checking

* The boolean doesn't reflect the original logic but a try-catch does.

* Address some code review comments and get us working again.

* Simplify logic now that we have a namespace object for self.config.args.

* Simplify logic for json log format checking.

* Simplify code for allowing our GraphTest cases to pass while also hiding compile stats from dbt ls/list .

* Simplify structured logging types.

* Fix up boolean logic and simplify via De'Morgan.

* Nix unneeded fixture.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-20 16:37:54 -08:00
Doug Beatty
a181cee6ae Improve error message for packages missing dbt_project.yml (#6685)
* Improve error message for packages missing `dbt_project.yml`

* Use black formatting

* Update capitalization of expected error message
2023-01-20 13:46:36 -07:00
Michelle Ark
a0ade13f5a dbt docs generate works with click (#6681) 2023-01-20 15:26:20 -05:00
Michelle Ark
9823a56e1d dbt build works in click (#6680)
* build working with click
2023-01-20 14:27:39 -05:00
Michelle Ark
3aeab73740 convert 069_build_tests (#6678) 2023-01-20 14:27:02 -05:00
Jeremy Cohen
9801eebc58 Consolidate changie entries from #6620 (#6684) 2023-01-20 19:58:40 +01:00
Jeremy Cohen
11c622230c Add run-operation to click CLI (#5552) (#6656)
* Add run-operation to click CLI

* Add changelog entry

* PR feedback

* Fix unit test
2023-01-20 01:58:47 +01:00
Chenyu Li
f0349488ed Seed and freshness works with click (#6651) 2023-01-19 16:14:26 -08:00
Chenyu Li
c85be323f5 fix MP_CONTEXT is not JSON serializable (#6650) 2023-01-19 08:16:58 -08:00
Peter Webb
6954c4df1b CT-1786: Port docs tests to pytest (#6608)
* CT-1786: Port docs tets to pytest

* Add generated CLI API docs

* CT-1786: Comply with the new style requirements

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-19 11:11:17 -05:00
Michelle Ark
30a1595f72 click working with list (#6641)
* list working with click
2023-01-19 09:28:32 -05:00
dave-connors-3
f841a7ca76 add backwards compatibility and default argument for incremental_predicates (#6628)
* add backwards compatibility and default argument

* changie <3

* Update .changes/unreleased/Fixes-20230117-101342.yaml

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-01-19 15:20:19 +01:00
Jeremy Cohen
07a004b301 convert 062_defer_state_tests (#6616)
* Fix --favor-state flag

* Convert 062_defer_state_tests

* Revert "Fix --favor-state flag"

This reverts commit ccbdcbad98b26822629364e6fdbd2780db0c20d3.

* Reformat

* Revert "Revert "Fix --favor-state flag""

This reverts commit fa9d2a09d6.
2023-01-19 11:00:09 +01:00
Jeremy Cohen
b05582de39 mv on_schema_change tests -> "adapter zone" (#6618)
* Mv incremental on_schema_change tests to 'adapter zone'

* Use type_string()

* Cleanup
2023-01-19 10:12:59 +01:00
Jeremy Cohen
fa7c4d19f0 Respect quoting config in dbt-py models (#6620)
* Respect quoting for 'this' in dbt-py models #6619

* Respect quoting for ref/source in dbt-py models #6103

* Add changelog entries
2023-01-19 09:34:08 +01:00
Chenyu Li
1913eac5ed Click snapshot as click command (#5972) (#6640) 2023-01-18 16:01:58 -08:00
Jeremy Cohen
066346faa2 convert 038_caching_tests (#6612)
* convert 038_caching_tests

* Adapt for dbt-snowflake

* PR feedback

* Reformat
2023-01-18 22:37:50 +01:00
Emily Rockman
0a03355ceb update test matrix (#6604) 2023-01-18 14:16:34 -06:00
David Bloss
43e24c5ae6 update gh action set-output variables (#6635)
* update gh action set-output variables

* add changie file
2023-01-18 11:23:13 -06:00
Gerda Shank
89d111a5f6 CT 1440 Fix code to emit ConnectionReused event (#6605)
* Refactor "set_connection_name" to properly handle reused connection

* Update test

* Changie

* Limit test of ConnectionUsed events to non-Windows
2023-01-17 13:18:07 -05:00
Gerda Shank
e1b5e68904 Convert 068_partial_parsing_tests (#6614)
* Convert partial parsing tests

* reformat
2023-01-17 12:22:31 -05:00
Jeremy Cohen
065ab2ebc2 Reformat tests/ (#6622)
* Run black + flake8 on tests dir

* Run pre-commit
2023-01-16 16:39:54 +01:00
Michelle Ark
20c95a4993 convert 057_run_query_tests (#6607) 2023-01-13 15:02:17 -05:00
Michelle Ark
c40b488cb4 convert 037_external_references (#6589) 2023-01-13 14:15:03 -05:00
Chenyu Li
585e7c59e8 migrate ref override (#6600) 2023-01-13 09:15:04 -08:00
Michelle Ark
7077c47551 converting 022_test_timezones (#6588) 2023-01-13 10:52:21 -05:00
Chenyu Li
f789b2535a convert adapter table materialization test (#6595) 2023-01-12 16:05:51 -08:00
Emily Rockman
2bfc6917e2 finish message rename in types.proto (#6594)
* finish message rename in types.proto

* add new parameter
2023-01-12 14:33:56 -06:00
github-actions[bot]
d74ae19523 Bumping version to 1.5.0a1 and generate changelog (#6593)
* Bumping version to 1.5.0a1 and generate CHANGELOG

* make relevvant changelog updates

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-01-12 12:59:28 -06:00
Michelle Ark
1c7c23ac73 convert 043_custom_alias_tests (#6590) 2023-01-12 11:07:26 -05:00
Jeremy Cohen
86e8722cd8 Call update_event_status earlier + rename an event (#6572)
* Rename HookFinished -> FinishedRunningStats

* Move update_event_status earlier when node finishes

* Add changelog entry

* Add update_event_status for skip

* Update changelog entry
2023-01-12 09:34:36 +01:00
Kshitij Aranke
53127daad8 [CT-921] dbt compile works in click (#5545) (#6586)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
resolves https://github.com/dbt-labs/dbt-core/issues/5545
2023-01-11 15:01:50 -08:00
Kshitij Aranke
91b20b7482 dbt test works with Click (#5556)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
resolves https://github.com/dbt-labs/dbt-core/issues/5556
2023-01-11 12:19:43 -08:00
Michelle Ark
7a61602738 Add --warn-error-options (#6520)
* adding --warn-error-options

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-10 19:43:52 -05:00
Doug Beatty
dd4b47d8b1 Check length of escaped strings in the adapter test (#6567)
* Check length of escaped strings in the adapter test

* One column per line to improve readability
2023-01-10 17:26:35 -07:00
Emily Rockman
eb200b4687 Rename exceptions (#6539)
* rename InternalException

* rename RuntimeException

* rename DatabaseException

* rename CompilationException

* cleanup renames in tests and postgres

* rename ValidationException

* rename IncompatibleSchemaException

* more renaming

* more renaming

* rename InternalException again

* convert ParsingException

* replace JSONValidationException and SemverException

* replace VersionsNotCompatibleException

* replace NotImplementedException

* replace FailedToConnectException

* replace InvalidConnectionException

* replace InvalidSelectorException

* replace DuplicateYamlKeyException

* replace ConnectionException

* minor cleanup

* update comment

* more cleanup

* add class decorator

* rename more exceptions

* more renamed, add changelog

* rename exception

* rework class deprecations

* removing testing line

* fix failing test

* rename newer exceptions

* fix failing test

* commit unsaved faile

* convert back an rpc exception

* remove class deprecations
2023-01-10 17:43:27 -06:00
mivanicova
0fc080d222 add defer_to_manifest in before_run to fix faulty deferred docs generate (#6488)
* add defer_to_manifest in before_run to fix faulty deferred docs generate

* add a changelog

* add declaration of defer_to_manifest to FreshnessTask and GraphRunnableTask

* fix: add defer_to_manifest method  to ListTask
2023-01-10 15:18:20 -08:00
AGPapa
5da63602b3 Adds buildable selection mode (#6366) 2023-01-10 15:15:48 -08:00
nshuman1
457ff3ef48 fixed minor typos (#6499)
* fixed minor typos

* adding changelog file

* linter

* Update Docs-20230102-170216.yaml
2023-01-10 13:36:21 -08:00
Gerda Shank
0dbdecef10 Ct 1716 cleanup logging events (#6561)
* Combine DbtProfileError log events

* Combine DbtProjectErrorException with DbtProjectError

* Combine cache logging events

* Changie

* fix ticket number

* Ooops. Add another file.

* fix serialization of profile names
2023-01-10 15:56:52 -05:00
Doug Beatty
b13b0e9492 Re-factor list of YAML keys for hooks to late-render (#6435)
* Re-factor list of YAML keys for hooks to late-render

* Add `pre_` and `post_hook` to list of late-rendered hooks

* Check for non-empty set intersection

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>

* Test functional synonymy of `*_hook` with `*-hook`

Test that `pre_hook`/`post_hook` are functionally synonymous with `pre-hook`/`post-hook` for model project config

* Undo bugfix to validate the new test fails

* Revert "Undo bugfix to validate the new test fails"

This reverts commit e83a2be2eb.

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-01-10 12:35:06 -07:00
Tim Mastny
b9fdfd9e36 Adds the meta field to node_info in logs (#6493)
* add meta attribute to nodeinfo for events

* also add meta to dataclass

* add to unit test to ensure meta is added

* adding functional test to check that meta is passed to nodeinfo during logging

* changelog

* remove used imported

* add tests with non-string keys

* renaming test dict keys

* add non-string value

* resolve failing test

* test additional non-string values

* fix flake8

* Stringify meta dict in node_info

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-01-10 13:42:51 -05:00
Peter Webb
4d6352db14 CT-1645: Bump mashumaro version to receive regression fix, and add unit test to confirm fix. (#6564) 2023-01-10 13:14:24 -05:00
Doug Beatty
9eb82c6497 Remove duplicated changelog entry for 5521 (#6563) 2023-01-10 11:13:50 -07:00
Jeremy Cohen
89cc89dfdf Fix changelog entry for 5989/6432 (#6560) 2023-01-10 06:57:39 -07:00
Kshitij Aranke
2b0f6597a4 [CT-1694] Deprecate event tracking tests (#6538) 2023-01-09 15:23:02 -08:00
Peter Webb
294def205f Fix event level regression (#6556)
* CT-6501: Make certain disabled test events debug-level again to match historical behavior.

* CT-1740: Add changelog entry.
2023-01-09 18:15:59 -05:00
Emily Rockman
34fa703466 fix log msg format (#6557) 2023-01-09 15:56:42 -06:00
Mila Page
ab3f8dcbfd Ct 1517/convert 060 persist docs (#6409)
* convert the test and fix an error due to a dead code seed

* Get rid of old test

* Remove unfortunately added files. Don't use that *

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-09 11:07:34 -08:00
Emily Rockman
02c20477b9 add deprecation decorator (#6540)
* add derecation decorator

* fix tests

* updated changelog and deprecation reason
2023-01-09 12:09:12 -06:00
Gerda Shank
d9a4ee126a CT 1549 reorg logging events to have two top level keys (#6553)
* Update types.proto

* pre-commit passes

* Cleanup tests and tweak EventLevels

* Put node_info back on SQLCommit. Add "level" to fire_event function.

* use event.message() in warn_or_error

* Fix logging test

* Changie

* Fix a couple of unit tests

* import Protocol from typing_extensions for 3.7
2023-01-09 11:33:44 -05:00
Michelle Ark
5b31cc4266 Remove UnsetProfileConfig (#6504)
remove UnsetProfileConfig
2023-01-06 20:12:40 -05:00
Chenyu Li
9bb1250869 merge main to feature/click-cli (#6483)
* merge main to feature/click-cli

* fix var hash

* move back changes in feature branch for deps

* fix logging issues
2023-01-06 15:11:23 -08:00
Kshitij Aranke
94d6d19fb4 [CT-1693] Port severity test to Pytest (#6537) 2023-01-06 11:59:53 -08:00
Callum McCann
d43c070007 Making Metric Timestamps Optional (#6402)
* changing to optional

* adding tests

* tests and changie

* pre-commit cleaning

* formatting fixes

* pre-commit update
2023-01-05 15:49:55 -05:00
Callum McCann
9ef236601b Updating metric attributes renamed event (#6518)
* updating message

* adding changie
2023-01-05 15:49:44 -05:00
justbldwn
9d6f961d2b 🎨 add missing fstrings, convert format to fstring (#6243)
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-01-05 15:28:27 -05:00
Niall Woodward
5453840950 DBT_FAVOR_STATE_STATE -> DBT_FAVOR_STATE (#6392)
* DBT_FAVOR_STATE_STATE -> DBT_FAVOR_STATE

* add changelog
2023-01-04 15:53:35 -08:00
Gerda Shank
d453964546 CT 1551 partial parsing logging events (#6474) 2023-01-04 14:28:26 -05:00
dependabot[bot]
748a932811 Update agate requirement from <1.6.4,>=1.6 to >=1.6,<1.7.1 in /core (#6506)
* Update agate requirement from <1.6.4,>=1.6 to >=1.6,<1.7.1 in /core

Updates the requirements on [agate](https://github.com/wireservice/agate) to permit the latest version.
- [Release notes](https://github.com/wireservice/agate/releases)
- [Changelog](https://github.com/wireservice/agate/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/wireservice/agate/compare/1.6.0...1.7.0)

---
updated-dependencies:
- dependency-name: agate
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-04 09:02:37 -08:00
Jeremy Cohen
8217ad4722 Fix issue link for 5859 changelog entry (#6496) 2023-01-04 10:46:52 +01:00
Michelle Ark
cc5a38ec5a Example click API usage (#6307)
* Example python lib w click, written docs

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-01-03 18:46:34 -05:00
Gerda Shank
6ef3fbbf76 Partial parsing bug with empty schema file - ensure None is not passed to load_yaml_text (#6494) 2023-01-03 17:14:56 -05:00
justbldwn
76fd12c7cd adding pre-commit install to make dev (#6417)
*  adding pre-commit install to make dev

* 🎨 updating format of Makefile and CONTRIBUTING.md

* 📝 adding changelog via changie new

*  adding dev_req to Makefile + docs

* 🎨 remove dev_req from docs, dry makefile

* Align names of `.PHONY` targets with their associated rules

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
2023-01-03 11:18:55 -07:00
Doug Beatty
9ecb6e50e4 Treat dense text blobs as binary for git grep (#6462) 2022-12-20 17:10:01 -07:00
Kshitij Aranke
ce9d0afb8a [CT-1591] Don't parse empty Python files (#6433) 2022-12-20 10:56:36 -08:00
Emily Rockman
c39ea807e8 add back validator_error_message (#6465) 2022-12-20 10:56:08 -06:00
Ikko Ashimine
1e35339389 Fix typo in util.py (#6037)
* Fix typo in util.py

identifer -> identifier

* Add change log

* Update .changes/unreleased/Fixes-20221117-220320.yaml

Co-authored-by: Emily Rockman <ebuschang@gmail.com>
2022-12-19 20:01:19 -06:00
Emily Rockman
304797b099 exception cleanup (#6347)
* starting to move jinja exceptions

* convert some exceptions

* add back old functions for backward compatibility

* organize

* more conversions

* more conversions

* add changelog

* split out CacheInconsistency

* more conversions

* convert even more

* convert parsingexceptions

* fix tests

* more conversions

* more conversions

* finish converting exception functions

* convert more tests

* standardize to msg

* remove some TODOs

* fix test param and check the rest

* add comment, move exceptions

* add types

* fix type errors

* fix type for adapter_response

* remove 0.13 version from message
2022-12-19 19:36:24 -06:00
Emily Rockman
b9bdb775ab Fix dependency changelog kind (#6452) 2022-12-19 13:13:38 -06:00
Michelle Ark
b0909b8f5d Functional test framework working with Click, dbtRunner (#6387)
dbtRunner, dbt.cli.requires - preflight, profile, project
2022-12-16 19:54:42 -05:00
Stu Kilgore
5d278dacf1 Make clean task work with click (#6369) 2022-12-15 11:19:16 -06:00
leahwicz
df93858b4b Updating 1.4.0b1 changelog (#6450) 2022-12-15 09:55:23 -05:00
dave-connors-3
e8da84fb9e Feature/support incremental predicates (#5702)
* pass predicated to merge strategy

* postgres delete and insert

* merge with predicates

* update to use arbitrary list of predicates, not dictionaries, merge and delete

* changie

* add functional test to adapter zone

* comma in test config

* add test for incremental predicates delete and insert postgres

* update test structure for inheritance

* handle predicates config for backwards compatibility

* test for predicates keyword

* Add generated CLI API docs

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-12-14 14:07:41 -08:00
Peter Webb
7e90e067af Paw/ct 1652 restore default logging (#6447)
* CT-1652: Restore stdout logging before logger is fully configured

* CT-1652: Add changelog entry

* CT-1652: formatting fix for black
2022-12-14 17:04:50 -05:00
Peter Webb
5e4e917de5 CT-1685: Restore certain aspects of legacy logging behavior important… (#6443)
* CT-1685: Restore certain aspects of legacy logging behavior important to dbt-rpc

* CT-1658: And changelog entry
2022-12-14 11:13:34 -05:00
github-actions[bot]
05dc0212e7 Bumping version to 1.4.0b1 and generate changelog (#6440)
* Bumping version to 1.4.0b1 and generate CHANGELOG

* Updating date

* Updating date

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-12-13 20:18:11 -05:00
Gerda Shank
c00052cbfb Add Optional back on "database" field of HasRelationMetadata (#6439) 2022-12-13 18:15:25 -05:00
Kshitij Aranke
3d54a83822 [CT-1284] Change Python model default materialization to table (#6432) 2022-12-13 15:07:56 -08:00
Gerda Shank
fafd5edbda CT 1644 node cleanup (#6427)
* Remove unneeded SQL compilation attributes from SeedNode

* Fix various places that referenced removed attributes

* Cleanup a few Unions

* More formatting in nodes.py

* Mypy passing. Untested.

* Unit tests working

* use "doc" in documentation unique_ids

* update some doc_ids

* Fix some artifact tests. Still need previous version.

* Update manifest/v8.json

* Move relation_names to parsing

* Fix a couple of tests

* Update some artifacts. snapshot_seed has wrong schema.

* Changie

* Tweak NodeType.Documentation

* Put store_failures property in the right place

* Fix setting relation_name
2022-12-13 12:39:35 -05:00
Josh Devlin
8478262580 Update docker README (#6423) 2022-12-13 11:12:34 -05:00
Kshitij Aranke
83b1fee062 Add aranke to core committers (#6431) 2022-12-12 15:13:03 -08:00
Emily Rockman
0fbbc896b2 Remove PR from most changelog kinds (#6374)
* update changie to require issue or pr, and allow multiple

* remove extraneous data from changelog files.

* allow for multiple PR/issues to be entered

* update contributing guide

* remove issue number from bot changelogs

* update format of PR

* fix dependency changelogs

* remove extra line

* remove extra lines, tweak contributor wording

* Update CONTRIBUTING.md

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2022-12-12 13:18:15 -06:00
Stu Kilgore
ce1aaec31d Adjust tox passenv to be multiline (#6405) (#6430) 2022-12-12 11:13:04 -06:00
Ian Knox
0544b08543 Add support for Python 3.11 (#6326)
* Get running with Python 3.11

* More tests passing, mypy still unhappy

* Upgrade to 3.11, and bump mashumaro

* patch importlib.import_module last

* lambda: Policy() default_factory on include and quote policy

* Add changelog entry

* Put a lambda on it

* Fix text formatting for log file

* Handle variant type return from e.log_level()

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Josh Taylor <joshuataylorx@gmail.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2022-12-08 18:34:03 +01:00
Ian Knox
1809852a0d dbt run works with Click (#6396) 2022-12-08 09:47:08 -06:00
Ian Knox
88d2ee4813 Profile works w Click (#6336) 2022-12-08 09:38:15 -06:00
bruno messias
bef6edb942 Fix dbt.config.get default values (python-model) (#6317)
* feat: add a list of default values to the ctx manager

* tests: dbt.get.config default values

* feat: validate the num of args in config.get

* feat: jinja template for dbt.config.get default values

* docs: changie yaml

* fix:typo on error message

Co-authored-by: Chenyu Li <chenyulee777@gmail.com>

Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
2022-12-07 15:52:38 -08:00
timle
99f27de934 Feature/dbt deps tarball (#4689)
* v0 - new dbt deps type: tarball url

in support of
https://github.com/dbt-labs/dbt-core/issues/4205

* flake8 fixes

* adding max size tarball condition

* clean up imports

* typing

* adding sha1 and subdirectory options; improve logging feedback

sha1: allow user to specify sha1 in packages.yaml, will only install if package matches
subdirectory: allow user to specify subdirectory of package in tarfile, if the package is a non standard structure (like with git subdirectory option)

* simple tests added

* flake fixes

* changes to support tests; adding exceptions; fire_event logging

* new logging events

* tarball exceptions added

* build out tests

* removing in memory tarball test

* update type codes to M - Misc

* adding new events to test_events

* fix spacing for flake

* add retry download code - as used in registry calls

* clean

* remove saving tar in memory inside tarfile object

will hit url multiple times instead

* remove duplicative code after refactor

* black updates

* black formatting

* black formatting

* refactor - no more in-memory tarfile - all as file operations now

- remove tarfile passing, always use tempfile instead
- reorganize system.* functions, removing duplicative code
- more notes on current flow and structure - esp need for pattern of 1) unpack 2) scan for package dir 3) copy to destination.
- cleaning

* cleaning and sync to new tarball code

* cleaning and sync to new tarball code

* requested changes from PR

https://github.com/dbt-labs/dbt-core/pull/4689#discussion_r812970847

* reversions from revision 2

removing sha1 check to simplify/mirror hub install pattern

* simplify/mirror hub install pattern

to simplify/mirror hub install pattern
- removing sha1 check
- supply name/version to act as our 'metadata' source

* simplify/mirror hub install pattern

simplify with goal of mirroring hub install pattern
- supporting subfolders like git packages, and sha1 checks are removed
- existing code from RegistryPinnedPackage (install() and download_and_untar()) performs the operations
- RegistryPinnedPackage install() and download_and_untar() are not currently set up as functions that can be used across classes - this should be moved to dbt.deps.base, or to a dbt.deps.common file - need dbt labs feedback on how to proceed (or leave as is)

* remove revisions, no longer doing package check

* slim down to basic tests

more complex features have been removed (sha1, subfolder) so testing is much simpler!

* fix naming to match hubs behavior

remove version from package folder name

* refactor install and download to upstream PinnedPackage class

i'm on the fence if this is right approach, but seems like most sensible after some thought

* Create Features-20221107-105018.yaml

* fix flake, black, mypy errors

* additional flake/black fixes

* Update .changes/unreleased/Features-20221107-105018.yaml

fix username on changelog

Co-authored-by: Emily Rockman <ebuschang@gmail.com>

* change to fstring

Co-authored-by: Emily Rockman <ebuschang@gmail.com>

* cleaning - remove comment

* remove comment/question for dbt team

* in support of issuecomment 1334055944

https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334055944

* in support of issuecomment 1334118433

https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334118433

* black fixes; remove debug bits

* remove `.format` & add 'tarball' as version

'tarball' as version so that the temp files format nicely:
[tempfile_location]/dbt_utils_2..tar.gz # old
vs
[tempfile_location]/dbt_utils_1.tarball.tar.gz # current

* port os.path refs in `PinnedPackage._install` to pathlib

* lowercase as per PR feedback

* update tests after removing version arg

goes along with 8787ba41af

Co-authored-by: Emily Rockman <ebuschang@gmail.com>
2022-12-07 15:48:17 -06:00
Jeremy Cohen
9c91f3a7bd Adjust tox passenv to be multiline (#6405) 2022-12-07 22:47:51 +01:00
Gerda Shank
1b6fed2ffd CT 1604 remove compiled classes (#6384)
* removed Compiled versions of nodes

* Remove compiled fields from dictionary if not compiled

* check compiled is False instead of attribute existence in env_var
processing

* Update artifacts test (CompiledSnapshotNode did not have SnapshotConfig)

* Changie

* more complicated 'compiling' check in env_var

* Update test_exit_codes.py
2022-12-07 15:21:05 -05:00
dependabot[bot]
0721f2c1b7 Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core (#6375)
* Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core

Bumps [mashumaro[msgpack]](https://github.com/Fatal1ty/mashumaro) from 3.1.1 to 3.2.
- [Release notes](https://github.com/Fatal1ty/mashumaro/releases)
- [Commits](https://github.com/Fatal1ty/mashumaro/compare/v3.1.1...v3.2)

---
updated-dependencies:
- dependency-name: mashumaro[msgpack]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-12-07 10:16:31 -05:00
Doug Beatty
b9a35da118 Fix intermittent database connection failure in Windows CI test (#6395)
* Fix intermittent database connection failure in Windows CI test

* Changelog entry
2022-12-06 18:34:39 -07:00
Peter Webb
60f80056b1 CT-1405: Refactor event logging code (#6291)
* CT-1405: Refactor event logging code

* CT-1405: Add changelog entry

* CT-1405: Add code to protect against using closed streams from past tests.

* CT-1405: Restore unit test which was only failing locally

* CT-1405: Document a hack with issue # to resolve it in the future

* CT-1405: Make black happy

* CT-1405: Get rid of confusing factory function and duplicated function

* CT-1405: Remove unused event from types.proto and auto-gen'd file
2022-12-06 15:51:52 -05:00
Stu Kilgore
540c3b79aa Prevent docs gen workflow on forks (#6390) 2022-12-06 11:14:20 -06:00
Stu Kilgore
77be2e4fdf Pin ubuntu at 20.04 for some workflows (#6380) 2022-12-05 10:55:36 -06:00
Gerda Shank
16f529e1d4 CT 1477 enrich logging events with data similar to legacy logger (#6325) 2022-12-02 19:29:25 -05:00
Doug Beatty
ebfcf2a9ef Update core/dbt/README.md to match current (#6371)
* Update core/dbt/README.md to match current

Add missing files/folders and alphabetize

* Changelog entry
2022-12-02 15:45:53 -07:00
Alexander Butler
67a8138b65 [fix] Fix the partial parse write path (#6081)
* Fix the partial parse path

Partial parse should use project root or it does not resolve to correct path. 
Eg. `target-path: ../some/dir/target`, if not ran from root, creates an erroneous folder.

* Run pre-commit

* Changie

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-12-02 17:23:37 -05:00
Michelle Ark
e91863de59 Set Flags from UserConfig (#6266)
flags with user config, flags.WHICH from invoked_subcommand if available
2022-12-02 15:23:58 -05:00
Michelle Ark
44b457c191 DepsTask with Click (#6260)
* deps with click, default --vars param, PartialProject.render_package_metadata
2022-11-22 17:52:59 -05:00
Chenyu Li
a0ec0b6f9d Project working with Click (#6142)
Co-authored-by: MichelleArk <michelle.ark@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-11-15 15:54:05 -08:00
Ian Knox
1ec54abdc4 Logging works with Click (#6088) 2022-10-25 14:14:08 -05:00
Ian Knox
5efc4aa066 Fix CI issue with tox (#6137) 2022-10-25 13:32:15 -05:00
Chenyu Li
847c0b9644 Tracking works with Click (#5972)
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2022-10-25 10:41:57 -07:00
659 changed files with 23366 additions and 35393 deletions

View File

@@ -1,13 +1,15 @@
[bumpversion]
current_version = 1.4.0a1
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
((?P<prekind>a|b|rc)
(?P<pre>\d+) # pre-release version num
)?
current_version = 1.5.0a1
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
(((?P<prekind>a|b|rc) # optional pre-release type
?(?P<num>[\d]+?)) # optional pre-release version number
\.?(?P<nightly>[a-z0-9]+)? # optional nightly release indicator
)? # expected matches: `1.5.0`, `1.5.0a1`, `1.5.0a1.dev123457`, expected failures: `1`, `1.5`, `1.5.2-a1`, `text1.5.0`
serialize =
{major}.{minor}.{patch}{prekind}{pre}
{major}.{minor}.{patch}{prekind}{num}.{nightly}
{major}.{minor}.{patch}{prekind}{num}
{major}.{minor}.{patch}
commit = False
tag = False
@@ -21,9 +23,11 @@ values =
rc
final
[bumpversion:part:pre]
[bumpversion:part:num]
first_value = 1
[bumpversion:part:nightly]
[bumpversion:file:core/setup.py]
[bumpversion:file:core/dbt/version.py]

View File

@@ -3,6 +3,7 @@
For information on prior major and minor releases, see their changelogs:
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)

View File

@@ -0,0 +1,6 @@
kind: "Dependencies"
body: "Bump ubuntu from 22.04 to 23.04"
time: 2023-02-06T00:09:26.00000Z
custom:
Author: dependabot[bot]
PR: 6865

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Update pathspec requirement from ~=0.9.0 to >=0.9,<0.11 in /core"
time: 2022-09-23T00:06:46.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 5917

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Bump black from 22.8.0 to 22.10.0"
time: 2022-10-07T00:08:48.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 6019

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Bump mashumaro[msgpack] from 3.0.4 to 3.1.1 in /core"
time: 2022-10-20T00:07:53.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 6108

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Update colorama requirement from <0.4.6,>=0.3.9 to >=0.3.9,<0.4.7 in /core"
time: 2022-10-26T00:09:10.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 6144

View File

@@ -1,7 +0,0 @@
kind: Docs
body: minor doc correction
time: 2022-09-08T15:41:57.689162-04:00
custom:
Author: andy-clapson
Issue: "5791"
PR: "5684"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Generate API docs for new CLI interface
time: 2022-10-07T09:06:56.446078-05:00
custom:
Author: stu-k
Issue: "5528"
PR: "6022"

View File

@@ -1,6 +0,0 @@
kind: Docs
time: 2022-10-17T17:14:11.715348-05:00
custom:
Author: paulbenschmidt
Issue: "5880"
PR: "324"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Fix rendering of sample code for metrics
time: 2022-11-16T15:57:43.204201+01:00
custom:
Author: jtcohen6
Issue: "323"
PR: "346"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: update link to installation instructions
time: 2023-02-07T12:38:07.336783-05:00
custom:
Author: ryancharris
Issue: None

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Fix JSON path to overview docs
time: 2023-02-09T08:29:01.432616-07:00
custom:
Author: halvorlu
Issue: "366"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Added favor-state flag to optionally favor state nodes even if unselected node
exists
time: 2022-04-08T16:54:59.696564+01:00
custom:
Author: daniel-murray josephberni
Issue: "2968"
PR: "5859"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Proto logging messages
time: 2022-08-17T15:48:57.225267-04:00
custom:
Author: gshank
Issue: "5610"
PR: "5643"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Friendlier error messages when packages.yml is malformed
time: 2022-09-12T12:59:35.121188+01:00
custom:
Author: jared-rimmer
Issue: "5486"
PR: "5812"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Migrate dbt-utils current_timestamp macros into core + adapters
time: 2022-09-14T09:56:25.97818-07:00
custom:
Author: colin-rogers-dbt
Issue: "5521"
PR: "5838"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Allow partitions in external tables to be supplied as a list
time: 2022-09-25T21:16:51.051239654+02:00
custom:
Author: pgoslatara
Issue: "5929"
PR: "5930"

View File

@@ -1,7 +0,0 @@
kind: Features
body: extend -f flag shorthand for seed command
time: 2022-10-03T11:07:05.381632-05:00
custom:
Author: dave-connors-3
Issue: "5990"
PR: "5991"

View File

@@ -1,8 +0,0 @@
kind: Features
body: This pulls the profile name from args when constructing a RuntimeConfig in lib.py,
enabling the dbt-server to override the value that's in the dbt_project.yml
time: 2022-11-02T15:00:03.000805-05:00
custom:
Author: racheldaniel
Issue: "6201"
PR: "6202"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Added an md5 function to the base context
time: 2022-11-14T18:52:07.788593+02:00
custom:
Author: haritamar
Issue: "6246"
PR: "6247"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Exposures support metrics in lineage
time: 2022-11-30T11:29:13.256034-05:00
custom:
Author: michelleark
Issue: "6057"
PR: "6342"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Have dbt debug spit out structured json logs with flags enabled.
time: 2023-01-07T00:31:57.516063-08:00
custom:
Author: versusfacit
Issue: "5353"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add adapter_response to dbt test and freshness result
time: 2023-01-18T23:38:01.857342+08:00
custom:
Author: aezomz
Issue: "2964"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Improve error message for packages missing `dbt_project.yml`
time: 2023-01-20T11:29:21.509967-07:00
custom:
Author: dbeatty10
Issue: "6663"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Adjust makefile to have clearer instructions for CI env var changes.
time: 2023-01-26T15:47:16.887327-08:00
custom:
Author: versusfacit
Issue: "6689"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Stand-alone Python module for PostgresColumn
time: 2023-01-27T16:28:12.212427-08:00
custom:
Author: nssalian
Issue: "6772"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Exposure owner requires one of name or email keys, and accepts additional arbitrary
keys
time: 2023-02-09T09:20:59.300272-05:00
custom:
Author: michelleark
Issue: "6833"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Account for disabled flags on models in schema files more completely
time: 2022-09-16T10:48:54.162273-05:00
custom:
Author: emmyoop
Issue: "3992"
PR: "5868"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add validation of enabled config for metrics, exposures and sources
time: 2022-10-10T11:32:18.752322-05:00
custom:
Author: emmyoop
Issue: "6030"
PR: "6038"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: check length of args of python model function before accessing it
time: 2022-10-11T16:07:15.464093-04:00
custom:
Author: chamini2
Issue: "6041"
PR: "6042"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Add functors to ensure event types with str-type attributes are initialized
to spec, even when provided non-str type params.
time: 2022-10-16T17:37:42.846683-07:00
custom:
Author: versusfacit
Issue: "5436"
PR: "5874"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Allow hooks to fail without halting execution flow
time: 2022-11-07T09:53:14.340257-06:00
custom:
Author: ChenyuLInx
Issue: "5625"
PR: "6059"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Clarify Error Message for how many models are allowed in a Python file
time: 2022-11-15T08:10:21.527884-05:00
custom:
Author: justbldwn
Issue: "6245"
PR: "6251"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: add merge_exclude_columns adapter tests
time: 2023-01-23T13:28:14.808748-06:00
custom:
Author: dave-connors-3
Issue: "6699"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Include adapter_response in NodeFinished run_result log event
time: 2023-01-24T11:58:37.74179-05:00
custom:
Author: gshank
Issue: "6703"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Sort cli vars before hashing for partial parsing
time: 2023-01-24T14:19:43.333628-05:00
custom:
Author: gshank
Issue: "6710"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: '[Regression] exposure_content referenced incorrectly'
time: 2023-01-25T19:17:39.942081-05:00
custom:
Author: Mathyoub
Issue: "6738"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Remove pin on packaging and stop using it for prerelease comparisons
time: 2023-02-01T15:44:18.279158-05:00
custom:
Author: gshank
Issue: "6834"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Readd depends_on.macros to SeedNode, to support seeds with hooks calling macros
time: 2023-02-03T13:55:57.853715+01:00
custom:
Author: jtcohen6
Issue: "6806"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix regression of --quiet cli parameter behavior
time: 2023-02-07T14:35:44.160163-05:00
custom:
Author: peterallenwebb
Issue: "6749"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Ensure results from hooks contain nodes when processing them
time: 2023-02-08T11:05:51.952494-06:00
custom:
Author: emmyoop
Issue: "6796"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Always flush stdout after logging
time: 2023-02-08T15:49:35.175874-05:00
custom:
Author: peterallenwebb
Issue: "6901"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Reapply logging fixes which were accidentally reverted
time: 2023-02-10T10:30:28.179997-05:00
custom:
Author: peterallenwebb
Issue: "6936"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: hoist dbt.cli.main into dbt.main namespace
time: 2023-02-13T11:28:22.809596-08:00
custom:
Author: aranke
Issue: ''

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Readd initialization events, --log-cache-events in new CLI
time: 2023-02-13T13:05:22.989477+01:00
custom:
Author: jtcohen6
Issue: "6933"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix previous state tests and disabled exposures, metrics
time: 2023-02-13T17:07:23.185679-05:00
custom:
Author: gshank
Issue: 6752 6753

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Put black config in explicit config
time: 2022-09-27T19:42:59.241433-07:00
custom:
Author: max-sixty
Issue: "5946"
PR: "5947"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Added flat_graph attribute the Manifest class's deepcopy() coverage
time: 2022-09-29T13:44:06.275941-04:00
custom:
Author: peterallenwebb
Issue: "5809"
PR: "5975"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add mypy configs so `mypy` passes from CLI
time: 2022-10-05T12:03:10.061263-07:00
custom:
Author: max-sixty
Issue: "5983"
PR: "5983"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Exception message cleanup.
time: 2022-10-07T09:46:27.682872-05:00
custom:
Author: emmyoop
Issue: "6023"
PR: "6024"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add dmypy cache to gitignore
time: 2022-10-07T14:00:44.227644-07:00
custom:
Author: max-sixty
Issue: "6028"
PR: "5978"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Provide useful errors when the value of 'materialized' is invalid
time: 2022-10-13T18:19:12.167548-04:00
custom:
Author: peterallenwebb
Issue: "5229"
PR: "6025"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fixed extra whitespace in strings introduced by black.
time: 2022-10-17T15:15:11.499246-05:00
custom:
Author: luke-bassett
Issue: "1350"
PR: "6086"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Clean up string formatting
time: 2022-10-17T15:58:44.676549-04:00
custom:
Author: eve-johns
Issue: "6068"
PR: "6082"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Remove the 'root_path' field from most nodes
time: 2022-10-28T10:48:37.687886-04:00
custom:
Author: gshank
Issue: "6171"
PR: "6172"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Combine certain logging events with different levels
time: 2022-10-28T11:03:44.887836-04:00
custom:
Author: gshank
Issue: "6173"
PR: "6174"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert threading tests to pytest
time: 2022-11-08T07:45:50.589147-06:00
custom:
Author: stu-k
Issue: "5942"
PR: "6226"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert postgres index tests to pytest
time: 2022-11-08T11:56:33.743042-06:00
custom:
Author: stu-k
Issue: "5770"
PR: "6228"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert use color tests to pytest
time: 2022-11-08T13:31:04.788547-06:00
custom:
Author: stu-k
Issue: "5771"
PR: "6230"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add github actions workflow to generate high level CLI API docs
time: 2022-11-16T13:00:37.916202-06:00
custom:
Author: stu-k
Issue: "5942"
PR: "6187"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-921] dbt compile works in click'
time: 2023-01-11T14:51:43.324107-08:00
custom:
Author: aranke
Issue: "5545"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix use of ConnectionReused logging event
time: 2023-01-13T13:25:13.023168-05:00
custom:
Author: gshank
Issue: "6168"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Port docs tests to pytest
time: 2023-01-13T15:07:00.477038-05:00
custom:
Author: peterallenwebb
Issue: "6573"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update deprecated github action command
time: 2023-01-17T11:17:37.046095-06:00
custom:
Author: davidbloss
Issue: "6153"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt snapshot works in click
time: 2023-01-17T16:25:05.973769-08:00
custom:
Author: ChenyuLInx
Issue: "5554"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt list working with click
time: 2023-01-17T21:37:29.91632-05:00
custom:
Author: michelleark
Issue: "5549"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add dbt run-operation to click CLI
time: 2023-01-19T10:53:04.154871+01:00
custom:
Author: jtcohen6
Issue: "5552"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt build working with new click framework
time: 2023-01-19T20:56:50.50549-05:00
custom:
Author: michelleark
Issue: "5541"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt docs generate works with new click framework
time: 2023-01-19T21:10:40.698851-05:00
custom:
Author: michelleark
Issue: "5543"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Replaced the EmptyLine event with a more general Formatting event, and added
a Note event.
time: 2023-01-20T17:22:54.45828-05:00
custom:
Author: peterallenwebb
Issue: "6481"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Small optimization on manifest parsing benefitting large DAGs
time: 2023-01-22T21:52:35.549814+01:00
custom:
Author: boxysean
Issue: "6697"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Revised and simplified various structured logging events
time: 2023-01-24T15:35:53.065356-05:00
custom:
Author: peterallenwebb
Issue: 6664 6665 6666

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt init works with click
time: 2023-01-24T17:51:10.74065-05:00
custom:
Author: michelleark
Issue: "5548"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs commands'
time: 2023-01-25T04:11:36.57506-08:00
custom:
Author: aranke
Issue: 5544 6722

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Migrate debug task to click
time: 2023-01-25T10:26:06.735994-06:00
custom:
Author: stu-k
Issue: "5546"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: ' Optimized GraphQueue to remove graph analysis bottleneck in large dags.'
time: 2023-01-26T13:59:39.518345-05:00
custom:
Author: peterallenwebb
Issue: "6759"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Implement --version for click cli
time: 2023-01-26T14:31:02.740282-06:00
custom:
Author: stu-k
Issue: "6757"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-1841] Convert custom target test to Pytest'
time: 2023-01-26T16:47:41.198714-08:00
custom:
Author: aranke
Issue: "6638"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: "Enables the new Click Cli on the commandline! \U0001F680"
time: 2023-01-30T17:57:52.65626-06:00
custom:
Author: iknox-fa
Issue: "6784"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: warn_error/warn_error_options mutual exclusivity in click
time: 2023-01-30T18:09:17.240662-05:00
custom:
Author: michelleark
Issue: "6579"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Lazily call --version
time: 2023-01-31T14:18:06.02312-06:00
custom:
Author: stu-k
Issue: "6812"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Moving simple_seed to adapter zone to help adapter test conversions
time: 2023-02-03T14:35:51.481856-08:00
custom:
Author: nssalian
Issue: CT-1959

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: flags.THREADS defaults to None
time: 2023-02-07T16:51:11.011984-05:00
custom:
Author: michelleark
Issue: "6887"

View File

@@ -4,21 +4,70 @@ headerPath: header.tpl.md
versionHeaderPath: ""
changelogPath: CHANGELOG.md
versionExt: md
envPrefix: "CHANGIE_"
versionFormat: '## dbt-core {{.Version}} - {{.Time.Format "January 02, 2006"}}'
kindFormat: '### {{.Kind}}'
changeFormat: '- {{.Body}} ([#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), [#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
kinds:
- label: Breaking Changes
- label: Features
- label: Fixes
- label: Docs
changeFormat: '- {{.Body}} ([dbt-docs/#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-docs/issues/{{.Custom.Issue}}), [dbt-docs/#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-docs/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[dbt-docs/#nbr](https://github.com/dbt-labs/dbt-docs/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: '- {{.Body}} ({{if ne .Custom.Issue ""}}[#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), {{end}}[#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: '- {{.Body}} ({{if ne .Custom.Issue ""}}[#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), {{end}}[#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
@@ -33,42 +82,53 @@ custom:
type: string
minLength: 3
- key: Issue
label: GitHub Issue Number
type: int
minInt: 1
- key: PR
label: GitHub Pull Request Number
type: int
minInt: 1
label: GitHub Issue Number (separated by a single space if multiple)
type: string
minLength: 1
footerFormat: |
{{- $contributorDict := dict }}
{{- /* any names added to this list should be all lowercase for later matching purposes */}}
{{- $core_team := list "michelleark" "peterallenwebb" "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "dependabot[bot]" "snyk-bot" "colin-rogers-dbt" }}
{{- /* ensure all names in this list are all lowercase for later matching purposes */}}
{{- $core_team := splitList " " .Env.CORE_TEAM }}
{{- /* ensure we always skip snyk and dependabot in addition to the core team */}}
{{- $maintainers := list "dependabot[bot]" "snyk-bot"}}
{{- range $team_member := $core_team }}
{{- $team_member_lower := lower $team_member }}
{{- $maintainers = append $maintainers $team_member_lower }}
{{- end }}
{{- range $change := .Changes }}
{{- $authorList := splitList " " $change.Custom.Author }}
{{- /* loop through all authors for a PR */}}
{{- /* loop through all authors for a single changelog */}}
{{- range $author := $authorList }}
{{- $authorLower := lower $author }}
{{- /* we only want to include non-core team contributors */}}
{{- if not (has $authorLower $core_team)}}
{{- /* Docs kind link back to dbt-docs instead of dbt-core PRs */}}
{{- $prLink := $change.Kind }}
{{- if eq $change.Kind "Docs" }}
{{- $prLink = "[dbt-docs/#pr](https://github.com/dbt-labs/dbt-docs/pull/pr)" | replace "pr" $change.Custom.PR }}
{{- else }}
{{- $prLink = "[#pr](https://github.com/dbt-labs/dbt-core/pull/pr)" | replace "pr" $change.Custom.PR }}
{{- end }}
{{- /* check if this contributor has other PRs associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $prList := get $contributorDict $author }}
{{- $prList = append $prList $prLink }}
{{- $contributorDict := set $contributorDict $author $prList }}
{{- else }}
{{- $prList := list $prLink }}
{{- $contributorDict := set $contributorDict $author $prList }}
{{- end }}
{{- end}}
{{- if not (has $authorLower $maintainers)}}
{{- $changeList := splitList " " $change.Custom.Author }}
{{- $IssueList := list }}
{{- $changeLink := $change.Kind }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }}
{{- $changes := splitList " " $change.Custom.PR }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- else }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- end }}
{{- /* check if this contributor has other changes associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $contributionList := get $contributorDict $author }}
{{- $contributionList = concat $contributionList $IssueList }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- else }}
{{- $contributionList := $IssueList }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- end }}
{{- end}}
{{- end}}
{{- end }}
{{- /* no indentation here for formatting so the final markdown doesn't have unneeded indentations */}}

View File

@@ -9,4 +9,4 @@ ignore =
E203 # makes Flake8 work like black
E741
E501 # long line checking is done in black
exclude = test
exclude = test/

2
.gitattributes vendored Normal file
View File

@@ -0,0 +1,2 @@
core/dbt/include/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary

8
.github/_README.md vendored
View File

@@ -138,7 +138,7 @@ Some triggers of note that we use:
id: fp
run: |
FILEPATH=.changes/unreleased/Dependencies-${{ steps.filename_time.outputs.time }}.yaml
echo "::set-output name=FILEPATH::$FILEPATH"
echo "FILEPATH=$FILEPATH" >> $GITHUB_OUTPUT
```
- Print out all variables you will reference as the first step of a job. This allows for easier debugging. The first job should log all inputs. Subsequent jobs should reference outputs of other jobs, if present.
@@ -163,9 +163,9 @@ Some triggers of note that we use:
- name: Set Variables
id: variables
run: |
echo "::set-output name=important_path::'performance/runner/Cargo.toml'"
echo "::set-output name=release_id::${{github.event.inputs.release_id}}"
echo "::set-output name=open_prs::${{github.event.inputs.open_prs}}"
echo "important_path='performance/runner/Cargo.toml'" >> $GITHUB_OUTPUT
echo "release_id=${{github.event.inputs.release_id}}" >> $GITHUB_OUTPUT
echo "open_prs=${{github.event.inputs.open_prs}}" >> $GITHUB_OUTPUT
job2:
needs: [job1]

View File

@@ -28,11 +28,12 @@ if __name__ == "__main__":
if package_request.status_code == 404:
if halt_on_missing:
sys.exit(1)
else:
# everything is the latest if the package doesn't exist
print(f"::set-output name=latest::{True}")
print(f"::set-output name=minor_latest::{True}")
sys.exit(0)
# everything is the latest if the package doesn't exist
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write("latest=True")
gh_output.write("minor_latest=True")
sys.exit(0)
# TODO: verify package meta is "correct"
# https://github.com/dbt-labs/dbt-core/issues/4640
@@ -91,5 +92,7 @@ if __name__ == "__main__":
latest = is_latest(pre_rel, new_version, current_latest)
minor_latest = is_latest(pre_rel, new_version, current_minor_latest)
print(f"::set-output name=latest::{latest}")
print(f"::set-output name=minor_latest::{minor_latest}")
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"latest={latest}")
gh_output.write(f"minor_latest={minor_latest}")

View File

@@ -40,7 +40,7 @@ jobs:
matrix:
include:
- label: "dependencies"
changie_kind: "Dependency"
changie_kind: "Dependencies"
- label: "snyk"
changie_kind: "Security"
runs-on: ubuntu-latest
@@ -58,4 +58,4 @@ jobs:
commit_message: "Add automated changelog yaml from template for bot PR"
changie_kind: ${{ matrix.changie_kind }}
label: ${{ matrix.label }}
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n Issue: 4904\n PR: ${{ github.event.pull_request.number }}"
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n PR: ${{ github.event.pull_request.number }}"

View File

@@ -0,0 +1,42 @@
# **what?**
# Cuts a new `*.latest` branch
# Also cleans up all files in `.changes/unreleased` and `.changes/previous verion on
# `main` and bumps `main` to the input version.
# **why?**
# Generally reduces the workload of engineers and reduces error. Allow automation.
# **when?**
# This will run when called manually.
name: Cut new release branch
on:
workflow_dispatch:
inputs:
version_to_bump_main:
description: 'The alpha version main should bump to (ex. 1.6.0a1)'
required: true
new_branch_name:
description: 'The full name of the new branch (ex. 1.5.latest)'
required: true
defaults:
run:
shell: bash
permissions:
contents: write
jobs:
cut_branch:
name: "Cut branch and clean up main for dbt-core"
# TODO: update to point at `main` once testing is complete
uses: dbt-labs/actions/.github/workflows/cut-release-branch.yaml@er/ct-495-cleanup
with:
version_to_bump_main: ${{ inputs.version_to_bump_main }}
new_branch_name: ${{ inputs.new_branch_name }}
PR_title: "Cleanup main after cutting new ${{ inputs.new_branch_name }} branch"
PR_body: "All adapter PRs will fail CI until the dbt-core PR has been merged due to release version conflicts."
secrets:
FISHTOWN_BOT_PAT: ${{ secrets.FISHTOWN_BOT_PAT }}

View File

@@ -34,6 +34,7 @@ jobs:
check_gen:
name: check if generation needed
runs-on: ubuntu-latest
if: ${{ github.event.pull_request.head.repo.fork == false }}
outputs:
cli_dir_changed: ${{ steps.check_cli.outputs.cli_dir_changed }}
docs_dir_changed: ${{ steps.check_docs.outputs.docs_dir_changed }}
@@ -44,8 +45,6 @@ jobs:
echo "env.CLI_DIR: ${{ env.CLI_DIR }}"
echo "env.DOCS_BUILD_DIR: ${{ env.DOCS_BUILD_DIR }}"
echo "env.DOCS_DIR: ${{ env.DOCS_DIR }}"
echo ">>>>> git log"
git log --pretty=oneline | head -5
- name: git checkout
uses: actions/checkout@v3

View File

@@ -73,7 +73,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
env:
TOXENV: "unit"
@@ -101,7 +101,9 @@ jobs:
- name: Get current date
if: always()
id: date
run: echo "::set-output name=date::$(date +'%Y-%m-%dT%H_%M_%S')" #no colons allowed for artifacts
run: |
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v2
if: always()
@@ -118,7 +120,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-20.04]
include:
- python-version: 3.8
@@ -168,7 +170,9 @@ jobs:
- name: Get current date
if: always()
id: date
run: echo "::set-output name=date::$(date +'%Y_%m_%dT%H_%M_%S')" #no colons allowed for artifacts
run: |
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v2
if: always()

109
.github/workflows/nightly-release.yml vendored Normal file
View File

@@ -0,0 +1,109 @@
# **what?**
# Nightly releases to GitHub and PyPI. This workflow produces the following outcome:
# - generate and validate data for night release (commit SHA, version number, release branch);
# - pass data to release workflow;
# - night release will be pushed to GitHub as a draft release;
# - night build will be pushed to test PyPI;
#
# **why?**
# Ensure an automated and tested release process for nightly builds
#
# **when?**
# This workflow runs on schedule or can be run manually on demand.
name: Nightly Test Release to GitHub and PyPI
on:
workflow_dispatch: # for manual triggering
schedule:
- cron: 0 9 * * *
permissions:
contents: write # this is the permission that allows creating a new release
defaults:
run:
shell: bash
env:
RELEASE_BRANCH: "main"
jobs:
aggregate-release-data:
runs-on: ubuntu-latest
outputs:
commit_sha: ${{ steps.resolve-commit-sha.outputs.release_commit }}
version_number: ${{ steps.nightly-release-version.outputs.number }}
release_branch: ${{ steps.release-branch.outputs.name }}
steps:
- name: "Checkout ${{ github.repository }} Branch ${{ env.RELEASE_BRANCH }}"
uses: actions/checkout@v3
with:
ref: ${{ env.RELEASE_BRANCH }}
- name: "Resolve Commit To Release"
id: resolve-commit-sha
run: |
commit_sha=$(git rev-parse HEAD)
echo "release_commit=$commit_sha" >> $GITHUB_OUTPUT
- name: "Get Current Version Number"
id: version-number-sources
run: |
current_version=`awk -F"current_version = " '{print $2}' .bumpversion.cfg | tr '\n' ' '`
echo "current_version=$current_version" >> $GITHUB_OUTPUT
- name: "Audit Version And Parse Into Parts"
id: semver
uses: dbt-labs/actions/parse-semver@v1.1.0
with:
version: ${{ steps.version-number-sources.outputs.current_version }}
- name: "Get Current Date"
id: current-date
run: echo "date=$(date +'%m%d%Y')" >> $GITHUB_OUTPUT
- name: "Generate Nightly Release Version Number"
id: nightly-release-version
run: |
number="${{ steps.semver.outputs.version }}.dev${{ steps.current-date.outputs.date }}"
echo "number=$number" >> $GITHUB_OUTPUT
- name: "Audit Nightly Release Version And Parse Into Parts"
uses: dbt-labs/actions/parse-semver@v1.1.0
with:
version: ${{ steps.nightly-release-version.outputs.number }}
- name: "Set Release Branch"
id: release-branch
run: |
echo "name=${{ env.RELEASE_BRANCH }}" >> $GITHUB_OUTPUT
log-outputs-aggregate-release-data:
runs-on: ubuntu-latest
needs: [aggregate-release-data]
steps:
- name: "[DEBUG] Log Outputs"
run: |
echo commit_sha : ${{ needs.aggregate-release-data.outputs.commit_sha }}
echo version_number: ${{ needs.aggregate-release-data.outputs.version_number }}
echo release_branch: ${{ needs.aggregate-release-data.outputs.release_branch }}
release-github-pypi:
needs: [aggregate-release-data]
uses: ./.github/workflows/release.yml
with:
sha: ${{ needs.aggregate-release-data.outputs.commit_sha }}
target_branch: ${{ needs.aggregate-release-data.outputs.release_branch }}
version_number: ${{ needs.aggregate-release-data.outputs.version_number }}
build_script_path: "scripts/build-dist.sh"
env_setup_script_path: "scripts/env-setup.sh"
s3_bucket_name: "core-team-artifacts"
package_test_command: "dbt --version"
test_run: true
nightly_release: true
secrets: inherit

View File

@@ -28,7 +28,33 @@ on:
permissions: read-all
jobs:
fetch-latest-branches:
runs-on: ubuntu-latest
outputs:
latest-branches: ${{ steps.get-latest-branches.outputs.repo-branches }}
steps:
- name: "Fetch dbt-core Latest Branches"
uses: dbt-labs/actions/fetch-repo-branches@v1.1.1
id: get-latest-branches
with:
repo_name: ${{ github.event.repository.name }}
organization: "dbt-labs"
pat: ${{ secrets.GITHUB_TOKEN }}
fetch_protected_branches_only: true
regex: "^1.[0-9]+.latest$"
perform_match_method: "match"
retries: 3
- name: "[ANNOTATION] ${{ github.event.repository.name }} - branches to test"
run: |
title="${{ github.event.repository.name }} - branches to test"
message="The workflow will run tests for the following branches of the ${{ github.event.repository.name }} repo: ${{ steps.get-latest-branches.outputs.repo-branches }}"
echo "::notice $title::$message"
kick-off-ci:
needs: [fetch-latest-branches]
name: Kick-off CI
runs-on: ubuntu-latest
@@ -39,7 +65,9 @@ jobs:
max-parallel: 1
fail-fast: false
matrix:
branch: [1.0.latest, 1.1.latest, 1.2.latest, 1.3.latest, main]
branch: ${{ fromJSON(needs.fetch-latest-branches.outputs.latest-branches) }}
include:
- branch: 'main'
steps:
- name: Call CI workflow for ${{ matrix.branch }} branch

View File

@@ -41,9 +41,9 @@ jobs:
id: version
run: |
IFS="." read -r MAJOR MINOR PATCH <<< ${{ github.event.inputs.version_number }}
echo "::set-output name=major::$MAJOR"
echo "::set-output name=minor::$MINOR"
echo "::set-output name=patch::$PATCH"
echo "major=$MAJOR" >> $GITHUB_OUTPUT
echo "minor=$MINOR" >> $GITHUB_OUTPUT
echo "patch=$PATCH" >> $GITHUB_OUTPUT
- name: Is pkg 'latest'
id: latest
@@ -70,8 +70,10 @@ jobs:
- name: Get docker build arg
id: build_arg
run: |
echo "::set-output name=build_arg_name::"$(echo ${{ github.event.inputs.package }} | sed 's/\-/_/g')
echo "::set-output name=build_arg_value::"$(echo ${{ github.event.inputs.package }} | sed 's/postgres/core/g')
BUILD_ARG_NAME=$(echo ${{ github.event.inputs.package }} | sed 's/\-/_/g')
BUILD_ARG_VALUE=$(echo ${{ github.event.inputs.package }} | sed 's/postgres/core/g')
echo "build_arg_name=$BUILD_ARG_NAME" >> $GITHUB_OUTPUT
echo "build_arg_value=$BUILD_ARG_VALUE" >> $GITHUB_OUTPUT
- name: Log in to the GHCR
uses: docker/login-action@v1

View File

@@ -1,24 +1,110 @@
# **what?**
# Take the given commit, run unit tests specifically on that sha, build and
# package it, and then release to GitHub and PyPi with that specific build
# Release workflow provides the following steps:
# - checkout the given commit;
# - validate version in sources and changelog file for given version;
# - bump the version and generate a changelog if needed;
# - merge all changes to the target branch if needed;
# - run unit and integration tests against given commit;
# - build and package that SHA;
# - release it to GitHub and PyPI with that specific build;
#
# **why?**
# Ensure an automated and tested release process
#
# **when?**
# This will only run manually with a given sha and version
# This workflow can be run manually on demand or can be called by other workflows
name: Release to GitHub and PyPi
name: Release to GitHub and PyPI
on:
workflow_dispatch:
inputs:
sha:
description: 'The last commit sha in the release'
required: true
description: "The last commit sha in the release"
type: string
required: true
target_branch:
description: "The branch to release from"
type: string
required: true
version_number:
description: 'The release version number (i.e. 1.0.0b1)'
required: true
description: "The release version number (i.e. 1.0.0b1)"
type: string
required: true
build_script_path:
description: "Build script path"
type: string
default: "scripts/build-dist.sh"
required: true
env_setup_script_path:
description: "Environment setup script path"
type: string
default: "scripts/env-setup.sh"
required: false
s3_bucket_name:
description: "AWS S3 bucket name"
type: string
default: "core-team-artifacts"
required: true
package_test_command:
description: "Package test command"
type: string
default: "dbt --version"
required: true
test_run:
description: "Test run (Publish release as draft)"
type: boolean
default: true
required: false
nightly_release:
description: "Nightly release to dev environment"
type: boolean
default: false
required: false
workflow_call:
inputs:
sha:
description: "The last commit sha in the release"
type: string
required: true
target_branch:
description: "The branch to release from"
type: string
required: true
version_number:
description: "The release version number (i.e. 1.0.0b1)"
type: string
required: true
build_script_path:
description: "Build script path"
type: string
default: "scripts/build-dist.sh"
required: true
env_setup_script_path:
description: "Environment setup script path"
type: string
default: "scripts/env-setup.sh"
required: false
s3_bucket_name:
description: "AWS S3 bucket name"
type: string
default: "core-team-artifacts"
required: true
package_test_command:
description: "Package test command"
type: string
default: "dbt --version"
required: true
test_run:
description: "Test run (Publish release as draft)"
type: boolean
default: true
required: false
nightly_release:
description: "Nightly release to dev environment"
type: boolean
default: false
required: false
permissions:
contents: write # this is the permission that allows creating a new release
@@ -28,175 +114,116 @@ defaults:
shell: bash
jobs:
unit:
name: Unit test
log-inputs:
name: Log Inputs
runs-on: ubuntu-latest
env:
TOXENV: "unit"
steps:
- name: Check out the repository
uses: actions/checkout@v2
with:
persist-credentials: false
ref: ${{ github.event.inputs.sha }}
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
- name: "[DEBUG] Print Variables"
run: |
pip install --user --upgrade pip
pip install tox
pip --version
tox --version
echo The last commit sha in the release: ${{ inputs.sha }}
echo The branch to release from: ${{ inputs.target_branch }}
echo The release version number: ${{ inputs.version_number }}
echo Build script path: ${{ inputs.build_script_path }}
echo Environment setup script path: ${{ inputs.env_setup_script_path }}
echo AWS S3 bucket name: ${{ inputs.s3_bucket_name }}
echo Package test command: ${{ inputs.package_test_command }}
echo Test run: ${{ inputs.test_run }}
echo Nightly release: ${{ inputs.nightly_release }}
- name: Run tox
run: tox
bump-version-generate-changelog:
name: Bump package version, Generate changelog
build:
name: build packages
uses: dbt-labs/dbt-release/.github/workflows/release-prep.yml@main
with:
sha: ${{ inputs.sha }}
version_number: ${{ inputs.version_number }}
target_branch: ${{ inputs.target_branch }}
env_setup_script_path: ${{ inputs.env_setup_script_path }}
test_run: ${{ inputs.test_run }}
nightly_release: ${{ inputs.nightly_release }}
secrets: inherit
log-outputs-bump-version-generate-changelog:
name: "[Log output] Bump package version, Generate changelog"
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog]
runs-on: ubuntu-latest
steps:
- name: Check out the repository
uses: actions/checkout@v2
with:
persist-credentials: false
ref: ${{ github.event.inputs.sha }}
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
- name: Print variables
run: |
pip install --user --upgrade pip
pip install --upgrade setuptools wheel twine check-wheel-contents
pip --version
echo Final SHA : ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
echo Changelog path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
- name: Build distributions
run: ./scripts/build-dist.sh
build-test-package:
name: Build, Test, Package
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog]
- name: Show distributions
run: ls -lh dist/
uses: dbt-labs/dbt-release/.github/workflows/build.yml@main
- name: Check distribution descriptions
run: |
twine check dist/*
with:
sha: ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
changelog_path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
build_script_path: ${{ inputs.build_script_path }}
s3_bucket_name: ${{ inputs.s3_bucket_name }}
package_test_command: ${{ inputs.package_test_command }}
test_run: ${{ inputs.test_run }}
nightly_release: ${{ inputs.nightly_release }}
- name: Check wheel contents
run: |
check-wheel-contents dist/*.whl --ignore W007,W008
- uses: actions/upload-artifact@v2
with:
name: dist
path: |
dist/
!dist/dbt-${{github.event.inputs.version_number}}.tar.gz
test-build:
name: verify packages
needs: [build, unit]
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
run: |
pip install --user --upgrade pip
pip install --upgrade wheel
pip --version
- uses: actions/download-artifact@v2
with:
name: dist
path: dist/
- name: Show distributions
run: ls -lh dist/
- name: Install wheel distributions
run: |
find ./dist/*.whl -maxdepth 1 -type f | xargs pip install --force-reinstall --find-links=dist/
- name: Check wheel distributions
run: |
dbt --version
- name: Install source distributions
run: |
find ./dist/*.gz -maxdepth 1 -type f | xargs pip install --force-reinstall --find-links=dist/
- name: Check source distributions
run: |
dbt --version
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
github-release:
name: GitHub Release
if: ${{ !failure() && !cancelled() }}
needs: test-build
needs: [bump-version-generate-changelog, build-test-package]
runs-on: ubuntu-latest
uses: dbt-labs/dbt-release/.github/workflows/github-release.yml@main
steps:
- uses: actions/download-artifact@v2
with:
name: dist
path: '.'
# Need to set an output variable because env variables can't be taken as input
# This is needed for the next step with releasing to GitHub
- name: Find release type
id: release_type
env:
IS_PRERELEASE: ${{ contains(github.event.inputs.version_number, 'rc') || contains(github.event.inputs.version_number, 'b') }}
run: |
echo ::set-output name=isPrerelease::$IS_PRERELEASE
- name: Creating GitHub Release
uses: softprops/action-gh-release@v1
with:
name: dbt-core v${{github.event.inputs.version_number}}
tag_name: v${{github.event.inputs.version_number}}
prerelease: ${{ steps.release_type.outputs.isPrerelease }}
target_commitish: ${{github.event.inputs.sha}}
body: |
[Release notes](https://github.com/dbt-labs/dbt-core/blob/main/CHANGELOG.md)
files: |
dbt_postgres-${{github.event.inputs.version_number}}-py3-none-any.whl
dbt_core-${{github.event.inputs.version_number}}-py3-none-any.whl
dbt-postgres-${{github.event.inputs.version_number}}.tar.gz
dbt-core-${{github.event.inputs.version_number}}.tar.gz
with:
sha: ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
changelog_path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
test_run: ${{ inputs.test_run }}
pypi-release:
name: Pypi release
name: PyPI Release
runs-on: ubuntu-latest
needs: [github-release]
needs: github-release
uses: dbt-labs/dbt-release/.github/workflows/pypi-release.yml@main
environment: PypiProd
steps:
- uses: actions/download-artifact@v2
with:
name: dist
path: 'dist'
with:
version_number: ${{ inputs.version_number }}
test_run: ${{ inputs.test_run }}
- name: Publish distribution to PyPI
uses: pypa/gh-action-pypi-publish@v1.4.2
with:
password: ${{ secrets.PYPI_API_TOKEN }}
secrets:
PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
TEST_PYPI_API_TOKEN: ${{ secrets.TEST_PYPI_API_TOKEN }}
slack-notification:
name: Slack Notification
if: ${{ failure() && (!inputs.test_run || inputs.nightly_release) }}
needs:
[
bump-version-generate-changelog,
build-test-package,
github-release,
pypi-release,
]
uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}

View File

@@ -20,106 +20,9 @@ on:
description: 'The version number to bump to (ex. 1.2.0, 1.3.0b1)'
required: true
permissions:
contents: write
pull-requests: write
jobs:
bump:
runs-on: ubuntu-latest
steps:
- name: "[DEBUG] Print Variables"
run: |
echo "all variables defined as inputs"
echo The version_number: ${{ github.event.inputs.version_number }}
- name: Check out the repository
uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: Install python dependencies
run: |
python3 -m venv env
source env/bin/activate
pip install --upgrade pip
- name: Add Homebrew to PATH
run: |
echo "/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin" >> $GITHUB_PATH
- name: Install Homebrew packages
run: |
brew install pre-commit
brew tap miniscruff/changie https://github.com/miniscruff/changie
brew install changie
- name: Audit Version and Parse Into Parts
id: semver
uses: dbt-labs/actions/parse-semver@v1
with:
version: ${{ github.event.inputs.version_number }}
- name: Set branch value
id: variables
run: |
echo "::set-output name=BRANCH_NAME::prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID"
- name: Create PR branch
run: |
git checkout -b ${{ steps.variables.outputs.BRANCH_NAME }}
git push origin ${{ steps.variables.outputs.BRANCH_NAME }}
git branch --set-upstream-to=origin/${{ steps.variables.outputs.BRANCH_NAME }} ${{ steps.variables.outputs.BRANCH_NAME }}
- name: Bump version
run: |
source env/bin/activate
pip install -r dev-requirements.txt
env/bin/bumpversion --allow-dirty --new-version ${{ github.event.inputs.version_number }} major
git status
- name: Run changie
run: |
if [[ ${{ steps.semver.outputs.is-pre-release }} -eq 1 ]]
then
changie batch ${{ steps.semver.outputs.base-version }} --move-dir '${{ steps.semver.outputs.base-version }}' --prerelease '${{ steps.semver.outputs.pre-release }}'
else
changie batch ${{ steps.semver.outputs.base-version }} --include '${{ steps.semver.outputs.base-version }}' --remove-prereleases
fi
changie merge
git status
# this step will fail on whitespace errors but also correct them
- name: Remove trailing whitespace
continue-on-error: true
run: |
pre-commit run trailing-whitespace --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
# this step will fail on newline errors but also correct them
- name: Removing extra newlines
continue-on-error: true
run: |
pre-commit run end-of-file-fixer --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
- name: Commit version bump to branch
uses: EndBug/add-and-commit@v7
with:
author_name: 'Github Build Bot'
author_email: 'buildbot@fishtownanalytics.com'
message: 'Bumping version to ${{ github.event.inputs.version_number }} and generate CHANGELOG'
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
push: 'origin origin/${{ steps.variables.outputs.BRANCH_NAME }}'
- name: Create Pull Request
uses: peter-evans/create-pull-request@v3
with:
author: 'Github Build Bot <buildbot@fishtownanalytics.com>'
base: ${{github.ref}}
title: 'Bumping version to ${{ github.event.inputs.version_number }} and generate changelog'
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
labels: |
Skip Changelog
version_bump_and_changie:
uses: dbt-labs/actions/.github/workflows/version-bump.yml@main
with:
version_number: ${{ inputs.version_number }}
secrets: inherit # ok since what we are calling is internally maintained

1
.gitignore vendored
View File

@@ -51,6 +51,7 @@ coverage.xml
*,cover
.hypothesis/
test.env
makefile.test.env
*.pytest_cache/

View File

@@ -5,12 +5,12 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## Previous Releases
For information on prior major and minor releases, see their changelogs:
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)

Some files were not shown because too many files have changed in this diff Show More