Compare commits

...

195 Commits

Author SHA1 Message Date
David Bloss
77fa660611 WIP: replace setuptools with hatch to build dbt 2023-02-14 11:38:58 -06:00
Chenyu Li
f38cbc4feb add helper function for load project/profile (#6964) 2023-02-14 08:47:43 -08:00
Gerda Shank
480e0e55c5 Set relation_name in tests at compile time (#6949) 2023-02-14 11:32:27 -05:00
Michelle Ark
e5c468bb93 dbt_project.yml log-path respects project directory (#6908)
dbt_project.yml log-path respects project directory
2023-02-14 11:30:11 -05:00
Emily Rockman
605c72e86e Automate cutting new .latest branches (#6854)
* first pass

* WIP

* add notes/stubs on more pieces

* more work

* more cleanup

* cleanup

* add more cleanup and generalization

* update to use reusable workflow

* add TODO
2023-02-14 08:09:16 -06:00
Emily Rockman
aad46ac5a8 Changelog Cleanup from feature branch merge (#6935)
* remove changelogs from 1.4

* remove yaml files for already released changes
2023-02-14 08:08:50 -06:00
Emily Rockman
d85618ef26 remove +nightly from version parsing (#6966)
* remove  from version parsing

* update comment

* undo partial version bump
2023-02-14 08:04:26 -06:00
Jeremy Cohen
1250f23c44 Fix petits logging regressions from feature/click-cli merge (#6940)
* Add back initialization events

* Fix log_cache_events. Default stdout logger knows less than it used to

* Add back exception handling events

* Revert "Add back exception handling events"

This reverts commit 26f22d91b660f51f0df6a59f9e5cae16b0ee6fe5.

* Add changelog entry

* Fix test by stringifying dict values

* Add generated CLI API docs

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-14 10:48:56 +01:00
Gerda Shank
daea7d59a7 CT 1921 add missing nodes to previous state test (#6961)
* Update WritableManifest.disabled to include metrics and exposures in disabled dictionary
2023-02-13 21:02:20 -05:00
Emily Rockman
4575757c2a remove '+nightly' from release test version (#6907)
* remove '+nightly' from release test version

* fix release branch variable name

* remove comment
2023-02-13 19:41:34 -06:00
Emily Rockman
d7a2f77705 Infer core team membership in changie (#6420)
* part 1 of env var for core team

* add logic to use env vars to generate changelog

* modify version bump to add members via env var

* pull in main and tweak

* add token

* changes for testing

* split step

* remove leading slash

* add version check

* more debugging

* try curl

* try more things

* try more things

* chnage auth

* put back token

* update permissions

* add back fishtown pat

* use new pat

* fix typo

* swap token

* comment out list teams

* change url

* debug path

* add continue

* change core case

* more tweaks

* send output to file

* add file view

* make array

* tweak

* remove []

* add quotes

* add tojson

* add quotes to set

* tweak

* fix id

* tweaks

* more

* more

* remove new lines

* more tweaks

* update to generate changelog

* remove debugging bits

* use central version-bump

* use correct author list

* testing with changelog team automation

* add new token to input

* move secret

* remove testing aspects from workflow

* clean up team logic

* explicitly send secret

* move bumpversion comment

* move comments

* point workflow back tp main

* point to branch for testing

* point back to main

* inherit secrets
2023-02-13 14:08:49 -06:00
Kshitij Aranke
4a4b89606b Hoist dbt.cli.main into dbt.main namespace (#6939) 2023-02-13 12:06:28 -08:00
Michelle Ark
1ebe2e7118 ExposureOwner: Rename to Owner, accept additional fields, require one of 'name' or 'email' (#6912)
update ExposureOwner to require one of email/name, allow additional fields
2023-02-13 10:13:15 -05:00
Gerda Shank
f1087e57bf Fix types.proto and test_events.py (#6947) 2023-02-10 16:31:34 -05:00
Michelle Ark
250537ba58 handle_and_check in main.py (#6941)
handle_and_check in cli/main.py
2023-02-10 15:39:31 -05:00
Peter Webb
ccc7222868 CT-2079: Re-apply accidentally reverted logging fix (#6937) 2023-02-10 14:40:57 -05:00
Chenyu Li
311a57a21e Merge pull request #6931 from dbt-labs/merge_feature_click_cli_main
Merge feature/click-cli into main

Co-authored-by: Ian Knox <ian.knox@dbtlabs.com >
Co-authored-by: Stu Kilgore <stu.kilgore@dbtlabs.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com >
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com >
Co-authored-by: Jeremy Cohen <jeremey@dbtlabs.com >
2023-02-09 18:03:04 -08:00
Chenyu Li
b7c45de6b1 remove old test 2023-02-09 15:55:24 -08:00
Chenyu Li
c53c3cf181 remove old main 2023-02-09 15:53:37 -08:00
Chenyu Li
a77d325c8a fix all tests 2023-02-09 15:51:42 -08:00
Kshitij Aranke
dd41384d82 revert core.dbt.contracts back to code from main 2023-02-09 15:32:34 -08:00
Github Build Bot
aa55fb2d30 Add generated CLI API docs 2023-02-09 22:59:54 +00:00
Kshitij Aranke
864f4efb8b Merge feature/click-cli into main 2023-02-09 14:57:10 -08:00
dependabot[bot]
83c5a8c24b Bump ubuntu from 22.04 to 23.04 (#6865)
* Bump ubuntu from 22.04 to 23.04

Bumps ubuntu from 22.04 to 23.04.

---
updated-dependencies:
- dependency-name: ubuntu
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-09 16:59:25 -05:00
Kshitij Aranke
57aef33fb3 Click cli main merge (#6926) 2023-02-09 12:21:39 -08:00
FishtownBuildBot
6d78e5e640 Add most recent dbt-docs changes (#6923)
* Add new index.html and changelog yaml files from dbt-docs

* Update .changes/unreleased/Docs-20230209-082901.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-02-09 13:23:12 -06:00
Ryan Harris
f54a876f65 Update link to dbt install docs (#6883)
* update dbt install link

* add changelog entry
2023-02-09 09:35:33 -08:00
Kshitij Aranke
8bbae7926b Fix Project Env Var Tests (#6916)
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2023-02-09 08:53:39 -08:00
Jeremy Cohen
db2b12021e Fix test_builtin_invocation_args_dict_function (#6898)
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2023-02-09 09:53:42 -06:00
Peter Webb
8b2c9bf39d Ensure flush() after logging write() (#6909)
* ct-2063: Ensure flush after logging, by using Python's logging subsystem directly

* ct-2063: Add changelog entry
2023-02-09 09:37:37 -05:00
Jeremy Cohen
298bf8a1d4 Add back depends_on for seeds - only macros, never nodes (#6851)
* Extend functional tests for seeds w hooks

* Add MacroDependsOn to seeds, raise exception for other deps

* Add changelog entry

* Fix unit tests

* Update upgrade_seed_content

* Cleanup

* Regen manifest v8 schema. Fix tests

* Be less magical

* PR feedback
2023-02-09 10:56:12 +01:00
Michelle Ark
77748571b4 profiles dir exists=False for dbt debug (#6910)
profiles dir exists=False for dbt debug
2023-02-08 18:32:06 -05:00
Stu Kilgore
8ce4c289c5 Docs generate doesn't write manifest (#6905) 2023-02-08 16:16:31 -06:00
Emily Rockman
abbece8876 1.4 regression: Check if status has node attribute (#6899)
* check for node

* add changelog

* add test for regression
2023-02-08 13:49:24 -06:00
dave-connors-3
3ad40372e6 add base class for merge exclude tests (#6700)
* add base class for merge exclude tests

* changie <33

* remove comments

* add comments to sql, remove and clarify contents of resultholder
2023-02-08 10:20:13 -08:00
Kshitij Aranke
c6d0e7c926 Fix Click CLI test DB Name (#6895) 2023-02-08 09:58:16 -08:00
Michelle Ark
bc015843d4 use UnsetProfile in deps and clean commands (#6890)
use unset_profile in deps and clean commands
2023-02-08 12:26:58 -05:00
Emily Rockman
df64511feb Dynamically list all .latest branches for scheduled testing (#6682)
* first pass at automating latest branches

* checkout repo first

* fetch all history

* reorg

* debugging

* update test id

* swap lines

* incorporate new branch aciton

* tweak vars
2023-02-08 08:02:21 -06:00
Michelle Ark
db0981afe7 NoneConfig for DebugTask (#6893)
NoneConfig for DebugTask
2023-02-07 21:33:15 -05:00
Michelle Ark
dcf6544f93 flags.THREADS defaults to None (#6887)
flags.THREADS defaults to None
2023-02-07 17:15:26 -05:00
Ian Knox
c2c8959fee Merge main to feature (#6817)
merge main to feature/click-cli
2023-02-07 16:32:31 -05:00
Peter Webb
ccb4fa26cd CT-1917: Fix a regression in the behavior of the -q/--quiet cli parameter (#6886) 2023-02-07 16:15:42 -05:00
Chenyu Li
d0b5d752df consolidate flags (#6788)
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-07 08:35:35 -08:00
Neelesh Salian
4c63b630de [CT-1959]: moving simple_seed tests to adapter zone (#6859)
* Formatting

* Changelog entry

* Rename to BaseSimpleSeedColumnOverride

* Better error handling

* Update test to include the BOM test

* Cleanup and formating

* Unused import remove

* nit line

* Pr comments
2023-02-06 19:51:32 -08:00
Kshitij Aranke
9c0b62b4f5 Fix CLI vars test to check for object instead of string (#6850) 2023-02-06 15:13:40 -08:00
Ian Knox
e08eede5e2 Remove unused cli_runner (#6877) 2023-02-06 15:24:09 -06:00
Ian Knox
05e53d4143 Test fix: TestProfileEnvVars::test_profile_env_vars (#6856) 2023-02-06 10:49:40 -06:00
colin-rogers-dbt
b2ea2b8b25 move test_store_test_failures.py to adapter zone (#6816) 2023-02-02 10:55:09 -08:00
Emily Rockman
2245d8d710 update regex to match all iterations (#6839)
* update regex to match all iterations

* convert to num to match all adapters

* add comments, remove extra .

* clarify with more comments

* Update .bumpversion.cfg

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>

---------

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
2023-02-02 12:16:06 -06:00
Gerda Shank
d9424cc710 CT 2000 fix semver prerelease comparisons (#6838)
* Modify semver.py to not use packaging.version.parse

* Changie
2023-02-02 12:22:48 -05:00
Stu Kilgore
0503c141b7 Lazily call --version (#6813)
* Lazily call --version

* Add generated CLI API docs

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-31 17:20:18 -08:00
Mila Page
1a6e4a00c7 Add clearer directions for custom test suite vars in Makefile. (#6764)
* Add clearer directions for custom test suite vars in Makefile.

* Fix up PR for review

* Fix erroneous whitespace.

* Fix a spelling error.

* Add documentation to discourage makefile edits but provide override tooling.

* Fix quotation marks. Very strange behavior

* Compact code and verify quotations happy inside bash and python.

* Fold comments into Makefile.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-31 13:53:55 -08:00
Mila Page
42b7caae19 Ct 1827/064 column comments tests conversion (#6766)
* Convert test and make it a bit more pytest-onic

* Ax old integration test.

* Run black on test conversion

* I didn't like how pytest was running the fixture so wrapped it into a closure.

* Merge converted test into persist docs.

* Move persist docs tests to the adapter zone. Prep for adapter tests.

* Fix up test names

* Fix name to be less confusing.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-31 12:58:19 -08:00
Emily Rockman
622e5fd71d fix contributor list generation (#6799) 2023-01-31 13:05:13 -06:00
Stu Kilgore
59d773ea7e Implement --version in click (#6802) 2023-01-31 12:40:01 -06:00
Kshitij Aranke
84bf5b4620 [CT-1947] Alias --models to --select for all commands except dbt ls (#6787)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-31 10:18:10 -08:00
Ian Knox
726c4d6c58 Enable the new Click Cli (#6785) 2023-01-30 19:30:01 -06:00
Michelle Ark
acc88d47a3 mutually exclusive handling for warn_error_options and warn_error params in Click CLI (#6771)
warn_error_options, warn_error mutual exclusivity with click
2023-01-30 18:38:36 -05:00
Chenyu Li
0a74594d09 move favor state arg to click (#6774) 2023-01-30 11:55:31 -08:00
Kshitij Aranke
d2f3cdd6de [CT-1841] Convert custom target test to Pytest (#6765) 2023-01-30 07:55:28 -08:00
Alexander Smolyakov
92d1ef8482 Update release workflow (#6778)
- Update AWS secrets
- Rework condition for Slack notification
2023-01-30 09:19:10 -06:00
Neelesh Salian
a8abc49632 [CT-1940] Stand-alone Python module for PostgresColumn (#6773) 2023-01-27 17:00:39 -08:00
Ian Knox
d6ac340df0 Merge main into feature/click-cli (#6761) 2023-01-27 15:07:30 -06:00
Neelesh Salian
c653330911 Adding nssalian to committers list (#6769) 2023-01-27 11:16:59 -08:00
Alexander Smolyakov
82d9b2fa87 [CI/CD] Update release workflow and introduce workflow for nightly releases (#6602)
* Add release workflows

* Update nightly-release.yml

* Set default `test_run` value to `true`

* Update .bumpversion.cfg

* Resolve review comment

- Update workflow docs
- Change workflow name
- Set `test_run` default value to `true`

* Update Slack secret

* PyPI
2023-01-27 09:04:31 -06:00
Mila Page
3f96fad4f9 Ct 1629/052 column quoting tests conversion (#6652)
* Test converted and reformatted for pytest.

* Ax old versions of 052 test

* Nix the 'os' import and black format

* Change names of models to be more PEP like

* cleanup code

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-26 12:23:02 -08:00
Peter Webb
c2c4757a2b Graph Analysis Optimization for Large Dags (#6720)
* Optimization to remove graph analysis bottleneck in large dags.

* Add changelog entry.
2023-01-26 14:27:42 -05:00
Kshitij Aranke
08b2d94ccd [CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs … (#6723)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-26 10:42:49 -08:00
Michelle Ark
7fa61f0816 dbt init works with click (#6698)
dbt init works with click
2023-01-26 12:45:15 -05:00
Mila Page
c65ba11ae6 Ct 1827/064 column comments tests conversion (#6654)
* Convert test and make it a bit more pytest-onic

* Ax old integration test.

* Run black on test conversion

* I didn't like how pytest was running the fixture so wrapped it into a closure.

* Merge converted test into persist docs.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-26 00:54:00 -08:00
Matthew Beall
b0651b13b5 change exposure_content to source_content (#6739)
* change `exposure_content` to `source_content`

* Adding changelog

Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2023-01-25 19:51:34 -05:00
Gerda Shank
a34521ec07 CT 1894 log partial parsing var changes and sort cli vars before hashing (#6713)
* Log information about vars_hash, normalize cli_vars before hashing

* Changie

* Add to test_events.py

* Update core/dbt/events/types.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-01-25 17:47:45 -05:00
Matthew McKnight
da47b90503 [CT-1630] Convert Column_types tests (#6690)
* init commit for column_types test conversion

* init start of test_column_types.py

* pass tes macros into both tests

* remove alt tests, remove old tests, push up working conversion

* rename base class, move to adapter zone so adapters can use

* typo fix
2023-01-25 14:57:16 -06:00
Stu Kilgore
d27016a4e7 Migrate debug task to click (#6728) 2023-01-25 14:52:57 -06:00
Peter Webb
db99e2f68d Event Clean-Up (#6716)
* CT-1857: Event cleanup

* Add changelog entry.
2023-01-25 13:51:52 -05:00
Michelle Ark
cbb9117ab9 test_init conversion (#6610)
* convert 044_init_tests
2023-01-24 16:22:26 -05:00
Gerda Shank
e2ccf011d9 CT 1886 include adapter_response in NodeFinished log message (#6709)
* Include adapter_response in run_result in NodeFinished log event

* Changie
2023-01-24 14:25:32 -05:00
Aezo
17014bfad3 add adapter_response for test (#6645)
resolves https://github.com/dbt-labs/dbt-core/issues/2964
2023-01-24 09:58:08 -08:00
Stu Kilgore
92b7166c10 Abstract manifest generation from tasks (#6565) 2023-01-24 11:05:57 -06:00
Peter Webb
7b464b8a49 CT-1718: Add Note and Formatting event types (#6691)
* CT-1718: Add Note and Formatting event types

* CT-1718: Add changelog entry
2023-01-23 16:39:29 -05:00
Sean McIntyre
5c765bf3e2 Cheeky performance improvement on big DAGs (#6694)
* Short-circuit set operations for nice speed boost

* Add changelog

* Fix issue

* Update .changes/unreleased/Under the Hood-20230122-215235.yaml

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-01-23 09:09:09 -07:00
Mila Page
93619a9a37 Ct 738/dbt debug log fix (#6541)
* Code cleanup and adding stderr to capture dbt

* Debug with --log-format json now prints structured logs.

* Add changelog.

* Move logs into miscellaneous and add values to test.

* nix whitespace and fix log levels

* List will now do structured logging when log format set to json.

* Add a quick None check.

* Add a get guard to class check.

* Better null checking

* The boolean doesn't reflect the original logic but a try-catch does.

* Address some code review comments and get us working again.

* Simplify logic now that we have a namespace object for self.config.args.

* Simplify logic for json log format checking.

* Simplify code for allowing our GraphTest cases to pass while also hiding compile stats from dbt ls/list .

* Simplify structured logging types.

* Fix up boolean logic and simplify via De'Morgan.

* Nix unneeded fixture.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-20 16:37:54 -08:00
Doug Beatty
a181cee6ae Improve error message for packages missing dbt_project.yml (#6685)
* Improve error message for packages missing `dbt_project.yml`

* Use black formatting

* Update capitalization of expected error message
2023-01-20 13:46:36 -07:00
Michelle Ark
a0ade13f5a dbt docs generate works with click (#6681) 2023-01-20 15:26:20 -05:00
Michelle Ark
9823a56e1d dbt build works in click (#6680)
* build working with click
2023-01-20 14:27:39 -05:00
Michelle Ark
3aeab73740 convert 069_build_tests (#6678) 2023-01-20 14:27:02 -05:00
Jeremy Cohen
9801eebc58 Consolidate changie entries from #6620 (#6684) 2023-01-20 19:58:40 +01:00
Jeremy Cohen
11c622230c Add run-operation to click CLI (#5552) (#6656)
* Add run-operation to click CLI

* Add changelog entry

* PR feedback

* Fix unit test
2023-01-20 01:58:47 +01:00
Chenyu Li
f0349488ed Seed and freshness works with click (#6651) 2023-01-19 16:14:26 -08:00
Chenyu Li
c85be323f5 fix MP_CONTEXT is not JSON serializable (#6650) 2023-01-19 08:16:58 -08:00
Peter Webb
6954c4df1b CT-1786: Port docs tests to pytest (#6608)
* CT-1786: Port docs tets to pytest

* Add generated CLI API docs

* CT-1786: Comply with the new style requirements

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-19 11:11:17 -05:00
Michelle Ark
30a1595f72 click working with list (#6641)
* list working with click
2023-01-19 09:28:32 -05:00
dave-connors-3
f841a7ca76 add backwards compatibility and default argument for incremental_predicates (#6628)
* add backwards compatibility and default argument

* changie <3

* Update .changes/unreleased/Fixes-20230117-101342.yaml

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-01-19 15:20:19 +01:00
Jeremy Cohen
07a004b301 convert 062_defer_state_tests (#6616)
* Fix --favor-state flag

* Convert 062_defer_state_tests

* Revert "Fix --favor-state flag"

This reverts commit ccbdcbad98b26822629364e6fdbd2780db0c20d3.

* Reformat

* Revert "Revert "Fix --favor-state flag""

This reverts commit fa9d2a09d693b1870bd724a694fce2883748c987.
2023-01-19 11:00:09 +01:00
Jeremy Cohen
b05582de39 mv on_schema_change tests -> "adapter zone" (#6618)
* Mv incremental on_schema_change tests to 'adapter zone'

* Use type_string()

* Cleanup
2023-01-19 10:12:59 +01:00
Jeremy Cohen
fa7c4d19f0 Respect quoting config in dbt-py models (#6620)
* Respect quoting for 'this' in dbt-py models #6619

* Respect quoting for ref/source in dbt-py models #6103

* Add changelog entries
2023-01-19 09:34:08 +01:00
Chenyu Li
1913eac5ed Click snapshot as click command (#5972) (#6640) 2023-01-18 16:01:58 -08:00
Jeremy Cohen
066346faa2 convert 038_caching_tests (#6612)
* convert 038_caching_tests

* Adapt for dbt-snowflake

* PR feedback

* Reformat
2023-01-18 22:37:50 +01:00
Emily Rockman
0a03355ceb update test matrix (#6604) 2023-01-18 14:16:34 -06:00
David Bloss
43e24c5ae6 update gh action set-output variables (#6635)
* update gh action set-output variables

* add changie file
2023-01-18 11:23:13 -06:00
Gerda Shank
89d111a5f6 CT 1440 Fix code to emit ConnectionReused event (#6605)
* Refactor "set_connection_name" to properly handle reused connection

* Update test

* Changie

* Limit test of ConnectionUsed events to non-Windows
2023-01-17 13:18:07 -05:00
Gerda Shank
e1b5e68904 Convert 068_partial_parsing_tests (#6614)
* Convert partial parsing tests

* reformat
2023-01-17 12:22:31 -05:00
Jeremy Cohen
065ab2ebc2 Reformat tests/ (#6622)
* Run black + flake8 on tests dir

* Run pre-commit
2023-01-16 16:39:54 +01:00
Michelle Ark
20c95a4993 convert 057_run_query_tests (#6607) 2023-01-13 15:02:17 -05:00
Michelle Ark
c40b488cb4 convert 037_external_references (#6589) 2023-01-13 14:15:03 -05:00
Chenyu Li
585e7c59e8 migrate ref override (#6600) 2023-01-13 09:15:04 -08:00
Michelle Ark
7077c47551 converting 022_test_timezones (#6588) 2023-01-13 10:52:21 -05:00
Chenyu Li
f789b2535a convert adapter table materialization test (#6595) 2023-01-12 16:05:51 -08:00
Emily Rockman
2bfc6917e2 finish message rename in types.proto (#6594)
* finish message rename in types.proto

* add new parameter
2023-01-12 14:33:56 -06:00
github-actions[bot]
d74ae19523 Bumping version to 1.5.0a1 and generate changelog (#6593)
* Bumping version to 1.5.0a1 and generate CHANGELOG

* make relevvant changelog updates

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-01-12 12:59:28 -06:00
Michelle Ark
1c7c23ac73 convert 043_custom_alias_tests (#6590) 2023-01-12 11:07:26 -05:00
Jeremy Cohen
86e8722cd8 Call update_event_status earlier + rename an event (#6572)
* Rename HookFinished -> FinishedRunningStats

* Move update_event_status earlier when node finishes

* Add changelog entry

* Add update_event_status for skip

* Update changelog entry
2023-01-12 09:34:36 +01:00
Kshitij Aranke
53127daad8 [CT-921] dbt compile works in click (#5545) (#6586)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
resolves https://github.com/dbt-labs/dbt-core/issues/5545
2023-01-11 15:01:50 -08:00
Kshitij Aranke
91b20b7482 dbt test works with Click (#5556)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
resolves https://github.com/dbt-labs/dbt-core/issues/5556
2023-01-11 12:19:43 -08:00
Michelle Ark
7a61602738 Add --warn-error-options (#6520)
* adding --warn-error-options

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-10 19:43:52 -05:00
Doug Beatty
dd4b47d8b1 Check length of escaped strings in the adapter test (#6567)
* Check length of escaped strings in the adapter test

* One column per line to improve readability
2023-01-10 17:26:35 -07:00
Emily Rockman
eb200b4687 Rename exceptions (#6539)
* rename InternalException

* rename RuntimeException

* rename DatabaseException

* rename CompilationException

* cleanup renames in tests and postgres

* rename ValidationException

* rename IncompatibleSchemaException

* more renaming

* more renaming

* rename InternalException again

* convert ParsingException

* replace JSONValidationException and SemverException

* replace VersionsNotCompatibleException

* replace NotImplementedException

* replace FailedToConnectException

* replace InvalidConnectionException

* replace InvalidSelectorException

* replace DuplicateYamlKeyException

* replace ConnectionException

* minor cleanup

* update comment

* more cleanup

* add class decorator

* rename more exceptions

* more renamed, add changelog

* rename exception

* rework class deprecations

* removing testing line

* fix failing test

* rename newer exceptions

* fix failing test

* commit unsaved faile

* convert back an rpc exception

* remove class deprecations
2023-01-10 17:43:27 -06:00
mivanicova
0fc080d222 add defer_to_manifest in before_run to fix faulty deferred docs generate (#6488)
* add defer_to_manifest in before_run to fix faulty deferred docs generate

* add a changelog

* add declaration of defer_to_manifest to FreshnessTask and GraphRunnableTask

* fix: add defer_to_manifest method  to ListTask
2023-01-10 15:18:20 -08:00
AGPapa
5da63602b3 Adds buildable selection mode (#6366) 2023-01-10 15:15:48 -08:00
nshuman1
457ff3ef48 fixed minor typos (#6499)
* fixed minor typos

* adding changelog file

* linter

* Update Docs-20230102-170216.yaml
2023-01-10 13:36:21 -08:00
Gerda Shank
0dbdecef10 Ct 1716 cleanup logging events (#6561)
* Combine DbtProfileError log events

* Combine DbtProjectErrorException with DbtProjectError

* Combine cache logging events

* Changie

* fix ticket number

* Ooops. Add another file.

* fix serialization of profile names
2023-01-10 15:56:52 -05:00
Doug Beatty
b13b0e9492 Re-factor list of YAML keys for hooks to late-render (#6435)
* Re-factor list of YAML keys for hooks to late-render

* Add `pre_` and `post_hook` to list of late-rendered hooks

* Check for non-empty set intersection

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>

* Test functional synonymy of `*_hook` with `*-hook`

Test that `pre_hook`/`post_hook` are functionally synonymous with `pre-hook`/`post-hook` for model project config

* Undo bugfix to validate the new test fails

* Revert "Undo bugfix to validate the new test fails"

This reverts commit e83a2be2eb6f0ef4d95ffcfaf77ea9287eb0fbca.

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-01-10 12:35:06 -07:00
Tim Mastny
b9fdfd9e36 Adds the meta field to node_info in logs (#6493)
* add meta attribute to nodeinfo for events

* also add meta to dataclass

* add to unit test to ensure meta is added

* adding functional test to check that meta is passed to nodeinfo during logging

* changelog

* remove used imported

* add tests with non-string keys

* renaming test dict keys

* add non-string value

* resolve failing test

* test additional non-string values

* fix flake8

* Stringify meta dict in node_info

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-01-10 13:42:51 -05:00
Peter Webb
4d6352db14 CT-1645: Bump mashumaro version to receive regression fix, and add unit test to confirm fix. (#6564) 2023-01-10 13:14:24 -05:00
Doug Beatty
9eb82c6497 Remove duplicated changelog entry for 5521 (#6563) 2023-01-10 11:13:50 -07:00
Jeremy Cohen
89cc89dfdf Fix changelog entry for 5989/6432 (#6560) 2023-01-10 06:57:39 -07:00
Kshitij Aranke
2b0f6597a4 [CT-1694] Deprecate event tracking tests (#6538) 2023-01-09 15:23:02 -08:00
Peter Webb
294def205f Fix event level regression (#6556)
* CT-6501: Make certain disabled test events debug-level again to match historical behavior.

* CT-1740: Add changelog entry.
2023-01-09 18:15:59 -05:00
Emily Rockman
34fa703466 fix log msg format (#6557) 2023-01-09 15:56:42 -06:00
Mila Page
ab3f8dcbfd Ct 1517/convert 060 persist docs (#6409)
* convert the test and fix an error due to a dead code seed

* Get rid of old test

* Remove unfortunately added files. Don't use that *

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-09 11:07:34 -08:00
Emily Rockman
02c20477b9 add deprecation decorator (#6540)
* add derecation decorator

* fix tests

* updated changelog and deprecation reason
2023-01-09 12:09:12 -06:00
Gerda Shank
d9a4ee126a CT 1549 reorg logging events to have two top level keys (#6553)
* Update types.proto

* pre-commit passes

* Cleanup tests and tweak EventLevels

* Put node_info back on SQLCommit. Add "level" to fire_event function.

* use event.message() in warn_or_error

* Fix logging test

* Changie

* Fix a couple of unit tests

* import Protocol from typing_extensions for 3.7
2023-01-09 11:33:44 -05:00
Michelle Ark
5b31cc4266 Remove UnsetProfileConfig (#6504)
remove UnsetProfileConfig
2023-01-06 20:12:40 -05:00
Chenyu Li
9bb1250869 merge main to feature/click-cli (#6483)
* merge main to feature/click-cli

* fix var hash

* move back changes in feature branch for deps

* fix logging issues
2023-01-06 15:11:23 -08:00
Kshitij Aranke
94d6d19fb4 [CT-1693] Port severity test to Pytest (#6537) 2023-01-06 11:59:53 -08:00
Callum McCann
d43c070007 Making Metric Timestamps Optional (#6402)
* changing to optional

* adding tests

* tests and changie

* pre-commit cleaning

* formatting fixes

* pre-commit update
2023-01-05 15:49:55 -05:00
Callum McCann
9ef236601b Updating metric attributes renamed event (#6518)
* updating message

* adding changie
2023-01-05 15:49:44 -05:00
justbldwn
9d6f961d2b 🎨 add missing fstrings, convert format to fstring (#6243)
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-01-05 15:28:27 -05:00
Niall Woodward
5453840950 DBT_FAVOR_STATE_STATE -> DBT_FAVOR_STATE (#6392)
* DBT_FAVOR_STATE_STATE -> DBT_FAVOR_STATE

* add changelog
2023-01-04 15:53:35 -08:00
Gerda Shank
d453964546 CT 1551 partial parsing logging events (#6474) 2023-01-04 14:28:26 -05:00
dependabot[bot]
748a932811 Update agate requirement from <1.6.4,>=1.6 to >=1.6,<1.7.1 in /core (#6506)
* Update agate requirement from <1.6.4,>=1.6 to >=1.6,<1.7.1 in /core

Updates the requirements on [agate](https://github.com/wireservice/agate) to permit the latest version.
- [Release notes](https://github.com/wireservice/agate/releases)
- [Changelog](https://github.com/wireservice/agate/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/wireservice/agate/compare/1.6.0...1.7.0)

---
updated-dependencies:
- dependency-name: agate
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-04 09:02:37 -08:00
Jeremy Cohen
8217ad4722 Fix issue link for 5859 changelog entry (#6496) 2023-01-04 10:46:52 +01:00
Michelle Ark
cc5a38ec5a Example click API usage (#6307)
* Example python lib w click, written docs

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-01-03 18:46:34 -05:00
Gerda Shank
6ef3fbbf76 Partial parsing bug with empty schema file - ensure None is not passed to load_yaml_text (#6494) 2023-01-03 17:14:56 -05:00
justbldwn
76fd12c7cd adding pre-commit install to make dev (#6417)
*  adding pre-commit install to make dev

* 🎨 updating format of Makefile and CONTRIBUTING.md

* 📝 adding changelog via changie new

*  adding dev_req to Makefile + docs

* 🎨 remove dev_req from docs, dry makefile

* Align names of `.PHONY` targets with their associated rules

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
2023-01-03 11:18:55 -07:00
Doug Beatty
9ecb6e50e4 Treat dense text blobs as binary for git grep (#6462) 2022-12-20 17:10:01 -07:00
Kshitij Aranke
ce9d0afb8a [CT-1591] Don't parse empty Python files (#6433) 2022-12-20 10:56:36 -08:00
Emily Rockman
c39ea807e8 add back validator_error_message (#6465) 2022-12-20 10:56:08 -06:00
Ikko Ashimine
1e35339389 Fix typo in util.py (#6037)
* Fix typo in util.py

identifer -> identifier

* Add change log

* Update .changes/unreleased/Fixes-20221117-220320.yaml

Co-authored-by: Emily Rockman <ebuschang@gmail.com>
2022-12-19 20:01:19 -06:00
Emily Rockman
304797b099 exception cleanup (#6347)
* starting to move jinja exceptions

* convert some exceptions

* add back old functions for backward compatibility

* organize

* more conversions

* more conversions

* add changelog

* split out CacheInconsistency

* more conversions

* convert even more

* convert parsingexceptions

* fix tests

* more conversions

* more conversions

* finish converting exception functions

* convert more tests

* standardize to msg

* remove some TODOs

* fix test param and check the rest

* add comment, move exceptions

* add types

* fix type errors

* fix type for adapter_response

* remove 0.13 version from message
2022-12-19 19:36:24 -06:00
Emily Rockman
b9bdb775ab Fix dependency changelog kind (#6452) 2022-12-19 13:13:38 -06:00
Michelle Ark
b0909b8f5d Functional test framework working with Click, dbtRunner (#6387)
dbtRunner, dbt.cli.requires - preflight, profile, project
2022-12-16 19:54:42 -05:00
Stu Kilgore
5d278dacf1 Make clean task work with click (#6369) 2022-12-15 11:19:16 -06:00
leahwicz
df93858b4b Updating 1.4.0b1 changelog (#6450) 2022-12-15 09:55:23 -05:00
dave-connors-3
e8da84fb9e Feature/support incremental predicates (#5702)
* pass predicated to merge strategy

* postgres delete and insert

* merge with predicates

* update to use arbitrary list of predicates, not dictionaries, merge and delete

* changie

* add functional test to adapter zone

* comma in test config

* add test for incremental predicates delete and insert postgres

* update test structure for inheritance

* handle predicates config for backwards compatibility

* test for predicates keyword

* Add generated CLI API docs

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-12-14 14:07:41 -08:00
Peter Webb
7e90e067af Paw/ct 1652 restore default logging (#6447)
* CT-1652: Restore stdout logging before logger is fully configured

* CT-1652: Add changelog entry

* CT-1652: formatting fix for black
2022-12-14 17:04:50 -05:00
Peter Webb
5e4e917de5 CT-1685: Restore certain aspects of legacy logging behavior important… (#6443)
* CT-1685: Restore certain aspects of legacy logging behavior important to dbt-rpc

* CT-1658: And changelog entry
2022-12-14 11:13:34 -05:00
github-actions[bot]
05dc0212e7 Bumping version to 1.4.0b1 and generate changelog (#6440)
* Bumping version to 1.4.0b1 and generate CHANGELOG

* Updating date

* Updating date

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-12-13 20:18:11 -05:00
Gerda Shank
c00052cbfb Add Optional back on "database" field of HasRelationMetadata (#6439) 2022-12-13 18:15:25 -05:00
Kshitij Aranke
3d54a83822 [CT-1284] Change Python model default materialization to table (#6432) 2022-12-13 15:07:56 -08:00
Gerda Shank
fafd5edbda CT 1644 node cleanup (#6427)
* Remove unneeded SQL compilation attributes from SeedNode

* Fix various places that referenced removed attributes

* Cleanup a few Unions

* More formatting in nodes.py

* Mypy passing. Untested.

* Unit tests working

* use "doc" in documentation unique_ids

* update some doc_ids

* Fix some artifact tests. Still need previous version.

* Update manifest/v8.json

* Move relation_names to parsing

* Fix a couple of tests

* Update some artifacts. snapshot_seed has wrong schema.

* Changie

* Tweak NodeType.Documentation

* Put store_failures property in the right place

* Fix setting relation_name
2022-12-13 12:39:35 -05:00
Josh Devlin
8478262580 Update docker README (#6423) 2022-12-13 11:12:34 -05:00
Kshitij Aranke
83b1fee062 Add aranke to core committers (#6431) 2022-12-12 15:13:03 -08:00
Emily Rockman
0fbbc896b2 Remove PR from most changelog kinds (#6374)
* update changie to require issue or pr, and allow multiple

* remove extraneous data from changelog files.

* allow for multiple PR/issues to be entered

* update contributing guide

* remove issue number from bot changelogs

* update format of PR

* fix dependency changelogs

* remove extra line

* remove extra lines, tweak contributor wording

* Update CONTRIBUTING.md

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2022-12-12 13:18:15 -06:00
Stu Kilgore
ce1aaec31d Adjust tox passenv to be multiline (#6405) (#6430) 2022-12-12 11:13:04 -06:00
Ian Knox
0544b08543 Add support for Python 3.11 (#6326)
* Get running with Python 3.11

* More tests passing, mypy still unhappy

* Upgrade to 3.11, and bump mashumaro

* patch importlib.import_module last

* lambda: Policy() default_factory on include and quote policy

* Add changelog entry

* Put a lambda on it

* Fix text formatting for log file

* Handle variant type return from e.log_level()

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Josh Taylor <joshuataylorx@gmail.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2022-12-08 18:34:03 +01:00
Ian Knox
1809852a0d dbt run works with Click (#6396) 2022-12-08 09:47:08 -06:00
Ian Knox
88d2ee4813 Profile works w Click (#6336) 2022-12-08 09:38:15 -06:00
bruno messias
bef6edb942 Fix dbt.config.get default values (python-model) (#6317)
* feat: add a list of default values to the ctx manager

* tests: dbt.get.config default values

* feat: validate the num of args in config.get

* feat: jinja template for dbt.config.get default values

* docs: changie yaml

* fix:typo on error message

Co-authored-by: Chenyu Li <chenyulee777@gmail.com>

Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
2022-12-07 15:52:38 -08:00
timle
99f27de934 Feature/dbt deps tarball (#4689)
* v0 - new dbt deps type: tarball url

in support of
https://github.com/dbt-labs/dbt-core/issues/4205

* flake8 fixes

* adding max size tarball condition

* clean up imports

* typing

* adding sha1 and subdirectory options; improve logging feedback

sha1: allow user to specify sha1 in packages.yaml, will only install if package matches
subdirectory: allow user to specify subdirectory of package in tarfile, if the package is a non standard structure (like with git subdirectory option)

* simple tests added

* flake fixes

* changes to support tests; adding exceptions; fire_event logging

* new logging events

* tarball exceptions added

* build out tests

* removing in memory tarball test

* update type codes to M - Misc

* adding new events to test_events

* fix spacing for flake

* add retry download code - as used in registry calls

* clean

* remove saving tar in memory inside tarfile object

will hit url multiple times instead

* remove duplicative code after refactor

* black updates

* black formatting

* black formatting

* refactor - no more in-memory tarfile - all as file operations now

- remove tarfile passing, always use tempfile instead
- reorganize system.* functions, removing duplicative code
- more notes on current flow and structure - esp need for pattern of 1) unpack 2) scan for package dir 3) copy to destination.
- cleaning

* cleaning and sync to new tarball code

* cleaning and sync to new tarball code

* requested changes from PR

https://github.com/dbt-labs/dbt-core/pull/4689#discussion_r812970847

* reversions from revision 2

removing sha1 check to simplify/mirror hub install pattern

* simplify/mirror hub install pattern

to simplify/mirror hub install pattern
- removing sha1 check
- supply name/version to act as our 'metadata' source

* simplify/mirror hub install pattern

simplify with goal of mirroring hub install pattern
- supporting subfolders like git packages, and sha1 checks are removed
- existing code from RegistryPinnedPackage (install() and download_and_untar()) performs the operations
- RegistryPinnedPackage install() and download_and_untar() are not currently set up as functions that can be used across classes - this should be moved to dbt.deps.base, or to a dbt.deps.common file - need dbt labs feedback on how to proceed (or leave as is)

* remove revisions, no longer doing package check

* slim down to basic tests

more complex features have been removed (sha1, subfolder) so testing is much simpler!

* fix naming to match hubs behavior

remove version from package folder name

* refactor install and download to upstream PinnedPackage class

i'm on the fence if this is right approach, but seems like most sensible after some thought

* Create Features-20221107-105018.yaml

* fix flake, black, mypy errors

* additional flake/black fixes

* Update .changes/unreleased/Features-20221107-105018.yaml

fix username on changelog

Co-authored-by: Emily Rockman <ebuschang@gmail.com>

* change to fstring

Co-authored-by: Emily Rockman <ebuschang@gmail.com>

* cleaning - remove comment

* remove comment/question for dbt team

* in support of issuecomment 1334055944

https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334055944

* in support of issuecomment 1334118433

https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334118433

* black fixes; remove debug bits

* remove `.format` & add 'tarball' as version

'tarball' as version so that the temp files format nicely:
[tempfile_location]/dbt_utils_2..tar.gz # old
vs
[tempfile_location]/dbt_utils_1.tarball.tar.gz # current

* port os.path refs in `PinnedPackage._install` to pathlib

* lowercase as per PR feedback

* update tests after removing version arg

goes along with 8787ba41af585b954d18c24c2dccbf147035cd9f

Co-authored-by: Emily Rockman <ebuschang@gmail.com>
2022-12-07 15:48:17 -06:00
Jeremy Cohen
9c91f3a7bd Adjust tox passenv to be multiline (#6405) 2022-12-07 22:47:51 +01:00
Gerda Shank
1b6fed2ffd CT 1604 remove compiled classes (#6384)
* removed Compiled versions of nodes

* Remove compiled fields from dictionary if not compiled

* check compiled is False instead of attribute existence in env_var
processing

* Update artifacts test (CompiledSnapshotNode did not have SnapshotConfig)

* Changie

* more complicated 'compiling' check in env_var

* Update test_exit_codes.py
2022-12-07 15:21:05 -05:00
dependabot[bot]
0721f2c1b7 Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core (#6375)
* Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core

Bumps [mashumaro[msgpack]](https://github.com/Fatal1ty/mashumaro) from 3.1.1 to 3.2.
- [Release notes](https://github.com/Fatal1ty/mashumaro/releases)
- [Commits](https://github.com/Fatal1ty/mashumaro/compare/v3.1.1...v3.2)

---
updated-dependencies:
- dependency-name: mashumaro[msgpack]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-12-07 10:16:31 -05:00
Doug Beatty
b9a35da118 Fix intermittent database connection failure in Windows CI test (#6395)
* Fix intermittent database connection failure in Windows CI test

* Changelog entry
2022-12-06 18:34:39 -07:00
Peter Webb
60f80056b1 CT-1405: Refactor event logging code (#6291)
* CT-1405: Refactor event logging code

* CT-1405: Add changelog entry

* CT-1405: Add code to protect against using closed streams from past tests.

* CT-1405: Restore unit test which was only failing locally

* CT-1405: Document a hack with issue # to resolve it in the future

* CT-1405: Make black happy

* CT-1405: Get rid of confusing factory function and duplicated function

* CT-1405: Remove unused event from types.proto and auto-gen'd file
2022-12-06 15:51:52 -05:00
Stu Kilgore
540c3b79aa Prevent docs gen workflow on forks (#6390) 2022-12-06 11:14:20 -06:00
Stu Kilgore
77be2e4fdf Pin ubuntu at 20.04 for some workflows (#6380) 2022-12-05 10:55:36 -06:00
Gerda Shank
16f529e1d4 CT 1477 enrich logging events with data similar to legacy logger (#6325) 2022-12-02 19:29:25 -05:00
Doug Beatty
ebfcf2a9ef Update core/dbt/README.md to match current (#6371)
* Update core/dbt/README.md to match current

Add missing files/folders and alphabetize

* Changelog entry
2022-12-02 15:45:53 -07:00
Alexander Butler
67a8138b65 [fix] Fix the partial parse write path (#6081)
* Fix the partial parse path

Partial parse should use project root or it does not resolve to correct path. 
Eg. `target-path: ../some/dir/target`, if not ran from root, creates an erroneous folder.

* Run pre-commit

* Changie

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-12-02 17:23:37 -05:00
Michelle Ark
e91863de59 Set Flags from UserConfig (#6266)
flags with user config, flags.WHICH from invoked_subcommand if available
2022-12-02 15:23:58 -05:00
leahwicz
85d0b5afc7 Reverting back to older ubuntu image (#6363)
* Reverting back to older ubuntu image

* Updating the structured logging workflow as well
2022-12-02 12:09:46 -05:00
Matthew McKnight
1fbcaa4484 reformatting of test after some spike investigation (#6314)
* reformatting of test after some spike investigation

* reformat code to pull tests back into base class definition, move a test to more appropriate spot
2022-12-01 16:54:58 -06:00
justbldwn
481235a943 clarify error log for number of allowed models in a Python file (#6251) 2022-12-01 14:43:36 -05:00
Michelle Ark
2289e45571 Exposures support metrics (#6342)
* exposures support metrics
2022-12-01 11:01:16 -05:00
dependabot[bot]
b5d303f12a Bump mashumaro[msgpack] from 3.0.4 to 3.1 in /core (#6108)
* Bump mashumaro[msgpack] from 3.0.4 to 3.1.1 in /core

Bumps [mashumaro[msgpack]](https://github.com/Fatal1ty/mashumaro) from 3.0.4 to 3.1.
- [Release notes](https://github.com/Fatal1ty/mashumaro/releases)
- [Commits](https://github.com/Fatal1ty/mashumaro/compare/v3.0.4...v3.1)

---
updated-dependencies:
- dependency-name: mashumaro[msgpack]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2022-11-30 17:32:47 -05:00
Mila Page
c3be975783 Ct 288/convert 070 incremental test (#6330)
* Convert incremental schema tests.

* Drop the old test.

* Bad git add. My disappoint is immeasurable and my day has been ruined.

* Adjustments for flake8.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2022-11-29 12:47:20 -08:00
Mila Page
47c2edb42a Ct 1518/convert 063 relation names tests (#6304)
* Convert old test.

Add documentation. Adapt and reenable previously skipped test.

* Convert test and adapt and comment for current standards.

* Remove old versions of tests.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2022-11-29 12:25:36 -08:00
Stu Kilgore
b3440417ad Add GHA workflow to build CLI API docs (#6187) 2022-11-29 13:30:47 -06:00
leahwicz
020f639c7a Update stale.yml (#6258) 2022-11-29 09:40:59 -05:00
Michelle Ark
44b457c191 DepsTask with Click (#6260)
* deps with click, default --vars param, PartialProject.render_package_metadata
2022-11-22 17:52:59 -05:00
Chenyu Li
a0ec0b6f9d Project working with Click (#6142)
Co-authored-by: MichelleArk <michelle.ark@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-11-15 15:54:05 -08:00
Ian Knox
1ec54abdc4 Logging works with Click (#6088) 2022-10-25 14:14:08 -05:00
Ian Knox
5efc4aa066 Fix CI issue with tox (#6137) 2022-10-25 13:32:15 -05:00
Chenyu Li
847c0b9644 Tracking works with Click (#5972)
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2022-10-25 10:41:57 -07:00
690 changed files with 27993 additions and 22517 deletions

View File

@@ -1,13 +1,15 @@
[bumpversion]
current_version = 1.4.0a1
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
((?P<prekind>a|b|rc)
(?P<pre>\d+) # pre-release version num
)?
current_version = 1.5.0a1
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
(((?P<prekind>a|b|rc) # optional pre-release type
?(?P<num>[\d]+?)) # optional pre-release version number
\.?(?P<nightly>[a-z0-9]+)? # optional nightly release indicator
)? # expected matches: `1.5.0`, `1.5.0a1`, `1.5.0a1.dev123457`, expected failures: `1`, `1.5`, `1.5.2-a1`, `text1.5.0`
serialize =
{major}.{minor}.{patch}{prekind}{pre}
{major}.{minor}.{patch}{prekind}{num}.{nightly}
{major}.{minor}.{patch}{prekind}{num}
{major}.{minor}.{patch}
commit = False
tag = False
@@ -21,9 +23,11 @@ values =
rc
final
[bumpversion:part:pre]
[bumpversion:part:num]
first_value = 1
[bumpversion:part:nightly]
[bumpversion:file:core/setup.py]
[bumpversion:file:core/dbt/version.py]

View File

@@ -3,6 +3,7 @@
For information on prior major and minor releases, see their changelogs:
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)

View File

@@ -0,0 +1,6 @@
kind: "Dependencies"
body: "Bump ubuntu from 22.04 to 23.04"
time: 2023-02-06T00:09:26.00000Z
custom:
Author: dependabot[bot]
PR: 6865

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Update pathspec requirement from ~=0.9.0 to >=0.9,<0.11 in /core"
time: 2022-09-23T00:06:46.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 5917

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Bump black from 22.8.0 to 22.10.0"
time: 2022-10-07T00:08:48.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 6019

View File

@@ -1,7 +0,0 @@
kind: "Dependency"
body: "Update colorama requirement from <0.4.6,>=0.3.9 to >=0.3.9,<0.4.7 in /core"
time: 2022-10-26T00:09:10.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 6144

View File

@@ -1,7 +0,0 @@
kind: Docs
body: minor doc correction
time: 2022-09-08T15:41:57.689162-04:00
custom:
Author: andy-clapson
Issue: "5791"
PR: "5684"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Generate API docs for new CLI interface
time: 2022-10-07T09:06:56.446078-05:00
custom:
Author: stu-k
Issue: "5528"
PR: "6022"

View File

@@ -1,6 +0,0 @@
kind: Docs
time: 2022-10-17T17:14:11.715348-05:00
custom:
Author: paulbenschmidt
Issue: "5880"
PR: "324"

View File

@@ -1,7 +0,0 @@
kind: Docs
body: Fix rendering of sample code for metrics
time: 2022-11-16T15:57:43.204201+01:00
custom:
Author: jtcohen6
Issue: "323"
PR: "346"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: update link to installation instructions
time: 2023-02-07T12:38:07.336783-05:00
custom:
Author: ryancharris
Issue: None

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Fix JSON path to overview docs
time: 2023-02-09T08:29:01.432616-07:00
custom:
Author: halvorlu
Issue: "366"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Added favor-state flag to optionally favor state nodes even if unselected node
exists
time: 2022-04-08T16:54:59.696564+01:00
custom:
Author: daniel-murray josephberni
Issue: "2968"
PR: "5859"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Proto logging messages
time: 2022-08-17T15:48:57.225267-04:00
custom:
Author: gshank
Issue: "5610"
PR: "5643"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Friendlier error messages when packages.yml is malformed
time: 2022-09-12T12:59:35.121188+01:00
custom:
Author: jared-rimmer
Issue: "5486"
PR: "5812"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Migrate dbt-utils current_timestamp macros into core + adapters
time: 2022-09-14T09:56:25.97818-07:00
custom:
Author: colin-rogers-dbt
Issue: "5521"
PR: "5838"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Allow partitions in external tables to be supplied as a list
time: 2022-09-25T21:16:51.051239654+02:00
custom:
Author: pgoslatara
Issue: "5929"
PR: "5930"

View File

@@ -1,7 +0,0 @@
kind: Features
body: extend -f flag shorthand for seed command
time: 2022-10-03T11:07:05.381632-05:00
custom:
Author: dave-connors-3
Issue: "5990"
PR: "5991"

View File

@@ -1,8 +0,0 @@
kind: Features
body: This pulls the profile name from args when constructing a RuntimeConfig in lib.py,
enabling the dbt-server to override the value that's in the dbt_project.yml
time: 2022-11-02T15:00:03.000805-05:00
custom:
Author: racheldaniel
Issue: "6201"
PR: "6202"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Added an md5 function to the base context
time: 2022-11-14T18:52:07.788593+02:00
custom:
Author: haritamar
Issue: "6246"
PR: "6247"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Have dbt debug spit out structured json logs with flags enabled.
time: 2023-01-07T00:31:57.516063-08:00
custom:
Author: versusfacit
Issue: "5353"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add adapter_response to dbt test and freshness result
time: 2023-01-18T23:38:01.857342+08:00
custom:
Author: aezomz
Issue: "2964"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Improve error message for packages missing `dbt_project.yml`
time: 2023-01-20T11:29:21.509967-07:00
custom:
Author: dbeatty10
Issue: "6663"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Adjust makefile to have clearer instructions for CI env var changes.
time: 2023-01-26T15:47:16.887327-08:00
custom:
Author: versusfacit
Issue: "6689"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Stand-alone Python module for PostgresColumn
time: 2023-01-27T16:28:12.212427-08:00
custom:
Author: nssalian
Issue: "6772"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Exposure owner requires one of name or email keys, and accepts additional arbitrary
keys
time: 2023-02-09T09:20:59.300272-05:00
custom:
Author: michelleark
Issue: "6833"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Account for disabled flags on models in schema files more completely
time: 2022-09-16T10:48:54.162273-05:00
custom:
Author: emmyoop
Issue: "3992"
PR: "5868"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add validation of enabled config for metrics, exposures and sources
time: 2022-10-10T11:32:18.752322-05:00
custom:
Author: emmyoop
Issue: "6030"
PR: "6038"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: check length of args of python model function before accessing it
time: 2022-10-11T16:07:15.464093-04:00
custom:
Author: chamini2
Issue: "6041"
PR: "6042"

View File

@@ -1,8 +0,0 @@
kind: Fixes
body: Add functors to ensure event types with str-type attributes are initialized
to spec, even when provided non-str type params.
time: 2022-10-16T17:37:42.846683-07:00
custom:
Author: versusfacit
Issue: "5436"
PR: "5874"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Allow hooks to fail without halting execution flow
time: 2022-11-07T09:53:14.340257-06:00
custom:
Author: ChenyuLInx
Issue: "5625"
PR: "6059"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: add merge_exclude_columns adapter tests
time: 2023-01-23T13:28:14.808748-06:00
custom:
Author: dave-connors-3
Issue: "6699"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Include adapter_response in NodeFinished run_result log event
time: 2023-01-24T11:58:37.74179-05:00
custom:
Author: gshank
Issue: "6703"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Sort cli vars before hashing for partial parsing
time: 2023-01-24T14:19:43.333628-05:00
custom:
Author: gshank
Issue: "6710"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: '[Regression] exposure_content referenced incorrectly'
time: 2023-01-25T19:17:39.942081-05:00
custom:
Author: Mathyoub
Issue: "6738"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Remove pin on packaging and stop using it for prerelease comparisons
time: 2023-02-01T15:44:18.279158-05:00
custom:
Author: gshank
Issue: "6834"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Readd depends_on.macros to SeedNode, to support seeds with hooks calling macros
time: 2023-02-03T13:55:57.853715+01:00
custom:
Author: jtcohen6
Issue: "6806"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix regression of --quiet cli parameter behavior
time: 2023-02-07T14:35:44.160163-05:00
custom:
Author: peterallenwebb
Issue: "6749"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Ensure results from hooks contain nodes when processing them
time: 2023-02-08T11:05:51.952494-06:00
custom:
Author: emmyoop
Issue: "6796"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Always flush stdout after logging
time: 2023-02-08T15:49:35.175874-05:00
custom:
Author: peterallenwebb
Issue: "6901"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Reapply logging fixes which were accidentally reverted
time: 2023-02-10T10:30:28.179997-05:00
custom:
Author: peterallenwebb
Issue: "6936"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Set relation_name in test nodes at compile time
time: 2023-02-10T19:41:57.386766-05:00
custom:
Author: gshank
Issue: "6930"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: hoist dbt.cli.main into dbt.main namespace
time: 2023-02-13T11:28:22.809596-08:00
custom:
Author: aranke
Issue: ''

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Readd initialization events, --log-cache-events in new CLI
time: 2023-02-13T13:05:22.989477+01:00
custom:
Author: jtcohen6
Issue: "6933"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix previous state tests and disabled exposures, metrics
time: 2023-02-13T17:07:23.185679-05:00
custom:
Author: gshank
Issue: 6752 6753

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Put black config in explicit config
time: 2022-09-27T19:42:59.241433-07:00
custom:
Author: max-sixty
Issue: "5946"
PR: "5947"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Added flat_graph attribute the Manifest class's deepcopy() coverage
time: 2022-09-29T13:44:06.275941-04:00
custom:
Author: peterallenwebb
Issue: "5809"
PR: "5975"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add mypy configs so `mypy` passes from CLI
time: 2022-10-05T12:03:10.061263-07:00
custom:
Author: max-sixty
Issue: "5983"
PR: "5983"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Exception message cleanup.
time: 2022-10-07T09:46:27.682872-05:00
custom:
Author: emmyoop
Issue: "6023"
PR: "6024"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Add dmypy cache to gitignore
time: 2022-10-07T14:00:44.227644-07:00
custom:
Author: max-sixty
Issue: "6028"
PR: "5978"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Provide useful errors when the value of 'materialized' is invalid
time: 2022-10-13T18:19:12.167548-04:00
custom:
Author: peterallenwebb
Issue: "5229"
PR: "6025"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fixed extra whitespace in strings introduced by black.
time: 2022-10-17T15:15:11.499246-05:00
custom:
Author: luke-bassett
Issue: "1350"
PR: "6086"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Clean up string formatting
time: 2022-10-17T15:58:44.676549-04:00
custom:
Author: eve-johns
Issue: "6068"
PR: "6082"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Remove the 'root_path' field from most nodes
time: 2022-10-28T10:48:37.687886-04:00
custom:
Author: gshank
Issue: "6171"
PR: "6172"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Combine certain logging events with different levels
time: 2022-10-28T11:03:44.887836-04:00
custom:
Author: gshank
Issue: "6173"
PR: "6174"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert threading tests to pytest
time: 2022-11-08T07:45:50.589147-06:00
custom:
Author: stu-k
Issue: "5942"
PR: "6226"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert postgres index tests to pytest
time: 2022-11-08T11:56:33.743042-06:00
custom:
Author: stu-k
Issue: "5770"
PR: "6228"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Convert use color tests to pytest
time: 2022-11-08T13:31:04.788547-06:00
custom:
Author: stu-k
Issue: "5771"
PR: "6230"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-921] dbt compile works in click'
time: 2023-01-11T14:51:43.324107-08:00
custom:
Author: aranke
Issue: "5545"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix use of ConnectionReused logging event
time: 2023-01-13T13:25:13.023168-05:00
custom:
Author: gshank
Issue: "6168"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Port docs tests to pytest
time: 2023-01-13T15:07:00.477038-05:00
custom:
Author: peterallenwebb
Issue: "6573"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update deprecated github action command
time: 2023-01-17T11:17:37.046095-06:00
custom:
Author: davidbloss
Issue: "6153"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt snapshot works in click
time: 2023-01-17T16:25:05.973769-08:00
custom:
Author: ChenyuLInx
Issue: "5554"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt list working with click
time: 2023-01-17T21:37:29.91632-05:00
custom:
Author: michelleark
Issue: "5549"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add dbt run-operation to click CLI
time: 2023-01-19T10:53:04.154871+01:00
custom:
Author: jtcohen6
Issue: "5552"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt build working with new click framework
time: 2023-01-19T20:56:50.50549-05:00
custom:
Author: michelleark
Issue: "5541"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt docs generate works with new click framework
time: 2023-01-19T21:10:40.698851-05:00
custom:
Author: michelleark
Issue: "5543"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Replaced the EmptyLine event with a more general Formatting event, and added
a Note event.
time: 2023-01-20T17:22:54.45828-05:00
custom:
Author: peterallenwebb
Issue: "6481"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Small optimization on manifest parsing benefitting large DAGs
time: 2023-01-22T21:52:35.549814+01:00
custom:
Author: boxysean
Issue: "6697"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Revised and simplified various structured logging events
time: 2023-01-24T15:35:53.065356-05:00
custom:
Author: peterallenwebb
Issue: 6664 6665 6666

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt init works with click
time: 2023-01-24T17:51:10.74065-05:00
custom:
Author: michelleark
Issue: "5548"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs commands'
time: 2023-01-25T04:11:36.57506-08:00
custom:
Author: aranke
Issue: 5544 6722

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Migrate debug task to click
time: 2023-01-25T10:26:06.735994-06:00
custom:
Author: stu-k
Issue: "5546"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: ' Optimized GraphQueue to remove graph analysis bottleneck in large dags.'
time: 2023-01-26T13:59:39.518345-05:00
custom:
Author: peterallenwebb
Issue: "6759"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Implement --version for click cli
time: 2023-01-26T14:31:02.740282-06:00
custom:
Author: stu-k
Issue: "6757"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-1841] Convert custom target test to Pytest'
time: 2023-01-26T16:47:41.198714-08:00
custom:
Author: aranke
Issue: "6638"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: "Enables the new Click Cli on the commandline! \U0001F680"
time: 2023-01-30T17:57:52.65626-06:00
custom:
Author: iknox-fa
Issue: "6784"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: warn_error/warn_error_options mutual exclusivity in click
time: 2023-01-30T18:09:17.240662-05:00
custom:
Author: michelleark
Issue: "6579"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Lazily call --version
time: 2023-01-31T14:18:06.02312-06:00
custom:
Author: stu-k
Issue: "6812"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Moving simple_seed to adapter zone to help adapter test conversions
time: 2023-02-03T14:35:51.481856-08:00
custom:
Author: nssalian
Issue: CT-1959

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: flags.THREADS defaults to None
time: 2023-02-07T16:51:11.011984-05:00
custom:
Author: michelleark
Issue: "6887"

View File

@@ -4,21 +4,70 @@ headerPath: header.tpl.md
versionHeaderPath: ""
changelogPath: CHANGELOG.md
versionExt: md
envPrefix: "CHANGIE_"
versionFormat: '## dbt-core {{.Version}} - {{.Time.Format "January 02, 2006"}}'
kindFormat: '### {{.Kind}}'
changeFormat: '- {{.Body}} ([#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), [#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
kinds:
- label: Breaking Changes
- label: Features
- label: Fixes
- label: Docs
changeFormat: '- {{.Body}} ([dbt-docs/#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-docs/issues/{{.Custom.Issue}}), [dbt-docs/#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-docs/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $IssueList := list }}
{{- $changes := splitList " " $.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[dbt-docs/#nbr](https://github.com/dbt-labs/dbt-docs/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: '- {{.Body}} ({{if ne .Custom.Issue ""}}[#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), {{end}}[#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: '- {{.Body}} ({{if ne .Custom.Issue ""}}[#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-core/issues/{{.Custom.Issue}}), {{end}}[#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-core/pull/{{.Custom.PR}}))'
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
@@ -33,42 +82,53 @@ custom:
type: string
minLength: 3
- key: Issue
label: GitHub Issue Number
type: int
minInt: 1
- key: PR
label: GitHub Pull Request Number
type: int
minInt: 1
label: GitHub Issue Number (separated by a single space if multiple)
type: string
minLength: 1
footerFormat: |
{{- $contributorDict := dict }}
{{- /* any names added to this list should be all lowercase for later matching purposes */}}
{{- $core_team := list "michelleark" "peterallenwebb" "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "dependabot[bot]" "snyk-bot" "colin-rogers-dbt" }}
{{- /* ensure all names in this list are all lowercase for later matching purposes */}}
{{- $core_team := splitList " " .Env.CORE_TEAM }}
{{- /* ensure we always skip snyk and dependabot in addition to the core team */}}
{{- $maintainers := list "dependabot[bot]" "snyk-bot"}}
{{- range $team_member := $core_team }}
{{- $team_member_lower := lower $team_member }}
{{- $maintainers = append $maintainers $team_member_lower }}
{{- end }}
{{- range $change := .Changes }}
{{- $authorList := splitList " " $change.Custom.Author }}
{{- /* loop through all authors for a PR */}}
{{- /* loop through all authors for a single changelog */}}
{{- range $author := $authorList }}
{{- $authorLower := lower $author }}
{{- /* we only want to include non-core team contributors */}}
{{- if not (has $authorLower $core_team)}}
{{- /* Docs kind link back to dbt-docs instead of dbt-core PRs */}}
{{- $prLink := $change.Kind }}
{{- if eq $change.Kind "Docs" }}
{{- $prLink = "[dbt-docs/#pr](https://github.com/dbt-labs/dbt-docs/pull/pr)" | replace "pr" $change.Custom.PR }}
{{- else }}
{{- $prLink = "[#pr](https://github.com/dbt-labs/dbt-core/pull/pr)" | replace "pr" $change.Custom.PR }}
{{- end }}
{{- /* check if this contributor has other PRs associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $prList := get $contributorDict $author }}
{{- $prList = append $prList $prLink }}
{{- $contributorDict := set $contributorDict $author $prList }}
{{- else }}
{{- $prList := list $prLink }}
{{- $contributorDict := set $contributorDict $author $prList }}
{{- end }}
{{- end}}
{{- if not (has $authorLower $maintainers)}}
{{- $changeList := splitList " " $change.Custom.Author }}
{{- $IssueList := list }}
{{- $changeLink := $change.Kind }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }}
{{- $changes := splitList " " $change.Custom.PR }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- else }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- end }}
{{- /* check if this contributor has other changes associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $contributionList := get $contributorDict $author }}
{{- $contributionList = concat $contributionList $IssueList }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- else }}
{{- $contributionList := $IssueList }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- end }}
{{- end}}
{{- end}}
{{- end }}
{{- /* no indentation here for formatting so the final markdown doesn't have unneeded indentations */}}

View File

@@ -9,4 +9,4 @@ ignore =
E203 # makes Flake8 work like black
E741
E501 # long line checking is done in black
exclude = test
exclude = test/

2
.gitattributes vendored Normal file
View File

@@ -0,0 +1,2 @@
core/dbt/include/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary

20
.github/_README.md vendored
View File

@@ -63,12 +63,12 @@ permissions:
contents: read
pull-requests: write
```
### Secrets
- When to use a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) vs the [GITHUB_TOKEN](https://docs.github.com/en/actions/security-guides/automatic-token-authentication) generated for the action?
The `GITHUB_TOKEN` is used by default. In most cases it is sufficient for what you need.
If you expect the workflow to result in a commit to that should retrigger workflows, you will need to use a Personal Access Token for the bot to commit the file. When using the GITHUB_TOKEN, the resulting commit will not trigger another GitHub Actions Workflow run. This is due to limitations set by GitHub. See [the docs](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#using-the-github_token-in-a-workflow) for a more detailed explanation.
For example, we must use a PAT in our workflow to commit a new changelog yaml file for bot PRs. Once the file has been committed to the branch, it should retrigger the check to validate that a changelog exists on the PR. Otherwise, it would stay in a failed state since the check would never retrigger.
@@ -105,7 +105,7 @@ Some triggers of note that we use:
```
# **what?**
# Describe what the action does.
# Describe what the action does.
# **why?**
# Why does this action exist?
@@ -138,7 +138,7 @@ Some triggers of note that we use:
id: fp
run: |
FILEPATH=.changes/unreleased/Dependencies-${{ steps.filename_time.outputs.time }}.yaml
echo "::set-output name=FILEPATH::$FILEPATH"
echo "FILEPATH=$FILEPATH" >> $GITHUB_OUTPUT
```
- Print out all variables you will reference as the first step of a job. This allows for easier debugging. The first job should log all inputs. Subsequent jobs should reference outputs of other jobs, if present.
@@ -158,14 +158,14 @@ Some triggers of note that we use:
echo "The build_script_path: ${{ inputs.build_script_path }}"
echo "The s3_bucket_name: ${{ inputs.s3_bucket_name }}"
echo "The package_test_command: ${{ inputs.package_test_command }}"
# collect all the variables that need to be used in subsequent jobs
- name: Set Variables
id: variables
run: |
echo "::set-output name=important_path::'performance/runner/Cargo.toml'"
echo "::set-output name=release_id::${{github.event.inputs.release_id}}"
echo "::set-output name=open_prs::${{github.event.inputs.open_prs}}"
echo "important_path='performance/runner/Cargo.toml'" >> $GITHUB_OUTPUT
echo "release_id=${{github.event.inputs.release_id}}" >> $GITHUB_OUTPUT
echo "open_prs=${{github.event.inputs.open_prs}}" >> $GITHUB_OUTPUT
job2:
needs: [job1]
@@ -190,7 +190,7 @@ ___
### Actions from the Marketplace
- Dont use external actions for things that can easily be accomplished manually.
- Always read through what an external action does before using it! Often an action in the GitHub Actions Marketplace can be replaced with a few lines in bash. This is much more maintainable (and wont change under us) and clear as to whats actually happening. It also prevents any
- Pin actions _we don't control_ to tags.
- Pin actions _we don't control_ to tags.
### Connecting to AWS
- Authenticate with the aws managed workflow
@@ -208,7 +208,7 @@ ___
```yaml
- name: Copy Artifacts from S3 via CLI
run: aws s3 cp ${{ env.s3_bucket }} . --recursive
run: aws s3 cp ${{ env.s3_bucket }} . --recursive
```
### Testing

View File

@@ -28,11 +28,12 @@ if __name__ == "__main__":
if package_request.status_code == 404:
if halt_on_missing:
sys.exit(1)
else:
# everything is the latest if the package doesn't exist
print(f"::set-output name=latest::{True}")
print(f"::set-output name=minor_latest::{True}")
sys.exit(0)
# everything is the latest if the package doesn't exist
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write("latest=True")
gh_output.write("minor_latest=True")
sys.exit(0)
# TODO: verify package meta is "correct"
# https://github.com/dbt-labs/dbt-core/issues/4640
@@ -91,5 +92,7 @@ if __name__ == "__main__":
latest = is_latest(pre_rel, new_version, current_latest)
minor_latest = is_latest(pre_rel, new_version, current_minor_latest)
print(f"::set-output name=latest::{latest}")
print(f"::set-output name=minor_latest::{minor_latest}")
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"latest={latest}")
gh_output.write(f"minor_latest={minor_latest}")

View File

@@ -40,7 +40,7 @@ jobs:
matrix:
include:
- label: "dependencies"
changie_kind: "Dependency"
changie_kind: "Dependencies"
- label: "snyk"
changie_kind: "Security"
runs-on: ubuntu-latest
@@ -58,4 +58,4 @@ jobs:
commit_message: "Add automated changelog yaml from template for bot PR"
changie_kind: ${{ matrix.changie_kind }}
label: ${{ matrix.label }}
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n Issue: 4904\n PR: ${{ github.event.pull_request.number }}"
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n PR: ${{ github.event.pull_request.number }}"

View File

@@ -0,0 +1,42 @@
# **what?**
# Cuts a new `*.latest` branch
# Also cleans up all files in `.changes/unreleased` and `.changes/previous verion on
# `main` and bumps `main` to the input version.
# **why?**
# Generally reduces the workload of engineers and reduces error. Allow automation.
# **when?**
# This will run when called manually.
name: Cut new release branch
on:
workflow_dispatch:
inputs:
version_to_bump_main:
description: 'The alpha version main should bump to (ex. 1.6.0a1)'
required: true
new_branch_name:
description: 'The full name of the new branch (ex. 1.5.latest)'
required: true
defaults:
run:
shell: bash
permissions:
contents: write
jobs:
cut_branch:
name: "Cut branch and clean up main for dbt-core"
# TODO: update to point at `main` once testing is complete
uses: dbt-labs/actions/.github/workflows/cut-release-branch.yaml@er/ct-495-cleanup
with:
version_to_bump_main: ${{ inputs.version_to_bump_main }}
new_branch_name: ${{ inputs.new_branch_name }}
PR_title: "Cleanup main after cutting new ${{ inputs.new_branch_name }} branch"
PR_body: "All adapter PRs will fail CI until the dbt-core PR has been merged due to release version conflicts."
secrets:
FISHTOWN_BOT_PAT: ${{ secrets.FISHTOWN_BOT_PAT }}

View File

@@ -0,0 +1,165 @@
# **what?**
# On push, if anything in core/dbt/docs or core/dbt/cli has been
# created or modified, regenerate the CLI API docs using sphinx.
# **why?**
# We watch for changes in core/dbt/cli because the CLI API docs rely on click
# and all supporting flags/params to be generated. We watch for changes in
# core/dbt/docs since any changes to sphinx configuration or any of the
# .rst files there could result in a differently build final index.html file.
# **when?**
# Whenever a change has been pushed to a branch, and only if there is a diff
# between the PR branch and main's core/dbt/cli and or core/dbt/docs dirs.
# TODO: add bot comment to PR informing contributor that the docs have been committed
# TODO: figure out why github action triggered pushes cause github to fail to report
# the status of jobs
name: Generate CLI API docs
on:
pull_request:
permissions:
contents: write
pull-requests: write
env:
CLI_DIR: ${{ github.workspace }}/core/dbt/cli
DOCS_DIR: ${{ github.workspace }}/core/dbt/docs
DOCS_BUILD_DIR: ${{ github.workspace }}/core/dbt/docs/build
jobs:
check_gen:
name: check if generation needed
runs-on: ubuntu-latest
if: ${{ github.event.pull_request.head.repo.fork == false }}
outputs:
cli_dir_changed: ${{ steps.check_cli.outputs.cli_dir_changed }}
docs_dir_changed: ${{ steps.check_docs.outputs.docs_dir_changed }}
steps:
- name: "[DEBUG] print variables"
run: |
echo "env.CLI_DIR: ${{ env.CLI_DIR }}"
echo "env.DOCS_BUILD_DIR: ${{ env.DOCS_BUILD_DIR }}"
echo "env.DOCS_DIR: ${{ env.DOCS_DIR }}"
- name: git checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
ref: ${{ github.head_ref }}
- name: set shas
id: set_shas
run: |
THIS_SHA=$(git rev-parse @)
LAST_SHA=$(git rev-parse @~1)
echo "this sha: $THIS_SHA"
echo "last sha: $LAST_SHA"
echo "this_sha=$THIS_SHA" >> $GITHUB_OUTPUT
echo "last_sha=$LAST_SHA" >> $GITHUB_OUTPUT
- name: check for changes in core/dbt/cli
id: check_cli
run: |
CLI_DIR_CHANGES=$(git diff \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.CLI_DIR }})
if [ -n "$CLI_DIR_CHANGES" ]; then
echo "changes found"
echo $CLI_DIR_CHANGES
echo "cli_dir_changed=true" >> $GITHUB_OUTPUT
exit 0
fi
echo "cli_dir_changed=false" >> $GITHUB_OUTPUT
echo "no changes found"
- name: check for changes in core/dbt/docs
id: check_docs
if: steps.check_cli.outputs.cli_dir_changed == 'false'
run: |
DOCS_DIR_CHANGES=$(git diff --name-only \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.DOCS_DIR }} ':!${{ env.DOCS_BUILD_DIR }}')
DOCS_BUILD_DIR_CHANGES=$(git diff --name-only \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.DOCS_BUILD_DIR }})
if [ -n "$DOCS_DIR_CHANGES" ] && [ -z "$DOCS_BUILD_DIR_CHANGES" ]; then
echo "changes found"
echo $DOCS_DIR_CHANGES
echo "docs_dir_changed=true" >> $GITHUB_OUTPUT
exit 0
fi
echo "docs_dir_changed=false" >> $GITHUB_OUTPUT
echo "no changes found"
gen_docs:
name: generate docs
runs-on: ubuntu-latest
needs: [check_gen]
if: |
needs.check_gen.outputs.cli_dir_changed == 'true'
|| needs.check_gen.outputs.docs_dir_changed == 'true'
steps:
- name: "[DEBUG] print variables"
run: |
echo "env.DOCS_DIR: ${{ env.DOCS_DIR }}"
echo "github head_ref: ${{ github.head_ref }}"
- name: git checkout
uses: actions/checkout@v3
with:
ref: ${{ github.head_ref }}
- name: install python
uses: actions/setup-python@v4.3.0
with:
python-version: 3.8
- name: install dev requirements
run: |
python3 -m venv env
source env/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt -r dev-requirements.txt
- name: generate docs
run: |
source env/bin/activate
cd ${{ env.DOCS_DIR }}
echo "cleaning existing docs"
make clean
echo "creating docs"
make html
- name: debug
run: |
echo ">>>>> status"
git status
echo ">>>>> remotes"
git remote -v
echo ">>>>> branch"
git branch -v
echo ">>>>> log"
git log --pretty=oneline | head -5
- name: commit docs
run: |
git config user.name 'Github Build Bot'
git config user.email 'buildbot@fishtownanalytics.com'
git commit -am "Add generated CLI API docs"
git push -u origin ${{ github.head_ref }}

View File

@@ -73,7 +73,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
env:
TOXENV: "unit"
@@ -101,7 +101,9 @@ jobs:
- name: Get current date
if: always()
id: date
run: echo "::set-output name=date::$(date +'%Y-%m-%dT%H_%M_%S')" #no colons allowed for artifacts
run: |
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v2
if: always()
@@ -118,8 +120,8 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10"]
os: [ubuntu-latest]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-20.04]
include:
- python-version: 3.8
os: windows-latest
@@ -168,7 +170,9 @@ jobs:
- name: Get current date
if: always()
id: date
run: echo "::set-output name=date::$(date +'%Y_%m_%dT%H_%M_%S')" #no colons allowed for artifacts
run: |
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v2
if: always()

109
.github/workflows/nightly-release.yml vendored Normal file
View File

@@ -0,0 +1,109 @@
# **what?**
# Nightly releases to GitHub and PyPI. This workflow produces the following outcome:
# - generate and validate data for night release (commit SHA, version number, release branch);
# - pass data to release workflow;
# - night release will be pushed to GitHub as a draft release;
# - night build will be pushed to test PyPI;
#
# **why?**
# Ensure an automated and tested release process for nightly builds
#
# **when?**
# This workflow runs on schedule or can be run manually on demand.
name: Nightly Test Release to GitHub and PyPI
on:
workflow_dispatch: # for manual triggering
schedule:
- cron: 0 9 * * *
permissions:
contents: write # this is the permission that allows creating a new release
defaults:
run:
shell: bash
env:
RELEASE_BRANCH: "main"
jobs:
aggregate-release-data:
runs-on: ubuntu-latest
outputs:
commit_sha: ${{ steps.resolve-commit-sha.outputs.release_commit }}
version_number: ${{ steps.nightly-release-version.outputs.number }}
release_branch: ${{ steps.release-branch.outputs.name }}
steps:
- name: "Checkout ${{ github.repository }} Branch ${{ env.RELEASE_BRANCH }}"
uses: actions/checkout@v3
with:
ref: ${{ env.RELEASE_BRANCH }}
- name: "Resolve Commit To Release"
id: resolve-commit-sha
run: |
commit_sha=$(git rev-parse HEAD)
echo "release_commit=$commit_sha" >> $GITHUB_OUTPUT
- name: "Get Current Version Number"
id: version-number-sources
run: |
current_version=`awk -F"current_version = " '{print $2}' .bumpversion.cfg | tr '\n' ' '`
echo "current_version=$current_version" >> $GITHUB_OUTPUT
- name: "Audit Version And Parse Into Parts"
id: semver
uses: dbt-labs/actions/parse-semver@v1.1.0
with:
version: ${{ steps.version-number-sources.outputs.current_version }}
- name: "Get Current Date"
id: current-date
run: echo "date=$(date +'%m%d%Y')" >> $GITHUB_OUTPUT
- name: "Generate Nightly Release Version Number"
id: nightly-release-version
run: |
number="${{ steps.semver.outputs.version }}.dev${{ steps.current-date.outputs.date }}"
echo "number=$number" >> $GITHUB_OUTPUT
- name: "Audit Nightly Release Version And Parse Into Parts"
uses: dbt-labs/actions/parse-semver@v1.1.0
with:
version: ${{ steps.nightly-release-version.outputs.number }}
- name: "Set Release Branch"
id: release-branch
run: |
echo "name=${{ env.RELEASE_BRANCH }}" >> $GITHUB_OUTPUT
log-outputs-aggregate-release-data:
runs-on: ubuntu-latest
needs: [aggregate-release-data]
steps:
- name: "[DEBUG] Log Outputs"
run: |
echo commit_sha : ${{ needs.aggregate-release-data.outputs.commit_sha }}
echo version_number: ${{ needs.aggregate-release-data.outputs.version_number }}
echo release_branch: ${{ needs.aggregate-release-data.outputs.release_branch }}
release-github-pypi:
needs: [aggregate-release-data]
uses: ./.github/workflows/release.yml
with:
sha: ${{ needs.aggregate-release-data.outputs.commit_sha }}
target_branch: ${{ needs.aggregate-release-data.outputs.release_branch }}
version_number: ${{ needs.aggregate-release-data.outputs.version_number }}
build_script_path: "scripts/build-dist.sh"
env_setup_script_path: "scripts/env-setup.sh"
s3_bucket_name: "core-team-artifacts"
package_test_command: "dbt --version"
test_run: true
nightly_release: true
secrets: inherit

View File

@@ -28,7 +28,33 @@ on:
permissions: read-all
jobs:
fetch-latest-branches:
runs-on: ubuntu-latest
outputs:
latest-branches: ${{ steps.get-latest-branches.outputs.repo-branches }}
steps:
- name: "Fetch dbt-core Latest Branches"
uses: dbt-labs/actions/fetch-repo-branches@v1.1.1
id: get-latest-branches
with:
repo_name: ${{ github.event.repository.name }}
organization: "dbt-labs"
pat: ${{ secrets.GITHUB_TOKEN }}
fetch_protected_branches_only: true
regex: "^1.[0-9]+.latest$"
perform_match_method: "match"
retries: 3
- name: "[ANNOTATION] ${{ github.event.repository.name }} - branches to test"
run: |
title="${{ github.event.repository.name }} - branches to test"
message="The workflow will run tests for the following branches of the ${{ github.event.repository.name }} repo: ${{ steps.get-latest-branches.outputs.repo-branches }}"
echo "::notice $title::$message"
kick-off-ci:
needs: [fetch-latest-branches]
name: Kick-off CI
runs-on: ubuntu-latest
@@ -39,7 +65,9 @@ jobs:
max-parallel: 1
fail-fast: false
matrix:
branch: [1.0.latest, 1.1.latest, 1.2.latest, 1.3.latest, main]
branch: ${{ fromJSON(needs.fetch-latest-branches.outputs.latest-branches) }}
include:
- branch: 'main'
steps:
- name: Call CI workflow for ${{ matrix.branch }} branch

View File

@@ -41,9 +41,9 @@ jobs:
id: version
run: |
IFS="." read -r MAJOR MINOR PATCH <<< ${{ github.event.inputs.version_number }}
echo "::set-output name=major::$MAJOR"
echo "::set-output name=minor::$MINOR"
echo "::set-output name=patch::$PATCH"
echo "major=$MAJOR" >> $GITHUB_OUTPUT
echo "minor=$MINOR" >> $GITHUB_OUTPUT
echo "patch=$PATCH" >> $GITHUB_OUTPUT
- name: Is pkg 'latest'
id: latest
@@ -70,8 +70,10 @@ jobs:
- name: Get docker build arg
id: build_arg
run: |
echo "::set-output name=build_arg_name::"$(echo ${{ github.event.inputs.package }} | sed 's/\-/_/g')
echo "::set-output name=build_arg_value::"$(echo ${{ github.event.inputs.package }} | sed 's/postgres/core/g')
BUILD_ARG_NAME=$(echo ${{ github.event.inputs.package }} | sed 's/\-/_/g')
BUILD_ARG_VALUE=$(echo ${{ github.event.inputs.package }} | sed 's/postgres/core/g')
echo "build_arg_name=$BUILD_ARG_NAME" >> $GITHUB_OUTPUT
echo "build_arg_value=$BUILD_ARG_VALUE" >> $GITHUB_OUTPUT
- name: Log in to the GHCR
uses: docker/login-action@v1

View File

@@ -1,24 +1,110 @@
# **what?**
# Take the given commit, run unit tests specifically on that sha, build and
# package it, and then release to GitHub and PyPi with that specific build
# Release workflow provides the following steps:
# - checkout the given commit;
# - validate version in sources and changelog file for given version;
# - bump the version and generate a changelog if needed;
# - merge all changes to the target branch if needed;
# - run unit and integration tests against given commit;
# - build and package that SHA;
# - release it to GitHub and PyPI with that specific build;
#
# **why?**
# Ensure an automated and tested release process
#
# **when?**
# This will only run manually with a given sha and version
# This workflow can be run manually on demand or can be called by other workflows
name: Release to GitHub and PyPi
name: Release to GitHub and PyPI
on:
workflow_dispatch:
inputs:
sha:
description: 'The last commit sha in the release'
required: true
description: "The last commit sha in the release"
type: string
required: true
target_branch:
description: "The branch to release from"
type: string
required: true
version_number:
description: 'The release version number (i.e. 1.0.0b1)'
required: true
description: "The release version number (i.e. 1.0.0b1)"
type: string
required: true
build_script_path:
description: "Build script path"
type: string
default: "scripts/build-dist.sh"
required: true
env_setup_script_path:
description: "Environment setup script path"
type: string
default: "scripts/env-setup.sh"
required: false
s3_bucket_name:
description: "AWS S3 bucket name"
type: string
default: "core-team-artifacts"
required: true
package_test_command:
description: "Package test command"
type: string
default: "dbt --version"
required: true
test_run:
description: "Test run (Publish release as draft)"
type: boolean
default: true
required: false
nightly_release:
description: "Nightly release to dev environment"
type: boolean
default: false
required: false
workflow_call:
inputs:
sha:
description: "The last commit sha in the release"
type: string
required: true
target_branch:
description: "The branch to release from"
type: string
required: true
version_number:
description: "The release version number (i.e. 1.0.0b1)"
type: string
required: true
build_script_path:
description: "Build script path"
type: string
default: "scripts/build-dist.sh"
required: true
env_setup_script_path:
description: "Environment setup script path"
type: string
default: "scripts/env-setup.sh"
required: false
s3_bucket_name:
description: "AWS S3 bucket name"
type: string
default: "core-team-artifacts"
required: true
package_test_command:
description: "Package test command"
type: string
default: "dbt --version"
required: true
test_run:
description: "Test run (Publish release as draft)"
type: boolean
default: true
required: false
nightly_release:
description: "Nightly release to dev environment"
type: boolean
default: false
required: false
permissions:
contents: write # this is the permission that allows creating a new release
@@ -28,175 +114,116 @@ defaults:
shell: bash
jobs:
unit:
name: Unit test
log-inputs:
name: Log Inputs
runs-on: ubuntu-latest
env:
TOXENV: "unit"
steps:
- name: Check out the repository
uses: actions/checkout@v2
with:
persist-credentials: false
ref: ${{ github.event.inputs.sha }}
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
- name: "[DEBUG] Print Variables"
run: |
pip install --user --upgrade pip
pip install tox
pip --version
tox --version
echo The last commit sha in the release: ${{ inputs.sha }}
echo The branch to release from: ${{ inputs.target_branch }}
echo The release version number: ${{ inputs.version_number }}
echo Build script path: ${{ inputs.build_script_path }}
echo Environment setup script path: ${{ inputs.env_setup_script_path }}
echo AWS S3 bucket name: ${{ inputs.s3_bucket_name }}
echo Package test command: ${{ inputs.package_test_command }}
echo Test run: ${{ inputs.test_run }}
echo Nightly release: ${{ inputs.nightly_release }}
- name: Run tox
run: tox
bump-version-generate-changelog:
name: Bump package version, Generate changelog
build:
name: build packages
uses: dbt-labs/dbt-release/.github/workflows/release-prep.yml@main
with:
sha: ${{ inputs.sha }}
version_number: ${{ inputs.version_number }}
target_branch: ${{ inputs.target_branch }}
env_setup_script_path: ${{ inputs.env_setup_script_path }}
test_run: ${{ inputs.test_run }}
nightly_release: ${{ inputs.nightly_release }}
secrets: inherit
log-outputs-bump-version-generate-changelog:
name: "[Log output] Bump package version, Generate changelog"
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog]
runs-on: ubuntu-latest
steps:
- name: Check out the repository
uses: actions/checkout@v2
with:
persist-credentials: false
ref: ${{ github.event.inputs.sha }}
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
- name: Print variables
run: |
pip install --user --upgrade pip
pip install --upgrade setuptools wheel twine check-wheel-contents
pip --version
echo Final SHA : ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
echo Changelog path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
- name: Build distributions
run: ./scripts/build-dist.sh
build-test-package:
name: Build, Test, Package
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog]
- name: Show distributions
run: ls -lh dist/
uses: dbt-labs/dbt-release/.github/workflows/build.yml@main
- name: Check distribution descriptions
run: |
twine check dist/*
with:
sha: ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
changelog_path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
build_script_path: ${{ inputs.build_script_path }}
s3_bucket_name: ${{ inputs.s3_bucket_name }}
package_test_command: ${{ inputs.package_test_command }}
test_run: ${{ inputs.test_run }}
nightly_release: ${{ inputs.nightly_release }}
- name: Check wheel contents
run: |
check-wheel-contents dist/*.whl --ignore W007,W008
- uses: actions/upload-artifact@v2
with:
name: dist
path: |
dist/
!dist/dbt-${{github.event.inputs.version_number}}.tar.gz
test-build:
name: verify packages
needs: [build, unit]
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install python dependencies
run: |
pip install --user --upgrade pip
pip install --upgrade wheel
pip --version
- uses: actions/download-artifact@v2
with:
name: dist
path: dist/
- name: Show distributions
run: ls -lh dist/
- name: Install wheel distributions
run: |
find ./dist/*.whl -maxdepth 1 -type f | xargs pip install --force-reinstall --find-links=dist/
- name: Check wheel distributions
run: |
dbt --version
- name: Install source distributions
run: |
find ./dist/*.gz -maxdepth 1 -type f | xargs pip install --force-reinstall --find-links=dist/
- name: Check source distributions
run: |
dbt --version
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
github-release:
name: GitHub Release
if: ${{ !failure() && !cancelled() }}
needs: test-build
needs: [bump-version-generate-changelog, build-test-package]
runs-on: ubuntu-latest
uses: dbt-labs/dbt-release/.github/workflows/github-release.yml@main
steps:
- uses: actions/download-artifact@v2
with:
name: dist
path: '.'
# Need to set an output variable because env variables can't be taken as input
# This is needed for the next step with releasing to GitHub
- name: Find release type
id: release_type
env:
IS_PRERELEASE: ${{ contains(github.event.inputs.version_number, 'rc') || contains(github.event.inputs.version_number, 'b') }}
run: |
echo ::set-output name=isPrerelease::$IS_PRERELEASE
- name: Creating GitHub Release
uses: softprops/action-gh-release@v1
with:
name: dbt-core v${{github.event.inputs.version_number}}
tag_name: v${{github.event.inputs.version_number}}
prerelease: ${{ steps.release_type.outputs.isPrerelease }}
target_commitish: ${{github.event.inputs.sha}}
body: |
[Release notes](https://github.com/dbt-labs/dbt-core/blob/main/CHANGELOG.md)
files: |
dbt_postgres-${{github.event.inputs.version_number}}-py3-none-any.whl
dbt_core-${{github.event.inputs.version_number}}-py3-none-any.whl
dbt-postgres-${{github.event.inputs.version_number}}.tar.gz
dbt-core-${{github.event.inputs.version_number}}.tar.gz
with:
sha: ${{ needs.bump-version-generate-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
changelog_path: ${{ needs.bump-version-generate-changelog.outputs.changelog_path }}
test_run: ${{ inputs.test_run }}
pypi-release:
name: Pypi release
name: PyPI Release
runs-on: ubuntu-latest
needs: [github-release]
needs: github-release
uses: dbt-labs/dbt-release/.github/workflows/pypi-release.yml@main
environment: PypiProd
steps:
- uses: actions/download-artifact@v2
with:
name: dist
path: 'dist'
with:
version_number: ${{ inputs.version_number }}
test_run: ${{ inputs.test_run }}
- name: Publish distribution to PyPI
uses: pypa/gh-action-pypi-publish@v1.4.2
with:
password: ${{ secrets.PYPI_API_TOKEN }}
secrets:
PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
TEST_PYPI_API_TOKEN: ${{ secrets.TEST_PYPI_API_TOKEN }}
slack-notification:
name: Slack Notification
if: ${{ failure() && (!inputs.test_run || inputs.nightly_release) }}
needs:
[
bump-version-generate-changelog,
build-test-package,
github-release,
pypi-release,
]
uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}

View File

@@ -9,13 +9,4 @@ permissions:
jobs:
stale:
runs-on: ubuntu-latest
steps:
# pinned at v4 (https://github.com/actions/stale/releases/tag/v4.0.0)
- uses: actions/stale@cdf15f641adb27a71842045a94023bef6945e3aa
with:
stale-issue-message: "This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days."
stale-pr-message: "This PR has been marked as Stale because it has been open for 180 days with no activity. If you would like the PR to remain open, please remove the stale label or comment on the PR, or it will be closed in 7 days."
close-issue-message: "Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers."
# mark issues/PRs stale when they haven't seen activity in 180 days
days-before-stale: 180
uses: dbt-labs/actions/.github/workflows/stale-bot-matrix.yml@main

View File

@@ -22,7 +22,7 @@ jobs:
# run the performance measurements on the current or default branch
test-schema:
name: Test Log Schema
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
env:
# turns warnings into errors
RUSTFLAGS: "-D warnings"

View File

@@ -20,106 +20,9 @@ on:
description: 'The version number to bump to (ex. 1.2.0, 1.3.0b1)'
required: true
permissions:
contents: write
pull-requests: write
jobs:
bump:
runs-on: ubuntu-latest
steps:
- name: "[DEBUG] Print Variables"
run: |
echo "all variables defined as inputs"
echo The version_number: ${{ github.event.inputs.version_number }}
- name: Check out the repository
uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: Install python dependencies
run: |
python3 -m venv env
source env/bin/activate
pip install --upgrade pip
- name: Add Homebrew to PATH
run: |
echo "/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin" >> $GITHUB_PATH
- name: Install Homebrew packages
run: |
brew install pre-commit
brew tap miniscruff/changie https://github.com/miniscruff/changie
brew install changie
- name: Audit Version and Parse Into Parts
id: semver
uses: dbt-labs/actions/parse-semver@v1
with:
version: ${{ github.event.inputs.version_number }}
- name: Set branch value
id: variables
run: |
echo "::set-output name=BRANCH_NAME::prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID"
- name: Create PR branch
run: |
git checkout -b ${{ steps.variables.outputs.BRANCH_NAME }}
git push origin ${{ steps.variables.outputs.BRANCH_NAME }}
git branch --set-upstream-to=origin/${{ steps.variables.outputs.BRANCH_NAME }} ${{ steps.variables.outputs.BRANCH_NAME }}
- name: Bump version
run: |
source env/bin/activate
pip install -r dev-requirements.txt
env/bin/bumpversion --allow-dirty --new-version ${{ github.event.inputs.version_number }} major
git status
- name: Run changie
run: |
if [[ ${{ steps.semver.outputs.is-pre-release }} -eq 1 ]]
then
changie batch ${{ steps.semver.outputs.base-version }} --move-dir '${{ steps.semver.outputs.base-version }}' --prerelease '${{ steps.semver.outputs.pre-release }}'
else
changie batch ${{ steps.semver.outputs.base-version }} --include '${{ steps.semver.outputs.base-version }}' --remove-prereleases
fi
changie merge
git status
# this step will fail on whitespace errors but also correct them
- name: Remove trailing whitespace
continue-on-error: true
run: |
pre-commit run trailing-whitespace --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
# this step will fail on newline errors but also correct them
- name: Removing extra newlines
continue-on-error: true
run: |
pre-commit run end-of-file-fixer --files .bumpversion.cfg CHANGELOG.md .changes/*
git status
- name: Commit version bump to branch
uses: EndBug/add-and-commit@v7
with:
author_name: 'Github Build Bot'
author_email: 'buildbot@fishtownanalytics.com'
message: 'Bumping version to ${{ github.event.inputs.version_number }} and generate CHANGELOG'
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
push: 'origin origin/${{ steps.variables.outputs.BRANCH_NAME }}'
- name: Create Pull Request
uses: peter-evans/create-pull-request@v3
with:
author: 'Github Build Bot <buildbot@fishtownanalytics.com>'
base: ${{github.ref}}
title: 'Bumping version to ${{ github.event.inputs.version_number }} and generate changelog'
branch: '${{ steps.variables.outputs.BRANCH_NAME }}'
labels: |
Skip Changelog
version_bump_and_changie:
uses: dbt-labs/actions/.github/workflows/version-bump.yml@main
with:
version_number: ${{ inputs.version_number }}
secrets: inherit # ok since what we are calling is internally maintained

2
.gitignore vendored
View File

@@ -11,6 +11,7 @@ __pycache__/
env*/
dbt_env/
build/
!core/dbt/docs/build
develop-eggs/
dist/
downloads/
@@ -50,6 +51,7 @@ coverage.xml
*,cover
.hypothesis/
test.env
makefile.test.env
*.pytest_cache/

View File

@@ -2,7 +2,7 @@
# Eventually the hooks described here will be run as tests before merging each PR.
# TODO: remove global exclusion of tests when testing overhaul is complete
exclude: ^test/
exclude: ^(test/|core/dbt/docs/build/)
# Force all unspecified python hooks to run python 3.8
default_language_version:

View File

@@ -5,12 +5,12 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## Previous Releases
For information on prior major and minor releases, see their changelogs:
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)

Some files were not shown because too many files have changed in this diff Show More