Compare commits

..

45 Commits

Author SHA1 Message Date
Callum McCann
ec1c9f0362 fixing metricflow not being present 2023-03-13 17:34:02 -07:00
Callum McCann
caeecf4a67 switching to postgres 2023-03-09 10:29:27 -08:00
Callum McCann
fcea5969ae adding metricflow as context macro library 2023-03-07 09:12:27 -08:00
Callum McCann
672a7d7fc8 straight dbt query 2023-03-03 08:43:43 -08:00
Callum McCann
a033aa0180 more updates for mypy 2023-03-02 15:19:41 -08:00
Callum McCann
38991cd3a9 updates for mypy 2023-02-28 13:14:10 -08:00
Callum McCann
4f966b0d54 fixes for precommit 2023-02-28 10:20:10 -08:00
Callum McCann
d3eaa37212 pre-commit fixes 2023-02-28 09:41:56 -08:00
Callum McCann
ce1759f793 fixing metric input 2023-02-28 09:37:01 -08:00
Callum McCann
4b3e797530 cli embedding and moving UCM 2023-02-27 15:43:23 -08:00
Callum McCann
3e9ed1ff9b Merge branch 'main' into merging-main 2023-02-27 11:58:16 -08:00
Callum McCann
9b0dce32b8 testing metricflow 2023-02-27 11:42:45 -08:00
Callum McCann
ce8e886f38 changing to snowflake for mf compatability 2023-02-26 14:31:31 -08:00
Callum McCann
12d02dc9d1 allowing hashable metric inputs 2023-02-26 14:24:04 -08:00
Callum McCann
72f59da9df renaming element name to name 2023-02-23 16:58:30 -06:00
Callum McCann
0dd99eac83 renaming element_name to name 2023-02-23 16:58:21 -06:00
Callum McCann
f0564f9d32 finally got constraints working 2023-02-23 14:28:28 -06:00
Callum McCann
278e4c7673 adding constraints but they do not work yet 2023-02-22 15:04:17 -06:00
Callum McCann
e3ec07d035 removing unused imports 2023-02-21 10:14:41 -06:00
Callum McCann
c7aa2ed7ef moving UCM 2023-02-21 09:55:11 -06:00
Callum McCann
e49e259950 Fixing imports+adding ModelValidation to manifest 2023-02-20 16:36:00 -06:00
Callum McCann
140597276b Renaming to aggregation state 2023-02-20 14:56:26 -06:00
Callum McCann
6712a5841a adding constraint and renaming agg file 2023-02-20 14:52:14 -06:00
Callum McCann
f4356d8dd2 adding entity origin 2023-02-17 15:08:37 -06:00
Callum McCann
7c715c5625 renaming folder 2023-02-17 14:47:01 -06:00
Callum McCann
5b9a24fd23 moving objects into contract graphs 2023-02-17 14:44:57 -06:00
Callum McCann
6378c13e7a fixing some errors on lookup 2023-02-17 13:24:59 -06:00
Callum McCann
2db94c5788 adding warning messages 2023-02-17 08:52:37 -06:00
Callum McCann
f25c8f39fc adding model as property 2023-02-17 08:34:53 -06:00
Callum McCann
3b8b191623 whitespace 2023-02-16 19:21:10 -06:00
Callum McCann
246fd66e8e finalizing validation 2023-02-16 19:05:12 -06:00
Callum McCann
817d39ac14 adding more rules :dance: 2023-02-16 16:21:04 -06:00
Callum McCann
85e27ac747 adding model validator and first 2 rules 2023-02-16 15:15:39 -06:00
Callum McCann
b5ca2e4c5f validation for unique names per entity 2023-02-15 16:20:51 -06:00
Callum McCann
e69b465c41 adding validation for metrics 2023-02-15 13:04:18 -06:00
Callum McCann
6937b321d6 fixing metric reference 2023-02-15 09:16:24 -06:00
Callum McCann
a6fc443abc adding user config model to manifest 2023-02-15 09:03:14 -06:00
Callum McCann
340cae3b43 moving semantics out 2023-02-14 13:03:18 -06:00
Callum McCann
91c5e2cc86 renaming attributes 2023-02-13 16:01:55 -06:00
Callum McCann
bafae0326b folder change and renaming back to expr 2023-02-13 08:49:44 -06:00
Callum McCann
7e1b788bd8 adding primary time aggregation rule 2023-02-11 21:40:40 -06:00
Callum McCann
1bd2fe09a1 renaming to transform model and adding boolean 2023-02-11 21:34:23 -06:00
Callum McCann
5b0197635d second commit - unclean commit sorry future me 2023-02-11 21:24:42 -06:00
Callum McCann
c1ad7b0f0e first commit - adding base constructs/classes 2023-02-10 08:45:15 -06:00
Callum McCann
2da925aa25 Adding entity node to core (#6648)
* first draft

* finishing first commit

* adding testing project

* adding changie

* cleaning

* removing blocks

* fixing proto error message

* updates to events

* fixing issues

* adding test dimension

* updating schemas

* updating manfiest.json

* removing old versions from compatability

* updating

* fixes

* fixing more bugs caught by tests

* updating tests
2023-01-31 09:03:06 -06:00
564 changed files with 30098 additions and 74689 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.7.0a1
current_version = 1.5.0b1
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

View File

@@ -3,8 +3,6 @@
For information on prior major and minor releases, see their changelogs:
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)

80
.changes/1.5.0-b1.md Normal file
View File

@@ -0,0 +1,80 @@
## dbt-core 1.5.0-b1 - February 17, 2023
### Features
- Data type constraints are now native to SQL table materializations. Enforce columns are specific data types and not null depending on database functionality. ([#6079](https://github.com/dbt-labs/dbt-core/issues/6079))
- Have dbt debug spit out structured json logs with flags enabled. ([#5353](https://github.com/dbt-labs/dbt-core/issues/5353))
- add adapter_response to dbt test and freshness result ([#2964](https://github.com/dbt-labs/dbt-core/issues/2964))
- Improve error message for packages missing `dbt_project.yml` ([#6663](https://github.com/dbt-labs/dbt-core/issues/6663))
- Adjust makefile to have clearer instructions for CI env var changes. ([#6689](https://github.com/dbt-labs/dbt-core/issues/6689))
- Stand-alone Python module for PostgresColumn ([#6772](https://github.com/dbt-labs/dbt-core/issues/6772))
- Exposure owner requires one of name or email keys, and accepts additional arbitrary keys ([#6833](https://github.com/dbt-labs/dbt-core/issues/6833))
- Parse 'group' resource ([#6921](https://github.com/dbt-labs/dbt-core/issues/6921))
### Fixes
- add merge_exclude_columns adapter tests ([#6699](https://github.com/dbt-labs/dbt-core/issues/6699))
- Include adapter_response in NodeFinished run_result log event ([#6703](https://github.com/dbt-labs/dbt-core/issues/6703))
- Sort cli vars before hashing for partial parsing ([#6710](https://github.com/dbt-labs/dbt-core/issues/6710))
- [Regression] exposure_content referenced incorrectly ([#6738](https://github.com/dbt-labs/dbt-core/issues/6738))
- Remove pin on packaging and stop using it for prerelease comparisons ([#6834](https://github.com/dbt-labs/dbt-core/issues/6834))
- Readd depends_on.macros to SeedNode, to support seeds with hooks calling macros ([#6806](https://github.com/dbt-labs/dbt-core/issues/6806))
- Fix regression of --quiet cli parameter behavior ([#6749](https://github.com/dbt-labs/dbt-core/issues/6749))
- Ensure results from hooks contain nodes when processing them ([#6796](https://github.com/dbt-labs/dbt-core/issues/6796))
- Always flush stdout after logging ([#6901](https://github.com/dbt-labs/dbt-core/issues/6901))
- Reapply logging fixes which were accidentally reverted ([#6936](https://github.com/dbt-labs/dbt-core/issues/6936))
- Set relation_name in test nodes at compile time ([#6930](https://github.com/dbt-labs/dbt-core/issues/6930))
- Readd initialization events, --log-cache-events in new CLI ([#6933](https://github.com/dbt-labs/dbt-core/issues/6933))
- Fix previous state tests and disabled exposures, metrics ([#6752](https://github.com/dbt-labs/dbt-core/issues/6752), [#6753](https://github.com/dbt-labs/dbt-core/issues/6753))
- Make use of hashlib.md5() FIPS compliant ([#6900](https://github.com/dbt-labs/dbt-core/issues/6900))
### Docs
- update link to installation instructions ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Fix JSON path to overview docs ([dbt-docs/#366](https://github.com/dbt-labs/dbt-docs/issues/366))
- Searchable column descriptions ([dbt-docs/#140](https://github.com/dbt-labs/dbt-docs/issues/140), [dbt-docs/#322](https://github.com/dbt-labs/dbt-docs/issues/322), [dbt-docs/#369](https://github.com/dbt-labs/dbt-docs/issues/369))
### Under the Hood
- [CT-921] dbt compile works in click ([#5545](https://github.com/dbt-labs/dbt-core/issues/5545))
- Fix use of ConnectionReused logging event ([#6168](https://github.com/dbt-labs/dbt-core/issues/6168))
- Port docs tests to pytest ([#6573](https://github.com/dbt-labs/dbt-core/issues/6573))
- Update deprecated github action command ([#6153](https://github.com/dbt-labs/dbt-core/issues/6153))
- dbt snapshot works in click ([#5554](https://github.com/dbt-labs/dbt-core/issues/5554))
- dbt list working with click ([#5549](https://github.com/dbt-labs/dbt-core/issues/5549))
- Add dbt run-operation to click CLI ([#5552](https://github.com/dbt-labs/dbt-core/issues/5552))
- dbt build working with new click framework ([#5541](https://github.com/dbt-labs/dbt-core/issues/5541))
- dbt docs generate works with new click framework ([#5543](https://github.com/dbt-labs/dbt-core/issues/5543))
- Replaced the EmptyLine event with a more general Formatting event, and added a Note event. ([#6481](https://github.com/dbt-labs/dbt-core/issues/6481))
- Small optimization on manifest parsing benefitting large DAGs ([#6697](https://github.com/dbt-labs/dbt-core/issues/6697))
- Revised and simplified various structured logging events ([#6664](https://github.com/dbt-labs/dbt-core/issues/6664), [#6665](https://github.com/dbt-labs/dbt-core/issues/6665), [#6666](https://github.com/dbt-labs/dbt-core/issues/6666))
- dbt init works with click ([#5548](https://github.com/dbt-labs/dbt-core/issues/5548))
- [CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs commands ([#5544](https://github.com/dbt-labs/dbt-core/issues/5544), [#6722](https://github.com/dbt-labs/dbt-core/issues/6722))
- Migrate debug task to click ([#5546](https://github.com/dbt-labs/dbt-core/issues/5546))
- Optimized GraphQueue to remove graph analysis bottleneck in large dags. ([#6759](https://github.com/dbt-labs/dbt-core/issues/6759))
- Implement --version for click cli ([#6757](https://github.com/dbt-labs/dbt-core/issues/6757))
- [CT-1841] Convert custom target test to Pytest ([#6638](https://github.com/dbt-labs/dbt-core/issues/6638))
- Remove BigQuery-specific btye abbreviations ([#6741](https://github.com/dbt-labs/dbt-core/issues/6741))
- warn_error/warn_error_options mutual exclusivity in click ([#6579](https://github.com/dbt-labs/dbt-core/issues/6579))
- Enables the new Click Cli on the commandline! 🚀 ([#6784](https://github.com/dbt-labs/dbt-core/issues/6784))
- Lazily call --version ([#6812](https://github.com/dbt-labs/dbt-core/issues/6812))
- Moving simple_seed to adapter zone to help adapter test conversions ([#CT-1959](https://github.com/dbt-labs/dbt-core/issues/CT-1959))
- flags.THREADS defaults to None ([#6887](https://github.com/dbt-labs/dbt-core/issues/6887))
- Fixing target type exposure error ([#6928](https://github.com/dbt-labs/dbt-core/issues/6928))
- Test binary serialization of logging events ([#6852](https://github.com/dbt-labs/dbt-core/issues/6852))
### Dependencies
- Bump ubuntu from 22.04 to 23.04 ([#6865](https://github.com/dbt-labs/dbt-core/pull/6865))
- Revert hoisting dbt.cli.main into the dbt.name namespace ([#](https://github.com/dbt-labs/dbt-core/pull/))
### Contributors
- [@aezomz](https://github.com/aezomz) ([#2964](https://github.com/dbt-labs/dbt-core/issues/2964))
- [@boxysean](https://github.com/boxysean) ([#6697](https://github.com/dbt-labs/dbt-core/issues/6697))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#6928](https://github.com/dbt-labs/dbt-core/issues/6928))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#6699](https://github.com/dbt-labs/dbt-core/issues/6699))
- [@davidbloss](https://github.com/davidbloss) ([#6153](https://github.com/dbt-labs/dbt-core/issues/6153))
- [@halvorlu](https://github.com/halvorlu) ([#366](https://github.com/dbt-labs/dbt-core/issues/366))
- [@nielspardon](https://github.com/nielspardon) ([#6900](https://github.com/dbt-labs/dbt-core/issues/6900))
- [@ryancharris](https://github.com/ryancharris) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@sungchun12](https://github.com/sungchun12) ([#6079](https://github.com/dbt-labs/dbt-core/issues/6079))

View File

@@ -0,0 +1,6 @@
kind: "Dependencies"
body: "Bump ubuntu from 22.04 to 23.04"
time: 2023-02-06T00:09:26.00000Z
custom:
Author: dependabot[bot]
PR: 6865

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Revert hoisting dbt.cli.main into the dbt.name namespace
time: 2023-02-15T09:17:59.04148-08:00
custom:
Author: aranke
PR: ''

View File

@@ -0,0 +1,6 @@
kind: Docs
body: update link to installation instructions
time: 2023-02-07T12:38:07.336783-05:00
custom:
Author: ryancharris
Issue: None

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Fix JSON path to overview docs
time: 2023-02-09T08:29:01.432616-07:00
custom:
Author: halvorlu
Issue: "366"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Searchable column descriptions
time: 2023-02-09T21:27:29.570243-07:00
custom:
Author: dbeatty10
Issue: 140 322 369

View File

@@ -0,0 +1,8 @@
kind: Features
body: Data type constraints are now native to SQL table materializations. Enforce
columns are specific data types and not null depending on database functionality.
time: 2022-11-18T14:11:20.868062-08:00
custom:
Author: sungchun12
Issue: "6079"
PR: "6271"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Have dbt debug spit out structured json logs with flags enabled.
time: 2023-01-07T00:31:57.516063-08:00
custom:
Author: versusfacit
Issue: "5353"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add adapter_response to dbt test and freshness result
time: 2023-01-18T23:38:01.857342+08:00
custom:
Author: aezomz
Issue: "2964"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Improve error message for packages missing `dbt_project.yml`
time: 2023-01-20T11:29:21.509967-07:00
custom:
Author: dbeatty10
Issue: "6663"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Adjust makefile to have clearer instructions for CI env var changes.
time: 2023-01-26T15:47:16.887327-08:00
custom:
Author: versusfacit
Issue: "6689"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Stand-alone Python module for PostgresColumn
time: 2023-01-27T16:28:12.212427-08:00
custom:
Author: nssalian
Issue: "6772"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Exposure owner requires one of name or email keys, and accepts additional arbitrary
keys
time: 2023-02-09T09:20:59.300272-05:00
custom:
Author: michelleark
Issue: "6833"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Parse 'group' resource
time: 2023-02-09T09:34:09.547006-05:00
custom:
Author: michelleark
Issue: "6921"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: add merge_exclude_columns adapter tests
time: 2023-01-23T13:28:14.808748-06:00
custom:
Author: dave-connors-3
Issue: "6699"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Include adapter_response in NodeFinished run_result log event
time: 2023-01-24T11:58:37.74179-05:00
custom:
Author: gshank
Issue: "6703"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Sort cli vars before hashing for partial parsing
time: 2023-01-24T14:19:43.333628-05:00
custom:
Author: gshank
Issue: "6710"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: '[Regression] exposure_content referenced incorrectly'
time: 2023-01-25T19:17:39.942081-05:00
custom:
Author: Mathyoub
Issue: "6738"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Remove pin on packaging and stop using it for prerelease comparisons
time: 2023-02-01T15:44:18.279158-05:00
custom:
Author: gshank
Issue: "6834"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Readd depends_on.macros to SeedNode, to support seeds with hooks calling macros
time: 2023-02-03T13:55:57.853715+01:00
custom:
Author: jtcohen6
Issue: "6806"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix regression of --quiet cli parameter behavior
time: 2023-02-07T14:35:44.160163-05:00
custom:
Author: peterallenwebb
Issue: "6749"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Ensure results from hooks contain nodes when processing them
time: 2023-02-08T11:05:51.952494-06:00
custom:
Author: emmyoop
Issue: "6796"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Always flush stdout after logging
time: 2023-02-08T15:49:35.175874-05:00
custom:
Author: peterallenwebb
Issue: "6901"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Reapply logging fixes which were accidentally reverted
time: 2023-02-10T10:30:28.179997-05:00
custom:
Author: peterallenwebb
Issue: "6936"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Set relation_name in test nodes at compile time
time: 2023-02-10T19:41:57.386766-05:00
custom:
Author: gshank
Issue: "6930"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Readd initialization events, --log-cache-events in new CLI
time: 2023-02-13T13:05:22.989477+01:00
custom:
Author: jtcohen6
Issue: "6933"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix previous state tests and disabled exposures, metrics
time: 2023-02-13T17:07:23.185679-05:00
custom:
Author: gshank
Issue: 6752 6753

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Make use of hashlib.md5() FIPS compliant
time: 2023-02-15T10:45:36.755797+01:00
custom:
Author: nielspardon
Issue: "6900"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-921] dbt compile works in click'
time: 2023-01-11T14:51:43.324107-08:00
custom:
Author: aranke
Issue: "5545"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix use of ConnectionReused logging event
time: 2023-01-13T13:25:13.023168-05:00
custom:
Author: gshank
Issue: "6168"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Port docs tests to pytest
time: 2023-01-13T15:07:00.477038-05:00
custom:
Author: peterallenwebb
Issue: "6573"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update deprecated github action command
time: 2023-01-17T11:17:37.046095-06:00
custom:
Author: davidbloss
Issue: "6153"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt snapshot works in click
time: 2023-01-17T16:25:05.973769-08:00
custom:
Author: ChenyuLInx
Issue: "5554"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt list working with click
time: 2023-01-17T21:37:29.91632-05:00
custom:
Author: michelleark
Issue: "5549"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add dbt run-operation to click CLI
time: 2023-01-19T10:53:04.154871+01:00
custom:
Author: jtcohen6
Issue: "5552"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt build working with new click framework
time: 2023-01-19T20:56:50.50549-05:00
custom:
Author: michelleark
Issue: "5541"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt docs generate works with new click framework
time: 2023-01-19T21:10:40.698851-05:00
custom:
Author: michelleark
Issue: "5543"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Replaced the EmptyLine event with a more general Formatting event, and added
a Note event.
time: 2023-01-20T17:22:54.45828-05:00
custom:
Author: peterallenwebb
Issue: "6481"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Small optimization on manifest parsing benefitting large DAGs
time: 2023-01-22T21:52:35.549814+01:00
custom:
Author: boxysean
Issue: "6697"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Revised and simplified various structured logging events
time: 2023-01-24T15:35:53.065356-05:00
custom:
Author: peterallenwebb
Issue: 6664 6665 6666

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: dbt init works with click
time: 2023-01-24T17:51:10.74065-05:00
custom:
Author: michelleark
Issue: "5548"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs commands'
time: 2023-01-25T04:11:36.57506-08:00
custom:
Author: aranke
Issue: 5544 6722

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Migrate debug task to click
time: 2023-01-25T10:26:06.735994-06:00
custom:
Author: stu-k
Issue: "5546"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: ' Optimized GraphQueue to remove graph analysis bottleneck in large dags.'
time: 2023-01-26T13:59:39.518345-05:00
custom:
Author: peterallenwebb
Issue: "6759"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Implement --version for click cli
time: 2023-01-26T14:31:02.740282-06:00
custom:
Author: stu-k
Issue: "6757"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: '[CT-1841] Convert custom target test to Pytest'
time: 2023-01-26T16:47:41.198714-08:00
custom:
Author: aranke
Issue: "6638"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove BigQuery-specific btye abbreviations
time: 2023-01-30T15:33:06.28965-07:00
custom:
Author: dbeatty10
Issue: "6741"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: "Enables the new Click Cli on the commandline! \U0001F680"
time: 2023-01-30T17:57:52.65626-06:00
custom:
Author: iknox-fa
Issue: "6784"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: warn_error/warn_error_options mutual exclusivity in click
time: 2023-01-30T18:09:17.240662-05:00
custom:
Author: michelleark
Issue: "6579"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Lazily call --version
time: 2023-01-31T14:18:06.02312-06:00
custom:
Author: stu-k
Issue: "6812"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Moving simple_seed to adapter zone to help adapter test conversions
time: 2023-02-03T14:35:51.481856-08:00
custom:
Author: nssalian
Issue: CT-1959

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: flags.THREADS defaults to None
time: 2023-02-07T16:51:11.011984-05:00
custom:
Author: michelleark
Issue: "6887"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fixing target type exposure error
time: 2023-02-10T08:46:47.72936-06:00
custom:
Author: callum-mcdata
Issue: "6928"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Test binary serialization of logging events
time: 2023-02-16T14:32:52.524225-05:00
custom:
Author: gshank
Issue: "6852"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Pin click<9 + sqlparse<0.5
time: 2023-07-19T12:37:43.716495+02:00
custom:
Author: jtcohen6
PR: "8146"

View File

@@ -0,0 +1,7 @@
kind: "Dependency"
body: "Bump mypy from 0.971 to 0.981"
time: 2022-09-27T00:08:22.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 5937

View File

@@ -0,0 +1,7 @@
kind: Docs
body: Improve displayed message under "Arguments" section for argumentless macro
time: 2023-01-13T09:48:55.574898-05:00
custom:
Author: MartinGuindon
Issue: "358"
PR: "359"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Fix for column tests not rendering on quoted columns
time: 2023-05-31T11:54:19.687363-04:00
custom:
Author: drewbanin
Issue: "201"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Remove static SQL codeblock for metrics
time: 2023-07-18T19:24:22.155323+02:00
custom:
Author: marcodamore
Issue: "436"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Adding the entity node
time: 2023-01-18T13:48:04.487817-06:00
custom:
Author: callum-mcdata
Issue: "6627"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Make project version optional
time: 2023-01-24T13:55:50.86071024-08:00
custom:
Author: seub
Issue: "6603"

View File

@@ -0,0 +1,6 @@
kind: Features
body: parse 'group' config on groupable nodes
time: 2023-02-14T22:51:34.936228-05:00
custom:
Author: michelleark
Issue: "6823"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Implemented new log cli parameters for finer-grained control.
time: 2023-02-16T14:45:34.038453-05:00
custom:
Author: peterallenwebb
Issue: "6639"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add access attribute to parsed nodes
time: 2023-02-18T09:28:16.448175-05:00
custom:
Author: gshank
Issue: "6824"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Remove trailing slashes from source paths (#6102)
time: 2022-10-30T10:21:14.660221Z
custom:
Author: jmg-duarte
Issue: "6102"
PR: "6179"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix compilation logic for ephemeral nodes
time: 2023-02-21T17:06:30.218568-05:00
custom:
Author: gshank
Issue: "6885"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix semver comparison logic by ensuring numeric values
time: 2023-02-24T00:13:38.23242+01:00
custom:
Author: jtcohen6
Issue: "7039"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Enable converting deprecation warnings to errors
time: 2023-07-18T12:55:18.03914-04:00
custom:
Author: michelleark
Issue: "8130"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Rename "constraint_enabled" to "contract"
time: 2023-02-17T10:52:23.212474-05:00
custom:
Author: gshank
Issue: "6748"

4
.gitattributes vendored
View File

@@ -1,6 +1,2 @@
core/dbt/include/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary
core/dbt/docs/build/html/searchindex.js binary
core/dbt/docs/build/html/index.html binary
performance/runner/Cargo.lock binary
core/dbt/events/types_pb2.py binary

53
.github/CODEOWNERS vendored
View File

@@ -11,24 +11,44 @@
# As a default for areas with no assignment,
# the core team as a whole will be assigned
* @dbt-labs/core-team
* @dbt-labs/core
### OSS Tooling Guild
# Changes to GitHub configurations including Actions
/.github/ @leahwicz
/.github/ @dbt-labs/guild-oss-tooling
.bumpversion.cfg @dbt-labs/guild-oss-tooling
### LANGUAGE
.changie.yaml @dbt-labs/guild-oss-tooling
# Language core modules
/core/dbt/config/ @dbt-labs/core-language
/core/dbt/context/ @dbt-labs/core-language
/core/dbt/contracts/ @dbt-labs/core-language
/core/dbt/deps/ @dbt-labs/core-language
/core/dbt/events/ @dbt-labs/core-language # structured logging
/core/dbt/parser/ @dbt-labs/core-language
pre-commit-config.yaml @dbt-labs/guild-oss-tooling
pytest.ini @dbt-labs/guild-oss-tooling
tox.ini @dbt-labs/guild-oss-tooling
# Language misc files
/core/dbt/dataclass_schema.py @dbt-labs/core-language
/core/dbt/hooks.py @dbt-labs/core-language
/core/dbt/node_types.py @dbt-labs/core-language
/core/dbt/semver.py @dbt-labs/core-language
### EXECUTION
# Execution core modules
/core/dbt/graph/ @dbt-labs/core-execution
/core/dbt/task/ @dbt-labs/core-execution
# Execution misc files
/core/dbt/compilation.py @dbt-labs/core-execution
/core/dbt/flags.py @dbt-labs/core-execution
/core/dbt/lib.py @dbt-labs/core-execution
/core/dbt/main.py @dbt-labs/core-execution
/core/dbt/profiler.py @dbt-labs/core-execution
/core/dbt/selected_resources.py @dbt-labs/core-execution
/core/dbt/tracking.py @dbt-labs/core-execution
/core/dbt/version.py @dbt-labs/core-execution
pyproject.toml @dbt-labs/guild-oss-tooling
requirements.txt @dbt-labs/guild-oss-tooling
dev_requirements.txt @dbt-labs/guild-oss-tooling
/core/setup.py @dbt-labs/guild-oss-tooling
/core/MANIFEST.in @dbt-labs/guild-oss-tooling
### ADAPTERS
@@ -40,7 +60,6 @@ dev_requirements.txt @dbt-labs/guild-oss-tooling
# Postgres plugin
/plugins/ @dbt-labs/core-adapters
/plugins/postgres/setup.py @dbt-labs/core-adapters @dbt-labs/guild-oss-tooling
# Functional tests for adapter plugins
/tests/adapter @dbt-labs/core-adapters
@@ -52,9 +71,5 @@ dev_requirements.txt @dbt-labs/guild-oss-tooling
# Perf regression testing framework
# This excludes the test project files itself since those aren't specific
# framework changes (excluded by not setting an owner next to it- no owner)
/performance @nathaniel-may
/performance @nathaniel-may
/performance/projects
### ARTIFACTS
/schemas/dbt @dbt-labs/cloud-artifacts

2
.github/_README.md vendored
View File

@@ -197,7 +197,7 @@ ___
```yaml
- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v2
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

View File

@@ -35,7 +35,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v1
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler

View File

@@ -13,7 +13,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v1
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler

View File

@@ -1,35 +1,23 @@
resolves #
[docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose) dbt-labs/docs.getdbt.com/#
resolves #
<!---
Include the number of the issue addressed by this PR above if applicable.
PRs for code changes without an associated issue *will not be merged*.
See CONTRIBUTING.md for more information.
Include the number of the docs issue that was opened for this PR. If
this change has no user-facing implications, "N/A" suffices instead. New
docs tickets can be created by clicking the link above or by going to
https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose.
-->
### Problem
### Description
<!---
Describe the problem this PR is solving. What is the application state
before this PR is merged?
-->
### Solution
<!---
Describe the way this PR solves the above problem. Add as much detail as you
can to help reviewers understand your changes. Include any alternatives and
tradeoffs you considered.
Describe the Pull Request here. Add any references and info to help reviewers
understand your changes. Include any tradeoffs you considered.
-->
### Checklist
- [ ] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me
- [ ] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [ ] I have [opened an issue to add/update docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose), or docs changes are not required/relevant for this PR
- [ ] I have run `changie new` to [create a changelog entry](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-a-changelog-entry)

View File

@@ -35,6 +35,6 @@ jobs:
github.event.pull_request.merged
&& contains(github.event.label.name, 'backport')
steps:
- uses: tibdex/backport@v2.0.3
- uses: tibdex/backport@v2.0.2
with:
github_token: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -50,7 +50,7 @@ jobs:
- name: Create and commit changelog on bot PR
if: ${{ contains(github.event.pull_request.labels.*.name, matrix.label) }}
id: bot_changelog
uses: emmyoop/changie_bot@v1.1.0
uses: emmyoop/changie_bot@v1.0.1
with:
GITHUB_TOKEN: ${{ secrets.FISHTOWN_BOT_PAT }}
commit_author_name: "Github Build Bot"

View File

@@ -0,0 +1,165 @@
# **what?**
# On push, if anything in core/dbt/docs or core/dbt/cli has been
# created or modified, regenerate the CLI API docs using sphinx.
# **why?**
# We watch for changes in core/dbt/cli because the CLI API docs rely on click
# and all supporting flags/params to be generated. We watch for changes in
# core/dbt/docs since any changes to sphinx configuration or any of the
# .rst files there could result in a differently build final index.html file.
# **when?**
# Whenever a change has been pushed to a branch, and only if there is a diff
# between the PR branch and main's core/dbt/cli and or core/dbt/docs dirs.
# TODO: add bot comment to PR informing contributor that the docs have been committed
# TODO: figure out why github action triggered pushes cause github to fail to report
# the status of jobs
name: Generate CLI API docs
on:
pull_request:
permissions:
contents: write
pull-requests: write
env:
CLI_DIR: ${{ github.workspace }}/core/dbt/cli
DOCS_DIR: ${{ github.workspace }}/core/dbt/docs
DOCS_BUILD_DIR: ${{ github.workspace }}/core/dbt/docs/build
jobs:
check_gen:
name: check if generation needed
runs-on: ubuntu-latest
if: ${{ github.event.pull_request.head.repo.fork == false }}
outputs:
cli_dir_changed: ${{ steps.check_cli.outputs.cli_dir_changed }}
docs_dir_changed: ${{ steps.check_docs.outputs.docs_dir_changed }}
steps:
- name: "[DEBUG] print variables"
run: |
echo "env.CLI_DIR: ${{ env.CLI_DIR }}"
echo "env.DOCS_BUILD_DIR: ${{ env.DOCS_BUILD_DIR }}"
echo "env.DOCS_DIR: ${{ env.DOCS_DIR }}"
- name: git checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
ref: ${{ github.head_ref }}
- name: set shas
id: set_shas
run: |
THIS_SHA=$(git rev-parse @)
LAST_SHA=$(git rev-parse @~1)
echo "this sha: $THIS_SHA"
echo "last sha: $LAST_SHA"
echo "this_sha=$THIS_SHA" >> $GITHUB_OUTPUT
echo "last_sha=$LAST_SHA" >> $GITHUB_OUTPUT
- name: check for changes in core/dbt/cli
id: check_cli
run: |
CLI_DIR_CHANGES=$(git diff \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.CLI_DIR }})
if [ -n "$CLI_DIR_CHANGES" ]; then
echo "changes found"
echo $CLI_DIR_CHANGES
echo "cli_dir_changed=true" >> $GITHUB_OUTPUT
exit 0
fi
echo "cli_dir_changed=false" >> $GITHUB_OUTPUT
echo "no changes found"
- name: check for changes in core/dbt/docs
id: check_docs
if: steps.check_cli.outputs.cli_dir_changed == 'false'
run: |
DOCS_DIR_CHANGES=$(git diff --name-only \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.DOCS_DIR }} ':!${{ env.DOCS_BUILD_DIR }}')
DOCS_BUILD_DIR_CHANGES=$(git diff --name-only \
${{ steps.set_shas.outputs.last_sha }} \
${{ steps.set_shas.outputs.this_sha }} \
-- ${{ env.DOCS_BUILD_DIR }})
if [ -n "$DOCS_DIR_CHANGES" ] && [ -z "$DOCS_BUILD_DIR_CHANGES" ]; then
echo "changes found"
echo $DOCS_DIR_CHANGES
echo "docs_dir_changed=true" >> $GITHUB_OUTPUT
exit 0
fi
echo "docs_dir_changed=false" >> $GITHUB_OUTPUT
echo "no changes found"
gen_docs:
name: generate docs
runs-on: ubuntu-latest
needs: [check_gen]
if: |
needs.check_gen.outputs.cli_dir_changed == 'true'
|| needs.check_gen.outputs.docs_dir_changed == 'true'
steps:
- name: "[DEBUG] print variables"
run: |
echo "env.DOCS_DIR: ${{ env.DOCS_DIR }}"
echo "github head_ref: ${{ github.head_ref }}"
- name: git checkout
uses: actions/checkout@v3
with:
ref: ${{ github.head_ref }}
- name: install python
uses: actions/setup-python@v4.3.0
with:
python-version: 3.8
- name: install dev requirements
run: |
python3 -m venv env
source env/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt -r dev-requirements.txt
- name: generate docs
run: |
source env/bin/activate
cd ${{ env.DOCS_DIR }}
echo "cleaning existing docs"
make clean
echo "creating docs"
make html
- name: debug
run: |
echo ">>>>> status"
git status
echo ">>>>> remotes"
git remote -v
echo ">>>>> branch"
git branch -v
echo ">>>>> log"
git log --pretty=oneline | head -5
- name: commit docs
run: |
git config user.name 'Github Build Bot'
git config user.email 'buildbot@fishtownanalytics.com'
git commit -am "Add generated CLI API docs"
git push -u origin ${{ github.head_ref }}

View File

@@ -18,8 +18,8 @@ permissions:
issues: write
jobs:
call-creation-action:
uses: dbt-labs/actions/.github/workflows/jira-creation-actions.yml@main
call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-creation.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -19,7 +19,7 @@ permissions:
jobs:
call-label-action:
uses: dbt-labs/actions/.github/workflows/jira-label-actions.yml@main
uses: dbt-labs/jira-actions/.github/workflows/jira-label.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -19,8 +19,8 @@ on:
permissions: read-all
jobs:
call-transition-action:
uses: dbt-labs/actions/.github/workflows/jira-transition-actions.yml@main
call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-transition.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -42,10 +42,10 @@ jobs:
steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v4.3.0
with:
python-version: '3.8'
@@ -53,8 +53,12 @@ jobs:
run: |
python -m pip install --user --upgrade pip
python -m pip --version
make dev
python -m pip install pre-commit
pre-commit --version
python -m pip install mypy==0.942
mypy --version
python -m pip install -r requirements.txt
python -m pip install -r dev-requirements.txt
dbt --version
- name: Run pre-commit hooks
@@ -69,17 +73,18 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
env:
TOXENV: "unit"
PYTEST_ADDOPTS: "-v --color=yes --csv unit_results.csv"
steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v4.3.0
with:
python-version: ${{ matrix.python-version }}
@@ -100,22 +105,22 @@ jobs:
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- name: Upload Unit Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
- uses: actions/upload-artifact@v2
if: always()
with:
name: unit_results_${{ matrix.python-version }}-${{ steps.date.outputs.date }}.csv
path: unit_results.csv
integration:
name: integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 60
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-20.04]
include:
- python-version: 3.8
@@ -125,22 +130,18 @@ jobs:
env:
TOXENV: integration
PYTEST_ADDOPTS: "-v --color=yes -n4 --csv integration_results.csv"
DBT_INVOCATION_ENV: github-actions
DBT_TEST_USER_1: dbt_test_user_1
DBT_TEST_USER_2: dbt_test_user_2
DBT_TEST_USER_3: dbt_test_user_3
DD_CIVISIBILITY_AGENTLESS_ENABLED: true
DD_API_KEY: ${{ secrets.DATADOG_API_KEY }}
DD_SITE: datadoghq.com
DD_ENV: ci
DD_SERVICE: ${{ github.event.repository.name }}
steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v4.3.0
with:
python-version: ${{ matrix.python-version }}
@@ -164,7 +165,7 @@ jobs:
tox --version
- name: Run tests
run: tox -- --ddtrace
run: tox
- name: Get current date
if: always()
@@ -173,17 +174,17 @@ jobs:
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v2
if: always()
with:
name: logs_${{ matrix.python-version }}_${{ matrix.os }}_${{ steps.date.outputs.date }}
path: ./logs
- name: Upload Integration Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
- uses: actions/upload-artifact@v2
if: always()
with:
name: integration_results_${{ matrix.python-version }}_${{ matrix.os }}_${{ steps.date.outputs.date }}.csv
path: integration_results.csv
build:
name: build packages
@@ -192,10 +193,10 @@ jobs:
steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v4.3.0
with:
python-version: '3.8'

View File

@@ -1,265 +0,0 @@
# **what?**
# This workflow models the performance characteristics of a point in time in dbt.
# It runs specific dbt commands on committed projects multiple times to create and
# commit information about the distribution to the current branch. For more information
# see the readme in the performance module at /performance/README.md.
#
# **why?**
# When developing new features, we can take quick performance samples and compare
# them against the commited baseline measurements produced by this workflow to detect
# some performance regressions at development time before they reach users.
#
# **when?**
# This is only run once directly after each release (for non-prereleases). If for some
# reason the results of a run are not satisfactory, it can also be triggered manually.
name: Model Performance Characteristics
on:
# runs after non-prereleases are published.
release:
types: [released]
# run manually from the actions tab
workflow_dispatch:
inputs:
release_id:
description: 'dbt version to model (must be non-prerelease in Pypi)'
type: string
required: true
env:
RUNNER_CACHE_PATH: performance/runner/target/release/runner
# both jobs need to write
permissions:
contents: write
pull-requests: write
jobs:
set-variables:
name: Setting Variables
runs-on: ubuntu-latest
outputs:
cache_key: ${{ steps.variables.outputs.cache_key }}
release_id: ${{ steps.semver.outputs.base-version }}
release_branch: ${{ steps.variables.outputs.release_branch }}
steps:
# explicitly checkout the performance runner from main regardless of which
# version we are modeling.
- name: Checkout
uses: actions/checkout@v3
with:
ref: main
- name: Parse version into parts
id: semver
uses: dbt-labs/actions/parse-semver@v1
with:
version: ${{ github.event.inputs.release_id || github.event.release.tag_name }}
# collect all the variables that need to be used in subsequent jobs
- name: Set variables
id: variables
run: |
# create a cache key that will be used in the next job. without this the
# next job would have to checkout from main and hash the files itself.
echo "cache_key=${{ runner.os }}-${{ hashFiles('performance/runner/Cargo.toml')}}-${{ hashFiles('performance/runner/src/*') }}" >> $GITHUB_OUTPUT
branch_name="${{steps.semver.outputs.major}}.${{steps.semver.outputs.minor}}.latest"
echo "release_branch=$branch_name" >> $GITHUB_OUTPUT
echo "release branch is inferred to be ${branch_name}"
latest-runner:
name: Build or Fetch Runner
runs-on: ubuntu-latest
needs: [set-variables]
env:
RUSTFLAGS: "-D warnings"
steps:
- name: '[DEBUG] print variables'
run: |
echo "all variables defined in set-variables"
echo "cache_key: ${{ needs.set-variables.outputs.cache_key }}"
echo "release_id: ${{ needs.set-variables.outputs.release_id }}"
echo "release_branch: ${{ needs.set-variables.outputs.release_branch }}"
# explicitly checkout the performance runner from main regardless of which
# version we are modeling.
- name: Checkout
uses: actions/checkout@v3
with:
ref: main
# attempts to access a previously cached runner
- uses: actions/cache@v3
id: cache
with:
path: ${{ env.RUNNER_CACHE_PATH }}
key: ${{ needs.set-variables.outputs.cache_key }}
- name: Fetch Rust Toolchain
if: steps.cache.outputs.cache-hit != 'true'
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: stable
override: true
- name: Add fmt
if: steps.cache.outputs.cache-hit != 'true'
run: rustup component add rustfmt
- name: Cargo fmt
if: steps.cache.outputs.cache-hit != 'true'
uses: actions-rs/cargo@v1
with:
command: fmt
args: --manifest-path performance/runner/Cargo.toml --all -- --check
- name: Test
if: steps.cache.outputs.cache-hit != 'true'
uses: actions-rs/cargo@v1
with:
command: test
args: --manifest-path performance/runner/Cargo.toml
- name: Build (optimized)
if: steps.cache.outputs.cache-hit != 'true'
uses: actions-rs/cargo@v1
with:
command: build
args: --release --manifest-path performance/runner/Cargo.toml
# the cache action automatically caches this binary at the end of the job
model:
# depends on `latest-runner` as a separate job so that failures in this job do not prevent
# a successfully tested and built binary from being cached.
needs: [set-variables, latest-runner]
name: Model a release
runs-on: ubuntu-latest
steps:
- name: '[DEBUG] print variables'
run: |
echo "all variables defined in set-variables"
echo "cache_key: ${{ needs.set-variables.outputs.cache_key }}"
echo "release_id: ${{ needs.set-variables.outputs.release_id }}"
echo "release_branch: ${{ needs.set-variables.outputs.release_branch }}"
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.8"
- name: Install dbt
run: pip install dbt-postgres==${{ needs.set-variables.outputs.release_id }}
- name: Install Hyperfine
run: wget https://github.com/sharkdp/hyperfine/releases/download/v1.11.0/hyperfine_1.11.0_amd64.deb && sudo dpkg -i hyperfine_1.11.0_amd64.deb
# explicitly checkout main to get the latest project definitions
- name: Checkout
uses: actions/checkout@v3
with:
ref: main
# this was built in the previous job so it will be there.
- name: Fetch Runner
uses: actions/cache@v3
id: cache
with:
path: ${{ env.RUNNER_CACHE_PATH }}
key: ${{ needs.set-variables.outputs.cache_key }}
- name: Move Runner
run: mv performance/runner/target/release/runner performance/app
- name: Change Runner Permissions
run: chmod +x ./performance/app
- name: '[DEBUG] ls baseline directory before run'
run: ls -R performance/baselines/
# `${{ github.workspace }}` is used to pass the absolute path
- name: Create directories
run: |
mkdir ${{ github.workspace }}/performance/tmp/
mkdir -p performance/baselines/${{ needs.set-variables.outputs.release_id }}/
# Run modeling with taking 20 samples
- name: Run Measurement
run: |
performance/app model -v ${{ needs.set-variables.outputs.release_id }} -b ${{ github.workspace }}/performance/baselines/ -p ${{ github.workspace }}/performance/projects/ -t ${{ github.workspace }}/performance/tmp/ -n 20
- name: '[DEBUG] ls baseline directory after run'
run: ls -R performance/baselines/
- uses: actions/upload-artifact@v3
with:
name: baseline
path: performance/baselines/${{ needs.set-variables.outputs.release_id }}/
create-pr:
name: Open PR for ${{ matrix.base-branch }}
# depends on `model` as a separate job so that the baseline can be committed to more than one branch
# i.e. release branch and main
needs: [set-variables, latest-runner, model]
runs-on: ubuntu-latest
strategy:
matrix:
include:
- base-branch: refs/heads/main
target-branch: performance-bot/main_${{ needs.set-variables.outputs.release_id }}_${{GITHUB.RUN_ID}}
- base-branch: refs/heads/${{ needs.set-variables.outputs.release_branch }}
target-branch: performance-bot/release_${{ needs.set-variables.outputs.release_id }}_${{GITHUB.RUN_ID}}
steps:
- name: '[DEBUG] print variables'
run: |
echo "all variables defined in set-variables"
echo "cache_key: ${{ needs.set-variables.outputs.cache_key }}"
echo "release_id: ${{ needs.set-variables.outputs.release_id }}"
echo "release_branch: ${{ needs.set-variables.outputs.release_branch }}"
- name: Checkout
uses: actions/checkout@v3
with:
ref: ${{ matrix.base-branch }}
- name: Create PR branch
run: |
git checkout -b ${{ matrix.target-branch }}
git push origin ${{ matrix.target-branch }}
git branch --set-upstream-to=origin/${{ matrix.target-branch }} ${{ matrix.target-branch }}
- uses: actions/download-artifact@v3
with:
name: baseline
path: performance/baselines/${{ needs.set-variables.outputs.release_id }}
- name: '[DEBUG] ls baselines after artifact download'
run: ls -R performance/baselines/
- name: Commit baseline
uses: EndBug/add-and-commit@v9
with:
add: 'performance/baselines/*'
author_name: 'Github Build Bot'
author_email: 'buildbot@fishtownanalytics.com'
message: 'adding performance baseline for ${{ needs.set-variables.outputs.release_id }}'
push: 'origin origin/${{ matrix.target-branch }}'
- name: Create Pull Request
uses: peter-evans/create-pull-request@v5
with:
author: 'Github Build Bot <buildbot@fishtownanalytics.com>'
base: ${{ matrix.base-branch }}
branch: '${{ matrix.target-branch }}'
title: 'Adding performance modeling for ${{needs.set-variables.outputs.release_id}} to ${{ matrix.base-branch }}'
body: 'Committing perf results for tracking for the ${{needs.set-variables.outputs.release_id}}'
labels: |
Skip Changelog
Performance

View File

@@ -1,7 +1,11 @@
# **what?**
# The purpose of this workflow is to trigger CI to run for each
# release branch and main branch on a regular cadence. If the CI workflow
# fails for a branch, it will post to #dev-core-alerts to raise awareness.
# fails for a branch, it will post to dev-core-alerts to raise awareness.
# The 'aurelien-baudet/workflow-dispatch' Action triggers the existing
# CI worklow file on the given branch to run so that even if we change the
# CI workflow file in the future, the one that is tailored for the given
# release branch will be used.
# **why?**
# Ensures release branches and main are always shippable and not broken.
@@ -24,8 +28,63 @@ on:
permissions: read-all
jobs:
run_tests:
uses: dbt-labs/actions/.github/workflows/release-branch-tests.yml@main
with:
workflows_to_run: '["main.yml"]'
secrets: inherit
fetch-latest-branches:
runs-on: ubuntu-latest
outputs:
latest-branches: ${{ steps.get-latest-branches.outputs.repo-branches }}
steps:
- name: "Fetch dbt-core Latest Branches"
uses: dbt-labs/actions/fetch-repo-branches@v1.1.1
id: get-latest-branches
with:
repo_name: ${{ github.event.repository.name }}
organization: "dbt-labs"
pat: ${{ secrets.GITHUB_TOKEN }}
fetch_protected_branches_only: true
regex: "^1.[0-9]+.latest$"
perform_match_method: "match"
retries: 3
- name: "[ANNOTATION] ${{ github.event.repository.name }} - branches to test"
run: |
title="${{ github.event.repository.name }} - branches to test"
message="The workflow will run tests for the following branches of the ${{ github.event.repository.name }} repo: ${{ steps.get-latest-branches.outputs.repo-branches }}"
echo "::notice $title::$message"
kick-off-ci:
needs: [fetch-latest-branches]
name: Kick-off CI
runs-on: ubuntu-latest
strategy:
# must run CI 1 branch at a time b/c the workflow-dispatch Action polls for
# latest run for results and it gets confused when we kick off multiple runs
# at once. There is a race condition so we will just run in sequential order.
max-parallel: 1
fail-fast: false
matrix:
branch: ${{ fromJSON(needs.fetch-latest-branches.outputs.latest-branches) }}
include:
- branch: 'main'
steps:
- name: Call CI workflow for ${{ matrix.branch }} branch
id: trigger-step
uses: aurelien-baudet/workflow-dispatch@v2.1.1
with:
workflow: main.yml
ref: ${{ matrix.branch }}
token: ${{ secrets.FISHTOWN_BOT_PAT }}
- name: Post failure to Slack
uses: ravsamhq/notify-slack-action@v1
if: ${{ always() && !contains(steps.trigger-step.outputs.workflow-conclusion,'success') }}
with:
status: ${{ job.status }}
notification_title: 'dbt-core scheduled run of "${{ matrix.branch }}" branch not successful'
message_format: ':x: CI on branch "${{ matrix.branch }}" ${{ steps.trigger-step.outputs.workflow-conclusion }}'
footer: 'Linked failed CI run ${{ steps.trigger-step.outputs.workflow-url }}'
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}

View File

@@ -36,7 +36,7 @@ jobs:
latest: ${{ steps.latest.outputs.latest }}
minor_latest: ${{ steps.latest.outputs.minor_latest }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v1
- name: Split version
id: version
run: |
@@ -60,7 +60,7 @@ jobs:
needs: [get_version_meta]
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v1
build_and_push:
name: Build images and push to GHCR
@@ -76,14 +76,14 @@ jobs:
echo "build_arg_value=$BUILD_ARG_VALUE" >> $GITHUB_OUTPUT
- name: Log in to the GHCR
uses: docker/login-action@v2
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push MAJOR.MINOR.PATCH tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v2
with:
file: docker/Dockerfile
push: True
@@ -94,7 +94,7 @@ jobs:
ghcr.io/dbt-labs/${{ github.event.inputs.package }}:${{ github.event.inputs.version_number }}
- name: Build and push MINOR.latest tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v2
if: ${{ needs.get_version_meta.outputs.minor_latest == 'True' }}
with:
file: docker/Dockerfile
@@ -106,7 +106,7 @@ jobs:
ghcr.io/dbt-labs/${{ github.event.inputs.package }}:${{ needs.get_version_meta.outputs.major }}.${{ needs.get_version_meta.outputs.minor }}.latest
- name: Build and push latest tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v2
if: ${{ needs.get_version_meta.outputs.latest == 'True' }}
with:
file: docker/Dockerfile

View File

@@ -37,17 +37,17 @@ jobs:
steps:
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Checkout dbt repo
uses: actions/checkout@v3
uses: actions/checkout@v2.3.4
with:
path: ${{ env.DBT_REPO_DIRECTORY }}
- name: Checkout schemas.getdbt.com repo
uses: actions/checkout@v3
uses: actions/checkout@v2.3.4
with:
repository: dbt-labs/schemas.getdbt.com
ref: 'main'
@@ -83,7 +83,7 @@ jobs:
fi
- name: Upload schema diff
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v2.2.4
if: ${{ failure() }}
with:
name: 'schema_schanges.txt'

View File

@@ -39,12 +39,12 @@ jobs:
steps:
- name: checkout dev
uses: actions/checkout@v3
uses: actions/checkout@v2
with:
persist-credentials: false
- name: Setup Python
uses: actions/setup-python@v4
uses: actions/setup-python@v2.2.2
with:
python-version: "3.8"

View File

@@ -1,155 +0,0 @@
# **what?**
# This workflow will test all test(s) at the input path given number of times to determine if it's flaky or not. You can test with any supported OS/Python combination.
# This is batched in 10 to allow more test iterations faster.
# **why?**
# Testing if a test is flaky and if a previously flaky test has been fixed. This allows easy testing on supported python versions and OS combinations.
# **when?**
# This is triggered manually from dbt-core.
name: Flaky Tester
on:
workflow_dispatch:
inputs:
branch:
description: 'Branch to check out'
type: string
required: true
default: 'main'
test_path:
description: 'Path to single test to run (ex: tests/functional/retry/test_retry.py::TestRetry::test_fail_fast)'
type: string
required: true
default: 'tests/functional/...'
python_version:
description: 'Version of Python to Test Against'
type: choice
options:
- '3.8'
- '3.9'
- '3.10'
- '3.11'
os:
description: 'OS to run test in'
type: choice
options:
- 'ubuntu-latest'
- 'macos-latest'
- 'windows-latest'
num_runs_per_batch:
description: 'Max number of times to run the test per batch. We always run 10 batches.'
type: number
required: true
default: '50'
permissions: read-all
defaults:
run:
shell: bash
jobs:
debug:
runs-on: ubuntu-latest
steps:
- name: "[DEBUG] Output Inputs"
run: |
echo "Branch: ${{ inputs.branch }}"
echo "test_path: ${{ inputs.test_path }}"
echo "python_version: ${{ inputs.python_version }}"
echo "os: ${{ inputs.os }}"
echo "num_runs_per_batch: ${{ inputs.num_runs_per_batch }}"
pytest:
runs-on: ${{ inputs.os }}
strategy:
# run all batches, even if one fails. This informs how flaky the test may be.
fail-fast: false
# using a matrix to speed up the jobs since the matrix will run in parallel when runners are available
matrix:
batch: ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"]
env:
PYTEST_ADDOPTS: "-v --color=yes -n4 --csv integration_results.csv"
DBT_TEST_USER_1: dbt_test_user_1
DBT_TEST_USER_2: dbt_test_user_2
DBT_TEST_USER_3: dbt_test_user_3
DD_CIVISIBILITY_AGENTLESS_ENABLED: true
DD_API_KEY: ${{ secrets.DATADOG_API_KEY }}
DD_SITE: datadoghq.com
DD_ENV: ci
DD_SERVICE: ${{ github.event.repository.name }}
steps:
- name: "Checkout code"
uses: actions/checkout@v3
with:
ref: ${{ inputs.branch }}
- name: "Setup Python"
uses: actions/setup-python@v4
with:
python-version: "${{ inputs.python_version }}"
- name: "Setup Dev Environment"
run: make dev
- name: "Set up postgres (linux)"
if: inputs.os == 'ubuntu-latest'
run: make setup-db
# mac and windows don't use make due to limitations with docker with those runners in GitHub
- name: "Set up postgres (macos)"
if: inputs.os == 'macos-latest'
uses: ./.github/actions/setup-postgres-macos
- name: "Set up postgres (windows)"
if: inputs.os == 'windows-latest'
uses: ./.github/actions/setup-postgres-windows
- name: "Test Command"
id: command
run: |
test_command="python -m pytest ${{ inputs.test_path }}"
echo "test_command=$test_command" >> $GITHUB_OUTPUT
- name: "Run test ${{ inputs.num_runs_per_batch }} times"
id: pytest
run: |
set +e
for ((i=1; i<=${{ inputs.num_runs_per_batch }}; i++))
do
echo "Running pytest iteration $i..."
python -m pytest --ddtrace ${{ inputs.test_path }}
exit_code=$?
if [[ $exit_code -eq 0 ]]; then
success=$((success + 1))
echo "Iteration $i: Success"
else
failure=$((failure + 1))
echo "Iteration $i: Failure"
fi
echo
echo "==========================="
echo "Successful runs: $success"
echo "Failed runs: $failure"
echo "==========================="
echo
done
echo "failure=$failure" >> $GITHUB_OUTPUT
- name: "Success and Failure Summary: ${{ inputs.os }}/Python ${{ inputs.python_version }}"
run: |
echo "Batch: ${{ matrix.batch }}"
echo "Successful runs: ${{ steps.pytest.outputs.success }}"
echo "Failed runs: ${{ steps.pytest.outputs.failure }}"
- name: "Error for Failures"
if: ${{ steps.pytest.outputs.failure }}
run: |
echo "Batch ${{ matrix.batch }} failed ${{ steps.pytest.outputs.failure }} of ${{ inputs.num_runs_per_batch }} tests"
exit 1

View File

@@ -24,8 +24,10 @@ permissions:
jobs:
triage_label:
if: contains(github.event.issue.labels.*.name, 'awaiting_response')
uses: dbt-labs/actions/.github/workflows/swap-labels.yml@main
with:
add_label: "triage"
remove_label: "awaiting_response"
secrets: inherit
runs-on: ubuntu-latest
steps:
- name: initial labeling
uses: andymckay/labeler@master
with:
add-labels: "triage"
remove-labels: "awaiting_response"

4
.gitignore vendored
View File

@@ -9,9 +9,9 @@ __pycache__/
# Distribution / packaging
.Python
env*/
.mf_dbt_venv
dbt_env/
build/
!tests/functional/build
!core/dbt/docs/build
develop-eggs/
dist/
@@ -29,8 +29,6 @@ var/
.mypy_cache/
.dmypy.json
logs/
.user.yml
profiles.yml
# PyInstaller
# Usually these files are written by a python script from a template

View File

@@ -1,7 +1,8 @@
# Configuration for pre-commit hooks (see https://pre-commit.com/).
# Eventually the hooks described here will be run as tests before merging each PR.
exclude: ^(core/dbt/docs/build/|core/dbt/events/types_pb2.py)
# TODO: remove global exclusion of tests when testing overhaul is complete
exclude: ^(test/|core/dbt/docs/build/)
# Force all unspecified python hooks to run python 3.8
default_language_version:
@@ -37,7 +38,7 @@ repos:
alias: flake8-check
stages: [manual]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.3.0
rev: v0.942
hooks:
- id: mypy
# N.B.: Mypy is... a bit fragile.

View File

@@ -5,13 +5,93 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.5.0-b1 - February 17, 2023
### Features
- Data type constraints are now native to SQL table materializations. Enforce columns are specific data types and not null depending on database functionality. ([#6079](https://github.com/dbt-labs/dbt-core/issues/6079))
- Have dbt debug spit out structured json logs with flags enabled. ([#5353](https://github.com/dbt-labs/dbt-core/issues/5353))
- add adapter_response to dbt test and freshness result ([#2964](https://github.com/dbt-labs/dbt-core/issues/2964))
- Improve error message for packages missing `dbt_project.yml` ([#6663](https://github.com/dbt-labs/dbt-core/issues/6663))
- Adjust makefile to have clearer instructions for CI env var changes. ([#6689](https://github.com/dbt-labs/dbt-core/issues/6689))
- Stand-alone Python module for PostgresColumn ([#6772](https://github.com/dbt-labs/dbt-core/issues/6772))
- Exposure owner requires one of name or email keys, and accepts additional arbitrary keys ([#6833](https://github.com/dbt-labs/dbt-core/issues/6833))
- Parse 'group' resource ([#6921](https://github.com/dbt-labs/dbt-core/issues/6921))
### Fixes
- add merge_exclude_columns adapter tests ([#6699](https://github.com/dbt-labs/dbt-core/issues/6699))
- Include adapter_response in NodeFinished run_result log event ([#6703](https://github.com/dbt-labs/dbt-core/issues/6703))
- Sort cli vars before hashing for partial parsing ([#6710](https://github.com/dbt-labs/dbt-core/issues/6710))
- [Regression] exposure_content referenced incorrectly ([#6738](https://github.com/dbt-labs/dbt-core/issues/6738))
- Remove pin on packaging and stop using it for prerelease comparisons ([#6834](https://github.com/dbt-labs/dbt-core/issues/6834))
- Readd depends_on.macros to SeedNode, to support seeds with hooks calling macros ([#6806](https://github.com/dbt-labs/dbt-core/issues/6806))
- Fix regression of --quiet cli parameter behavior ([#6749](https://github.com/dbt-labs/dbt-core/issues/6749))
- Ensure results from hooks contain nodes when processing them ([#6796](https://github.com/dbt-labs/dbt-core/issues/6796))
- Always flush stdout after logging ([#6901](https://github.com/dbt-labs/dbt-core/issues/6901))
- Reapply logging fixes which were accidentally reverted ([#6936](https://github.com/dbt-labs/dbt-core/issues/6936))
- Set relation_name in test nodes at compile time ([#6930](https://github.com/dbt-labs/dbt-core/issues/6930))
- Readd initialization events, --log-cache-events in new CLI ([#6933](https://github.com/dbt-labs/dbt-core/issues/6933))
- Fix previous state tests and disabled exposures, metrics ([#6752](https://github.com/dbt-labs/dbt-core/issues/6752), [#6753](https://github.com/dbt-labs/dbt-core/issues/6753))
- Make use of hashlib.md5() FIPS compliant ([#6900](https://github.com/dbt-labs/dbt-core/issues/6900))
### Docs
- update link to installation instructions ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Fix JSON path to overview docs ([dbt-docs/#366](https://github.com/dbt-labs/dbt-docs/issues/366))
- Searchable column descriptions ([dbt-docs/#140](https://github.com/dbt-labs/dbt-docs/issues/140), [dbt-docs/#322](https://github.com/dbt-labs/dbt-docs/issues/322), [dbt-docs/#369](https://github.com/dbt-labs/dbt-docs/issues/369))
### Under the Hood
- [CT-921] dbt compile works in click ([#5545](https://github.com/dbt-labs/dbt-core/issues/5545))
- Fix use of ConnectionReused logging event ([#6168](https://github.com/dbt-labs/dbt-core/issues/6168))
- Port docs tests to pytest ([#6573](https://github.com/dbt-labs/dbt-core/issues/6573))
- Update deprecated github action command ([#6153](https://github.com/dbt-labs/dbt-core/issues/6153))
- dbt snapshot works in click ([#5554](https://github.com/dbt-labs/dbt-core/issues/5554))
- dbt list working with click ([#5549](https://github.com/dbt-labs/dbt-core/issues/5549))
- Add dbt run-operation to click CLI ([#5552](https://github.com/dbt-labs/dbt-core/issues/5552))
- dbt build working with new click framework ([#5541](https://github.com/dbt-labs/dbt-core/issues/5541))
- dbt docs generate works with new click framework ([#5543](https://github.com/dbt-labs/dbt-core/issues/5543))
- Replaced the EmptyLine event with a more general Formatting event, and added a Note event. ([#6481](https://github.com/dbt-labs/dbt-core/issues/6481))
- Small optimization on manifest parsing benefitting large DAGs ([#6697](https://github.com/dbt-labs/dbt-core/issues/6697))
- Revised and simplified various structured logging events ([#6664](https://github.com/dbt-labs/dbt-core/issues/6664), [#6665](https://github.com/dbt-labs/dbt-core/issues/6665), [#6666](https://github.com/dbt-labs/dbt-core/issues/6666))
- dbt init works with click ([#5548](https://github.com/dbt-labs/dbt-core/issues/5548))
- [CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs commands ([#5544](https://github.com/dbt-labs/dbt-core/issues/5544), [#6722](https://github.com/dbt-labs/dbt-core/issues/6722))
- Migrate debug task to click ([#5546](https://github.com/dbt-labs/dbt-core/issues/5546))
- Optimized GraphQueue to remove graph analysis bottleneck in large dags. ([#6759](https://github.com/dbt-labs/dbt-core/issues/6759))
- Implement --version for click cli ([#6757](https://github.com/dbt-labs/dbt-core/issues/6757))
- [CT-1841] Convert custom target test to Pytest ([#6638](https://github.com/dbt-labs/dbt-core/issues/6638))
- Remove BigQuery-specific btye abbreviations ([#6741](https://github.com/dbt-labs/dbt-core/issues/6741))
- warn_error/warn_error_options mutual exclusivity in click ([#6579](https://github.com/dbt-labs/dbt-core/issues/6579))
- Enables the new Click Cli on the commandline! 🚀 ([#6784](https://github.com/dbt-labs/dbt-core/issues/6784))
- Lazily call --version ([#6812](https://github.com/dbt-labs/dbt-core/issues/6812))
- Moving simple_seed to adapter zone to help adapter test conversions ([#CT-1959](https://github.com/dbt-labs/dbt-core/issues/CT-1959))
- flags.THREADS defaults to None ([#6887](https://github.com/dbt-labs/dbt-core/issues/6887))
- Fixing target type exposure error ([#6928](https://github.com/dbt-labs/dbt-core/issues/6928))
- Test binary serialization of logging events ([#6852](https://github.com/dbt-labs/dbt-core/issues/6852))
### Dependencies
- Bump ubuntu from 22.04 to 23.04 ([#6865](https://github.com/dbt-labs/dbt-core/pull/6865))
- Revert hoisting dbt.cli.main into the dbt.name namespace ([#](https://github.com/dbt-labs/dbt-core/pull/))
### Contributors
- [@aezomz](https://github.com/aezomz) ([#2964](https://github.com/dbt-labs/dbt-core/issues/2964))
- [@boxysean](https://github.com/boxysean) ([#6697](https://github.com/dbt-labs/dbt-core/issues/6697))
- [@callum-mcdata](https://github.com/callum-mcdata) ([#6928](https://github.com/dbt-labs/dbt-core/issues/6928))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#6699](https://github.com/dbt-labs/dbt-core/issues/6699))
- [@davidbloss](https://github.com/davidbloss) ([#6153](https://github.com/dbt-labs/dbt-core/issues/6153))
- [@halvorlu](https://github.com/halvorlu) ([#366](https://github.com/dbt-labs/dbt-core/issues/366))
- [@nielspardon](https://github.com/nielspardon) ([#6900](https://github.com/dbt-labs/dbt-core/issues/6900))
- [@ryancharris](https://github.com/ryancharris) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@sungchun12](https://github.com/sungchun12) ([#6079](https://github.com/dbt-labs/dbt-core/issues/6079))
## Previous Releases
For information on prior major and minor releases, see their changelogs:
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)

View File

@@ -5,10 +5,10 @@
1. [About this document](#about-this-document)
2. [Getting the code](#getting-the-code)
3. [Setting up an environment](#setting-up-an-environment)
4. [Running dbt-core in development](#running-dbt-core-in-development)
4. [Running `dbt` in development](#running-dbt-core-in-development)
5. [Testing dbt-core](#testing)
6. [Debugging](#debugging)
7. [Adding or modifying a changelog entry](#adding-or-modifying-a-changelog-entry)
7. [Adding a changelog entry](#adding-a-changelog-entry)
8. [Submitting a Pull Request](#submitting-a-pull-request)
## About this document
@@ -56,7 +56,7 @@ There are some tools that will be helpful to you in developing locally. While th
These are the tools used in `dbt-core` development and testing:
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.8, 3.9, 3.10 and 3.11
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.7, 3.8, 3.9, 3.10 and 3.11
- [`pytest`](https://docs.pytest.org/en/latest/) to define, discover, and run tests
- [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting
- [`black`](https://github.com/psf/black) for code formatting
@@ -113,7 +113,7 @@ When installed in this way, any changes you make to your local copy of the sourc
With your virtualenv activated, the `dbt` script should point back to the source code you've cloned on your machine. You can verify this by running `which dbt`. This command should show you a path to an executable in your virtualenv.
Configure your [profile](https://docs.getdbt.com/docs/configure-your-profile) as necessary to connect to your target databases. It may be a good idea to add a new profile pointing to a local Postgres instance, or a specific test sandbox within your data warehouse if appropriate. Make sure to create a profile before running integration tests.
Configure your [profile](https://docs.getdbt.com/docs/configure-your-profile) as necessary to connect to your target databases. It may be a good idea to add a new profile pointing to a local Postgres instance, or a specific test sandbox within your data warehouse if appropriate.
## Testing
@@ -163,7 +163,7 @@ suites.
#### `tox`
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.8, Python 3.9, Python 3.10 and Python 3.11 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py38`. The configuration for these tests in located in `tox.ini`.
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.7, Python 3.8, Python 3.9, Python 3.10 and Python 3.11 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration for these tests in located in `tox.ini`.
#### `pytest`
@@ -171,10 +171,12 @@ Finally, you can also run a specific test or group of tests using [`pytest`](htt
```sh
# run all unit tests in a file
python3 -m pytest tests/unit/test_graph.py
python3 -m pytest test/unit/test_graph.py
# run a specific unit test
python3 -m pytest tests/unit/test_graph.py::GraphTest::test__dependency_list
# run specific Postgres functional tests
python3 -m pytest test/unit/test_graph.py::GraphTest::test__dependency_list
# run specific Postgres integration tests (old way)
python3 -m pytest -m profile_postgres test/integration/074_postgres_unlogged_table_tests
# run specific Postgres integration tests (new way)
python3 -m pytest tests/functional/sources
```
@@ -183,8 +185,9 @@ python3 -m pytest tests/functional/sources
### Unit, Integration, Functional?
Here are some general rules for adding tests:
* unit tests (`tests/unit`) dont need to access a database; "pure Python" tests should be written as unit tests
* functional tests (`tests/functional`) cover anything that interacts with a database, namely adapter
* unit tests (`test/unit` & `tests/unit`) dont need to access a database; "pure Python" tests should be written as unit tests
* functional tests (`test/integration` & `tests/functional`) cover anything that interacts with a database, namely adapter
* *everything in* `test/*` *is being steadily migrated to* `tests/*`
## Debugging

View File

@@ -3,13 +3,13 @@
# See `/docker` for a generic and production-ready docker file
##
FROM ubuntu:22.04
FROM ubuntu:23.04
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
software-properties-common gpg-agent \
software-properties-common \
&& add-apt-repository ppa:git-core/ppa -y \
&& apt-get dist-upgrade -y \
&& apt-get install -y --no-install-recommends \
@@ -30,9 +30,16 @@ RUN apt-get update \
unixodbc-dev \
&& add-apt-repository ppa:deadsnakes/ppa \
&& apt-get install -y \
python-is-python3 \
python-dev-is-python3 \
python \
python-dev \
python3-pip \
python3.6 \
python3.6-dev \
python3-pip \
python3.6-venv \
python3.7 \
python3.7-dev \
python3.7-venv \
python3.8 \
python3.8-dev \
python3.8-venv \

View File

@@ -30,18 +30,13 @@ CI_FLAGS =\
.PHONY: dev_req
dev_req: ## Installs dbt-* packages in develop mode along with only development dependencies.
@\
pip install -r dev-requirements.txt
pip install -r editable-requirements.txt
pip install -r dev-requirements.txt -r editable-requirements.txt
.PHONY: dev
dev: dev_req ## Installs dbt-* packages in develop mode along with development dependencies and pre-commit.
@\
pre-commit install
.PHONY: proto_types
proto_types: ## generates google protobuf python file from types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/types.proto
.PHONY: mypy
mypy: .env ## Runs mypy against staged changes for static type checking.
@\

View File

@@ -1,19 +1,14 @@
# these are all just exports, #noqa them so flake8 will be happy
# TODO: Should we still include this in the `adapters` namespace?
from dbt.contracts.connection import Credentials # noqa: F401
from dbt.adapters.base.meta import available # noqa: F401
from dbt.adapters.base.connections import BaseConnectionManager # noqa: F401
from dbt.adapters.base.relation import ( # noqa: F401
from dbt.contracts.connection import Credentials # noqa
from dbt.adapters.base.meta import available # noqa
from dbt.adapters.base.connections import BaseConnectionManager # noqa
from dbt.adapters.base.relation import ( # noqa
BaseRelation,
RelationType,
SchemaSearchMap,
)
from dbt.adapters.base.column import Column # noqa: F401
from dbt.adapters.base.impl import ( # noqa: F401
AdapterConfig,
BaseAdapter,
PythonJobHelper,
ConstraintSupport,
)
from dbt.adapters.base.plugin import AdapterPlugin # noqa: F401
from dbt.adapters.base.column import Column # noqa
from dbt.adapters.base.impl import AdapterConfig, BaseAdapter, PythonJobHelper # noqa
from dbt.adapters.base.plugin import AdapterPlugin # noqa

View File

@@ -60,7 +60,6 @@ class Column:
"float",
"double precision",
"float8",
"double",
]
def is_integer(self) -> bool:

View File

@@ -2,48 +2,45 @@ import abc
from concurrent.futures import as_completed, Future
from contextlib import contextmanager
from datetime import datetime
from enum import Enum
import time
from itertools import chain
from typing import (
Any,
Optional,
Tuple,
Callable,
Dict,
Iterable,
Iterator,
Type,
Dict,
Any,
List,
Mapping,
Optional,
Iterator,
Set,
Tuple,
Type,
Union,
)
from dbt.contracts.graph.nodes import ColumnLevelConstraint, ConstraintType, ModelLevelConstraint
import agate
import pytz
from dbt.exceptions import (
DbtInternalError,
DbtRuntimeError,
DbtValidationError,
MacroArgTypeError,
MacroResultError,
QuoteConfigTypeError,
NotImplementedError,
NullRelationCacheAttemptedError,
NullRelationDropAttemptedError,
QuoteConfigTypeError,
RelationReturnedMultipleResultsError,
RenameToNoneAttemptedError,
DbtRuntimeError,
SnapshotTargetIncompleteError,
SnapshotTargetNotSnapshotTableError,
UnexpectedNonTimestampError,
UnexpectedNullError,
UnexpectedNonTimestampError,
)
from dbt.adapters.protocol import AdapterConfig, ConnectionManagerProtocol
from dbt.adapters.protocol import (
AdapterConfig,
ConnectionManagerProtocol,
)
from dbt.clients.agate_helper import empty_table, merge_tables, table_from_rows
from dbt.clients.jinja import MacroGenerator
from dbt.contracts.graph.manifest import Manifest, MacroManifest
@@ -55,8 +52,6 @@ from dbt.events.types import (
CodeExecution,
CodeExecutionStatus,
CatalogGenerationError,
ConstraintNotSupported,
ConstraintNotEnforced,
)
from dbt.utils import filter_null_values, executor, cast_to_str, AttrDict
@@ -70,19 +65,13 @@ from dbt.adapters.base.relation import (
)
from dbt.adapters.base import Column as BaseColumn
from dbt.adapters.base import Credentials
from dbt.adapters.cache import RelationsCache, _make_ref_key_dict
from dbt import deprecations
from dbt.adapters.cache import RelationsCache, _make_ref_key_msg
GET_CATALOG_MACRO_NAME = "get_catalog"
FRESHNESS_MACRO_NAME = "collect_freshness"
class ConstraintSupport(str, Enum):
ENFORCED = "enforced"
NOT_ENFORCED = "not_enforced"
NOT_SUPPORTED = "not_supported"
def _expect_row_value(key: str, row: agate.Row):
if key not in row.keys():
raise DbtInternalError(
@@ -187,7 +176,6 @@ class BaseAdapter(metaclass=AdapterMeta):
- truncate_relation
- rename_relation
- get_columns_in_relation
- get_column_schema_from_query
- expand_column_types
- list_relations_without_caching
- is_cancelable
@@ -214,14 +202,6 @@ class BaseAdapter(metaclass=AdapterMeta):
# for use in materializations
AdapterSpecificConfigs: Type[AdapterConfig] = AdapterConfig
CONSTRAINT_SUPPORT = {
ConstraintType.check: ConstraintSupport.NOT_SUPPORTED,
ConstraintType.not_null: ConstraintSupport.ENFORCED,
ConstraintType.unique: ConstraintSupport.NOT_ENFORCED,
ConstraintType.primary_key: ConstraintSupport.NOT_ENFORCED,
ConstraintType.foreign_key: ConstraintSupport.ENFORCED,
}
def __init__(self, config):
self.config = config
self.cache = RelationsCache()
@@ -274,7 +254,7 @@ class BaseAdapter(metaclass=AdapterMeta):
@available.parse(lambda *a, **k: ("", empty_table()))
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
self, sql: str, auto_begin: bool = False, fetch: bool = False
) -> Tuple[AdapterResponse, agate.Table]:
"""Execute the given SQL. This is a thin wrapper around
ConnectionManager.execute.
@@ -283,35 +263,10 @@ class BaseAdapter(metaclass=AdapterMeta):
:param bool auto_begin: If set, and dbt is not currently inside a
transaction, automatically begin one.
:param bool fetch: If set, fetch results.
:param Optional[int] limit: If set, only fetch n number of rows
:return: A tuple of the query status and results (empty if fetch=False).
:rtype: Tuple[AdapterResponse, agate.Table]
"""
return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)
def validate_sql(self, sql: str) -> AdapterResponse:
"""Submit the given SQL to the engine for validation, but not execution.
This should throw an appropriate exception if the input SQL is invalid, although
in practice that will generally be handled by delegating to an existing method
for execution and allowing the error handler to take care of the rest.
:param str sql: The sql to validate
"""
raise NotImplementedError("`validate_sql` is not implemented for this adapter!")
@available.parse(lambda *a, **k: [])
def get_column_schema_from_query(self, sql: str) -> List[BaseColumn]:
"""Get a list of the Columns with names and data types from the given sql."""
_, cursor = self.connections.add_select_query(sql)
columns = [
self.Column.create(
column_name, self.connections.data_type_code_to_name(column_type_code)
)
# https://peps.python.org/pep-0249/#description
for column_name, column_type_code, *_ in cursor.description
]
return columns
return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch)
@available.parse(lambda *a, **k: ("", empty_table()))
def get_partitions_metadata(self, table: str) -> Tuple[agate.Table]:
@@ -395,7 +350,7 @@ class BaseAdapter(metaclass=AdapterMeta):
return {
self.Relation.create_from(self.config, node).without_identifier()
for node in manifest.nodes.values()
if (node.is_relational and not node.is_ephemeral_model and not node.is_external_node)
if (node.is_relational and not node.is_ephemeral_model)
}
def _get_catalog_schemas(self, manifest: Manifest) -> SchemaSearchMap:
@@ -426,7 +381,7 @@ class BaseAdapter(metaclass=AdapterMeta):
return info_schema_name_map
def _relations_cache_for_schemas(
self, manifest: Manifest, cache_schemas: Optional[Set[BaseRelation]] = None
self, manifest: Manifest, cache_schemas: Set[BaseRelation] = None
) -> None:
"""Populate the relations cache for the given schemas. Returns an
iterable of the schemas populated, as strings.
@@ -462,7 +417,7 @@ class BaseAdapter(metaclass=AdapterMeta):
self,
manifest: Manifest,
clear: bool = False,
required_schemas: Optional[Set[BaseRelation]] = None,
required_schemas: Set[BaseRelation] = None,
) -> None:
"""Run a query that gets a populated cache of the relations in the
database and set the cache on this adapter.
@@ -748,23 +703,11 @@ class BaseAdapter(metaclass=AdapterMeta):
# we can't build the relations cache because we don't have a
# manifest so we can't run any operations.
relations = self.list_relations_without_caching(schema_relation)
# if the cache is already populated, add this schema in
# otherwise, skip updating the cache and just ignore
if self.cache:
for relation in relations:
self.cache.add(relation)
if not relations:
# it's possible that there were no relations in some schemas. We want
# to insert the schemas we query into the cache's `.schemas` attribute
# so we can check it later
self.cache.update_schemas([(database, schema)])
fire_event(
ListRelations(
database=cast_to_str(database),
schema=schema,
relations=[_make_ref_key_dict(x) for x in relations],
relations=[_make_ref_key_msg(x) for x in relations],
)
)
@@ -796,6 +739,7 @@ class BaseAdapter(metaclass=AdapterMeta):
schema: str,
identifier: str,
) -> List[BaseRelation]:
matches = []
search = self._make_match_kwargs(database, schema, identifier)
@@ -996,7 +940,7 @@ class BaseAdapter(metaclass=AdapterMeta):
manifest: Optional[Manifest] = None,
project: Optional[str] = None,
context_override: Optional[Dict[str, Any]] = None,
kwargs: Optional[Dict[str, Any]] = None,
kwargs: Dict[str, Any] = None,
text_only_columns: Optional[Iterable[str]] = None,
) -> AttrDict:
"""Look macro_name up in the manifest and execute its results.
@@ -1073,6 +1017,7 @@ class BaseAdapter(metaclass=AdapterMeta):
schemas: Set[str],
manifest: Manifest,
) -> agate.Table:
kwargs = {"information_schema": information_schema, "schemas": schemas}
table = self.execute_macro(
GET_CATALOG_MACRO_NAME,
@@ -1114,7 +1059,7 @@ class BaseAdapter(metaclass=AdapterMeta):
loaded_at_field: str,
filter: Optional[str],
manifest: Optional[Manifest] = None,
) -> Tuple[Optional[AdapterResponse], Dict[str, Any]]:
) -> Tuple[AdapterResponse, Dict[str, Any]]:
"""Calculate the freshness of sources in dbt, and return it"""
kwargs: Dict[str, Any] = {
"source": source,
@@ -1123,19 +1068,8 @@ class BaseAdapter(metaclass=AdapterMeta):
}
# run the macro
# in older versions of dbt-core, the 'collect_freshness' macro returned the table of results directly
# starting in v1.5, by default, we return both the table and the adapter response (metadata about the query)
result: Union[
AttrDict, # current: contains AdapterResponse + agate.Table
agate.Table, # previous: just table
]
result = self.execute_macro(FRESHNESS_MACRO_NAME, kwargs=kwargs, manifest=manifest)
if isinstance(result, agate.Table):
deprecations.warn("collect-freshness-return-signature")
adapter_response = None
table = result
else:
adapter_response, table = result.response, result.table # type: ignore[attr-defined]
adapter_response, table = result.response, result.table # type: ignore[attr-defined]
# now we have a 1-row table of the maximum `loaded_at_field` value and
# the current time according to the db.
if len(table) != 1 or len(table[0]) != 2:
@@ -1316,119 +1250,6 @@ class BaseAdapter(metaclass=AdapterMeta):
# This returns a callable macro
return model_context[macro_name]
@classmethod
def _parse_column_constraint(cls, raw_constraint: Dict[str, Any]) -> ColumnLevelConstraint:
try:
ColumnLevelConstraint.validate(raw_constraint)
return ColumnLevelConstraint.from_dict(raw_constraint)
except Exception:
raise DbtValidationError(f"Could not parse constraint: {raw_constraint}")
@classmethod
def render_column_constraint(cls, constraint: ColumnLevelConstraint) -> Optional[str]:
"""Render the given constraint as DDL text. Should be overriden by adapters which need custom constraint
rendering."""
constraint_expression = constraint.expression or ""
rendered_column_constraint = None
if constraint.type == ConstraintType.check and constraint_expression:
rendered_column_constraint = f"check ({constraint_expression})"
elif constraint.type == ConstraintType.not_null:
rendered_column_constraint = f"not null {constraint_expression}"
elif constraint.type == ConstraintType.unique:
rendered_column_constraint = f"unique {constraint_expression}"
elif constraint.type == ConstraintType.primary_key:
rendered_column_constraint = f"primary key {constraint_expression}"
elif constraint.type == ConstraintType.foreign_key and constraint_expression:
rendered_column_constraint = f"references {constraint_expression}"
elif constraint.type == ConstraintType.custom and constraint_expression:
rendered_column_constraint = constraint_expression
if rendered_column_constraint:
rendered_column_constraint = rendered_column_constraint.strip()
return rendered_column_constraint
@available
@classmethod
def render_raw_columns_constraints(cls, raw_columns: Dict[str, Dict[str, Any]]) -> List:
rendered_column_constraints = []
for v in raw_columns.values():
col_name = cls.quote(v["name"]) if v.get("quote") else v["name"]
rendered_column_constraint = [f"{col_name} {v['data_type']}"]
for con in v.get("constraints", None):
constraint = cls._parse_column_constraint(con)
c = cls.process_parsed_constraint(constraint, cls.render_column_constraint)
if c is not None:
rendered_column_constraint.append(c)
rendered_column_constraints.append(" ".join(rendered_column_constraint))
return rendered_column_constraints
@classmethod
def process_parsed_constraint(
cls, parsed_constraint: Union[ColumnLevelConstraint, ModelLevelConstraint], render_func
) -> Optional[str]:
if (
parsed_constraint.warn_unsupported
and cls.CONSTRAINT_SUPPORT[parsed_constraint.type] == ConstraintSupport.NOT_SUPPORTED
):
warn_or_error(
ConstraintNotSupported(constraint=parsed_constraint.type.value, adapter=cls.type())
)
if (
parsed_constraint.warn_unenforced
and cls.CONSTRAINT_SUPPORT[parsed_constraint.type] == ConstraintSupport.NOT_ENFORCED
):
warn_or_error(
ConstraintNotEnforced(constraint=parsed_constraint.type.value, adapter=cls.type())
)
if cls.CONSTRAINT_SUPPORT[parsed_constraint.type] != ConstraintSupport.NOT_SUPPORTED:
return render_func(parsed_constraint)
return None
@classmethod
def _parse_model_constraint(cls, raw_constraint: Dict[str, Any]) -> ModelLevelConstraint:
try:
ModelLevelConstraint.validate(raw_constraint)
c = ModelLevelConstraint.from_dict(raw_constraint)
return c
except Exception:
raise DbtValidationError(f"Could not parse constraint: {raw_constraint}")
@available
@classmethod
def render_raw_model_constraints(cls, raw_constraints: List[Dict[str, Any]]) -> List[str]:
return [c for c in map(cls.render_raw_model_constraint, raw_constraints) if c is not None]
@classmethod
def render_raw_model_constraint(cls, raw_constraint: Dict[str, Any]) -> Optional[str]:
constraint = cls._parse_model_constraint(raw_constraint)
return cls.process_parsed_constraint(constraint, cls.render_model_constraint)
@classmethod
def render_model_constraint(cls, constraint: ModelLevelConstraint) -> Optional[str]:
"""Render the given constraint as DDL text. Should be overriden by adapters which need custom constraint
rendering."""
constraint_prefix = f"constraint {constraint.name} " if constraint.name else ""
column_list = ", ".join(constraint.columns)
if constraint.type == ConstraintType.check and constraint.expression:
return f"{constraint_prefix}check ({constraint.expression})"
elif constraint.type == ConstraintType.unique:
constraint_expression = f" {constraint.expression}" if constraint.expression else ""
return f"{constraint_prefix}unique{constraint_expression} ({column_list})"
elif constraint.type == ConstraintType.primary_key:
constraint_expression = f" {constraint.expression}" if constraint.expression else ""
return f"{constraint_prefix}primary key{constraint_expression} ({column_list})"
elif constraint.type == ConstraintType.foreign_key and constraint.expression:
return f"{constraint_prefix}foreign key ({column_list}) references {constraint.expression}"
elif constraint.type == ConstraintType.custom and constraint.expression:
return f"{constraint_prefix}{constraint.expression}"
else:
return None
COLUMNS_EQUAL_SQL = """
with diff_count as (
@@ -1462,6 +1283,7 @@ join diff_count using (id)
def catch_as_completed(
futures, # typing: List[Future[agate.Table]]
) -> Tuple[agate.Table, List[Exception]]:
# catalogs: agate.Table = agate.Table(rows=[])
tables: List[agate.Table] = []
exceptions: List[Exception] = []

Some files were not shown because too many files have changed in this diff Show More