Compare commits

...

21 Commits

Author SHA1 Message Date
Jeremy Cohen
1d8c5af513 Experiment: column types from empty query 2023-01-20 21:15:48 +01:00
Peter Webb
6954c4df1b CT-1786: Port docs tests to pytest (#6608)
* CT-1786: Port docs tets to pytest

* Add generated CLI API docs

* CT-1786: Comply with the new style requirements

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-19 11:11:17 -05:00
dave-connors-3
f841a7ca76 add backwards compatibility and default argument for incremental_predicates (#6628)
* add backwards compatibility and default argument

* changie <3

* Update .changes/unreleased/Fixes-20230117-101342.yaml

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-01-19 15:20:19 +01:00
Jeremy Cohen
07a004b301 convert 062_defer_state_tests (#6616)
* Fix --favor-state flag

* Convert 062_defer_state_tests

* Revert "Fix --favor-state flag"

This reverts commit ccbdcbad98b26822629364e6fdbd2780db0c20d3.

* Reformat

* Revert "Revert "Fix --favor-state flag""

This reverts commit fa9d2a09d6.
2023-01-19 11:00:09 +01:00
Jeremy Cohen
b05582de39 mv on_schema_change tests -> "adapter zone" (#6618)
* Mv incremental on_schema_change tests to 'adapter zone'

* Use type_string()

* Cleanup
2023-01-19 10:12:59 +01:00
Jeremy Cohen
fa7c4d19f0 Respect quoting config in dbt-py models (#6620)
* Respect quoting for 'this' in dbt-py models #6619

* Respect quoting for ref/source in dbt-py models #6103

* Add changelog entries
2023-01-19 09:34:08 +01:00
Jeremy Cohen
066346faa2 convert 038_caching_tests (#6612)
* convert 038_caching_tests

* Adapt for dbt-snowflake

* PR feedback

* Reformat
2023-01-18 22:37:50 +01:00
Emily Rockman
0a03355ceb update test matrix (#6604) 2023-01-18 14:16:34 -06:00
David Bloss
43e24c5ae6 update gh action set-output variables (#6635)
* update gh action set-output variables

* add changie file
2023-01-18 11:23:13 -06:00
Gerda Shank
89d111a5f6 CT 1440 Fix code to emit ConnectionReused event (#6605)
* Refactor "set_connection_name" to properly handle reused connection

* Update test

* Changie

* Limit test of ConnectionUsed events to non-Windows
2023-01-17 13:18:07 -05:00
Gerda Shank
e1b5e68904 Convert 068_partial_parsing_tests (#6614)
* Convert partial parsing tests

* reformat
2023-01-17 12:22:31 -05:00
Jeremy Cohen
065ab2ebc2 Reformat tests/ (#6622)
* Run black + flake8 on tests dir

* Run pre-commit
2023-01-16 16:39:54 +01:00
Michelle Ark
20c95a4993 convert 057_run_query_tests (#6607) 2023-01-13 15:02:17 -05:00
Michelle Ark
c40b488cb4 convert 037_external_references (#6589) 2023-01-13 14:15:03 -05:00
Chenyu Li
585e7c59e8 migrate ref override (#6600) 2023-01-13 09:15:04 -08:00
Michelle Ark
7077c47551 converting 022_test_timezones (#6588) 2023-01-13 10:52:21 -05:00
Chenyu Li
f789b2535a convert adapter table materialization test (#6595) 2023-01-12 16:05:51 -08:00
Emily Rockman
2bfc6917e2 finish message rename in types.proto (#6594)
* finish message rename in types.proto

* add new parameter
2023-01-12 14:33:56 -06:00
github-actions[bot]
d74ae19523 Bumping version to 1.5.0a1 and generate changelog (#6593)
* Bumping version to 1.5.0a1 and generate CHANGELOG

* make relevvant changelog updates

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-01-12 12:59:28 -06:00
Michelle Ark
1c7c23ac73 convert 043_custom_alias_tests (#6590) 2023-01-12 11:07:26 -05:00
Jeremy Cohen
86e8722cd8 Call update_event_status earlier + rename an event (#6572)
* Rename HookFinished -> FinishedRunningStats

* Move update_event_status earlier when node finishes

* Add changelog entry

* Add update_event_status for skip

* Update changelog entry
2023-01-12 09:34:36 +01:00
316 changed files with 4951 additions and 5161 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.4.0b1
current_version = 1.5.0a1
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)

View File

@@ -3,6 +3,7 @@
For information on prior major and minor releases, see their changelogs:
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)

View File

@@ -1,88 +0,0 @@
## dbt-core 1.4.0-b1 - December 15, 2022
### Features
- Added favor-state flag to optionally favor state nodes even if unselected node exists ([#2968](https://github.com/dbt-labs/dbt-core/issues/2968))
- Update structured logging. Convert to using protobuf messages. Ensure events are enriched with node_info. ([#5610](https://github.com/dbt-labs/dbt-core/issues/5610))
- Friendlier error messages when packages.yml is malformed ([#5486](https://github.com/dbt-labs/dbt-core/issues/5486))
- Allow partitions in external tables to be supplied as a list ([#5929](https://github.com/dbt-labs/dbt-core/issues/5929))
- extend -f flag shorthand for seed command ([#5990](https://github.com/dbt-labs/dbt-core/issues/5990))
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201))
- Adding tarball install method for packages. Allowing package tarball to be specified via url in the packages.yaml. ([#4205](https://github.com/dbt-labs/dbt-core/issues/4205))
- Added an md5 function to the base context ([#6246](https://github.com/dbt-labs/dbt-core/issues/6246))
- Exposures support metrics in lineage ([#6057](https://github.com/dbt-labs/dbt-core/issues/6057))
- Add support for Python 3.11 ([#6147](https://github.com/dbt-labs/dbt-core/issues/6147))
- incremental predicates ([#5680](https://github.com/dbt-labs/dbt-core/issues/5680))
### Fixes
- Account for disabled flags on models in schema files more completely ([#3992](https://github.com/dbt-labs/dbt-core/issues/3992))
- Add validation of enabled config for metrics, exposures and sources ([#6030](https://github.com/dbt-labs/dbt-core/issues/6030))
- check length of args of python model function before accessing it ([#6041](https://github.com/dbt-labs/dbt-core/issues/6041))
- Add functors to ensure event types with str-type attributes are initialized to spec, even when provided non-str type params. ([#5436](https://github.com/dbt-labs/dbt-core/issues/5436))
- Allow hooks to fail without halting execution flow ([#5625](https://github.com/dbt-labs/dbt-core/issues/5625))
- Clarify Error Message for how many models are allowed in a Python file ([#6245](https://github.com/dbt-labs/dbt-core/issues/6245))
- After this, will be possible to use default values for dbt.config.get ([#6309](https://github.com/dbt-labs/dbt-core/issues/6309))
- Use full path for writing manifest ([#6055](https://github.com/dbt-labs/dbt-core/issues/6055))
- [CT-1284] Change Python model default materialization to table ([#6345](https://github.com/dbt-labs/dbt-core/issues/6345))
- Repair a regression which prevented basic logging before the logging subsystem is completely configured. ([#6434](https://github.com/dbt-labs/dbt-core/issues/6434))
### Docs
- minor doc correction ([dbt-docs/#5791](https://github.com/dbt-labs/dbt-docs/issues/5791))
- Generate API docs for new CLI interface ([dbt-docs/#5528](https://github.com/dbt-labs/dbt-docs/issues/5528))
- ([dbt-docs/#5880](https://github.com/dbt-labs/dbt-docs/issues/5880))
- Fix rendering of sample code for metrics ([dbt-docs/#323](https://github.com/dbt-labs/dbt-docs/issues/323))
- Alphabetize `core/dbt/README.md` ([dbt-docs/#6368](https://github.com/dbt-labs/dbt-docs/issues/6368))
### Under the Hood
- Put black config in explicit config ([#5946](https://github.com/dbt-labs/dbt-core/issues/5946))
- Added flat_graph attribute the Manifest class's deepcopy() coverage ([#5809](https://github.com/dbt-labs/dbt-core/issues/5809))
- Add mypy configs so `mypy` passes from CLI ([#5983](https://github.com/dbt-labs/dbt-core/issues/5983))
- Exception message cleanup. ([#6023](https://github.com/dbt-labs/dbt-core/issues/6023))
- Add dmypy cache to gitignore ([#6028](https://github.com/dbt-labs/dbt-core/issues/6028))
- Provide useful errors when the value of 'materialized' is invalid ([#5229](https://github.com/dbt-labs/dbt-core/issues/5229))
- Clean up string formatting ([#6068](https://github.com/dbt-labs/dbt-core/issues/6068))
- Fixed extra whitespace in strings introduced by black. ([#1350](https://github.com/dbt-labs/dbt-core/issues/1350))
- Remove the 'root_path' field from most nodes ([#6171](https://github.com/dbt-labs/dbt-core/issues/6171))
- Combine certain logging events with different levels ([#6173](https://github.com/dbt-labs/dbt-core/issues/6173))
- Convert threading tests to pytest ([#5942](https://github.com/dbt-labs/dbt-core/issues/5942))
- Convert postgres index tests to pytest ([#5770](https://github.com/dbt-labs/dbt-core/issues/5770))
- Convert use color tests to pytest ([#5771](https://github.com/dbt-labs/dbt-core/issues/5771))
- Add github actions workflow to generate high level CLI API docs ([#5942](https://github.com/dbt-labs/dbt-core/issues/5942))
- Functionality-neutral refactor of event logging system to improve encapsulation and modularity. ([#6139](https://github.com/dbt-labs/dbt-core/issues/6139))
- Consolidate ParsedNode and CompiledNode classes ([#6383](https://github.com/dbt-labs/dbt-core/issues/6383))
- Prevent doc gen workflow from running on forks ([#6386](https://github.com/dbt-labs/dbt-core/issues/6386))
- Fix intermittent database connection failure in Windows CI test ([#6394](https://github.com/dbt-labs/dbt-core/issues/6394))
- Refactor and clean up manifest nodes ([#6426](https://github.com/dbt-labs/dbt-core/issues/6426))
- Restore important legacy logging behaviors, following refactor which removed them ([#6437](https://github.com/dbt-labs/dbt-core/issues/6437))
### Dependencies
- Update pathspec requirement from ~=0.9.0 to >=0.9,<0.11 in /core ([#5917](https://github.com/dbt-labs/dbt-core/pull/5917))
- Bump black from 22.8.0 to 22.10.0 ([#6019](https://github.com/dbt-labs/dbt-core/pull/6019))
- Bump mashumaro[msgpack] from 3.0.4 to 3.1.1 in /core ([#6108](https://github.com/dbt-labs/dbt-core/pull/6108))
- Update colorama requirement from <0.4.6,>=0.3.9 to >=0.3.9,<0.4.7 in /core ([#6144](https://github.com/dbt-labs/dbt-core/pull/6144))
- Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904))
### Contributors
- [@andy-clapson](https://github.com/andy-clapson) ([dbt-docs/#5791](https://github.com/dbt-labs/dbt-docs/issues/5791))
- [@chamini2](https://github.com/chamini2) ([#6041](https://github.com/dbt-labs/dbt-core/issues/6041))
- [@daniel-murray](https://github.com/daniel-murray) ([#2968](https://github.com/dbt-labs/dbt-core/issues/2968))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#5990](https://github.com/dbt-labs/dbt-core/issues/5990))
- [@dbeatty10](https://github.com/dbeatty10) ([dbt-docs/#6368](https://github.com/dbt-labs/dbt-docs/issues/6368), [#6394](https://github.com/dbt-labs/dbt-core/issues/6394))
- [@devmessias](https://github.com/devmessias) ([#6309](https://github.com/dbt-labs/dbt-core/issues/6309))
- [@eve-johns](https://github.com/eve-johns) ([#6068](https://github.com/dbt-labs/dbt-core/issues/6068))
- [@haritamar](https://github.com/haritamar) ([#6246](https://github.com/dbt-labs/dbt-core/issues/6246))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5486](https://github.com/dbt-labs/dbt-core/issues/5486))
- [@josephberni](https://github.com/josephberni) ([#2968](https://github.com/dbt-labs/dbt-core/issues/2968))
- [@joshuataylor](https://github.com/joshuataylor) ([#6147](https://github.com/dbt-labs/dbt-core/issues/6147))
- [@justbldwn](https://github.com/justbldwn) ([#6245](https://github.com/dbt-labs/dbt-core/issues/6245))
- [@luke-bassett](https://github.com/luke-bassett) ([#1350](https://github.com/dbt-labs/dbt-core/issues/1350))
- [@max-sixty](https://github.com/max-sixty) ([#5946](https://github.com/dbt-labs/dbt-core/issues/5946), [#5983](https://github.com/dbt-labs/dbt-core/issues/5983), [#6028](https://github.com/dbt-labs/dbt-core/issues/6028))
- [@paulbenschmidt](https://github.com/paulbenschmidt) ([dbt-docs/#5880](https://github.com/dbt-labs/dbt-docs/issues/5880))
- [@pgoslatara](https://github.com/pgoslatara) ([#5929](https://github.com/dbt-labs/dbt-core/issues/5929))
- [@racheldaniel](https://github.com/racheldaniel) ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201))
- [@timle2](https://github.com/timle2) ([#4205](https://github.com/dbt-labs/dbt-core/issues/4205))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#5680](https://github.com/dbt-labs/dbt-core/issues/5680))

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Update pathspec requirement from ~=0.9.0 to >=0.9,<0.11 in /core"
time: 2022-09-23T00:06:46.00000Z
custom:
Author: dependabot[bot]
PR: "5917"

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Bump black from 22.8.0 to 22.10.0"
time: 2022-10-07T00:08:48.00000Z
custom:
Author: dependabot[bot]
PR: "6019"

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Bump mashumaro[msgpack] from 3.0.4 to 3.1.1 in /core"
time: 2022-10-20T00:07:53.00000Z
custom:
Author: dependabot[bot]
PR: "6108"

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Update colorama requirement from <0.4.6,>=0.3.9 to >=0.3.9,<0.4.7 in /core"
time: 2022-10-26T00:09:10.00000Z
custom:
Author: dependabot[bot]
PR: "6144"

View File

@@ -1,7 +0,0 @@
kind: "Dependencies"
body: "Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core"
time: 2022-12-05T00:21:18.00000Z
custom:
Author: dependabot[bot]
Issue: 4904
PR: 6375

View File

@@ -1,6 +0,0 @@
kind: Docs
body: minor doc correction
time: 2022-09-08T15:41:57.689162-04:00
custom:
Author: andy-clapson
Issue: "5791"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Generate API docs for new CLI interface
time: 2022-10-07T09:06:56.446078-05:00
custom:
Author: stu-k
Issue: "5528"

View File

@@ -1,5 +0,0 @@
kind: Docs
time: 2022-10-17T17:14:11.715348-05:00
custom:
Author: paulbenschmidt
Issue: "5880"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Fix rendering of sample code for metrics
time: 2022-11-16T15:57:43.204201+01:00
custom:
Author: jtcohen6
Issue: "323"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Alphabetize `core/dbt/README.md`
time: 2022-12-02T15:05:23.695333-07:00
custom:
Author: dbeatty10
Issue: "6368"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Added favor-state flag to optionally favor state nodes even if unselected node
exists
time: 2022-04-08T16:54:59.696564+01:00
custom:
Author: daniel-murray josephberni
Issue: "5016"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Update structured logging. Convert to using protobuf messages. Ensure events are enriched with node_info.
time: 2022-08-17T15:48:57.225267-04:00
custom:
Author: gshank
Issue: "5610"

View File

@@ -1,7 +0,0 @@
kind: Features
body: incremental predicates
time: 2022-08-23T08:57:27.640804-05:00
custom:
Author: dave-connors-3
Issue: "5680"
PR: "5702"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Friendlier error messages when packages.yml is malformed
time: 2022-09-12T12:59:35.121188+01:00
custom:
Author: jared-rimmer
Issue: "5486"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Allow partitions in external tables to be supplied as a list
time: 2022-09-25T21:16:51.051239654+02:00
custom:
Author: pgoslatara
Issue: "5929"

View File

@@ -1,6 +0,0 @@
kind: Features
body: extend -f flag shorthand for seed command
time: 2022-10-03T11:07:05.381632-05:00
custom:
Author: dave-connors-3
Issue: "5990"

View File

@@ -1,7 +0,0 @@
kind: Features
body: This pulls the profile name from args when constructing a RuntimeConfig in lib.py,
enabling the dbt-server to override the value that's in the dbt_project.yml
time: 2022-11-02T15:00:03.000805-05:00
custom:
Author: racheldaniel
Issue: "6201"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Adding tarball install method for packages. Allowing package tarball to be specified
via url in the packages.yaml.
time: 2022-11-07T10:50:18.464545-05:00
custom:
Author: timle2
Issue: "4205"
PR: "4689"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Added an md5 function to the base context
time: 2022-11-14T18:52:07.788593+02:00
custom:
Author: haritamar
Issue: "6246"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Exposures support metrics in lineage
time: 2022-11-30T11:29:13.256034-05:00
custom:
Author: michelleark
Issue: "6057"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Add support for Python 3.11
time: 2022-12-06T15:07:04.753127+01:00
custom:
Author: joshuataylor MichelleArk jtcohen6
Issue: "6147"
PR: "6326"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Account for disabled flags on models in schema files more completely
time: 2022-09-16T10:48:54.162273-05:00
custom:
Author: emmyoop
Issue: "3992"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Add validation of enabled config for metrics, exposures and sources
time: 2022-10-10T11:32:18.752322-05:00
custom:
Author: emmyoop
Issue: "6030"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: check length of args of python model function before accessing it
time: 2022-10-11T16:07:15.464093-04:00
custom:
Author: chamini2
Issue: "6041"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Add functors to ensure event types with str-type attributes are initialized
to spec, even when provided non-str type params.
time: 2022-10-16T17:37:42.846683-07:00
custom:
Author: versusfacit
Issue: "5436"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Allow hooks to fail without halting execution flow
time: 2022-11-07T09:53:14.340257-06:00
custom:
Author: ChenyuLInx
Issue: "5625"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Clarify Error Message for how many models are allowed in a Python file
time: 2022-11-15T08:10:21.527884-05:00
custom:
Author: justbldwn
Issue: "6245"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: After this, will be possible to use default values for dbt.config.get
time: 2022-11-24T16:34:19.039512764-03:00
custom:
Author: devmessias
Issue: "6309"
PR: "6317"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Use full path for writing manifest
time: 2022-12-02T16:48:59.029519-05:00
custom:
Author: gshank
Issue: "6055"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: '[CT-1284] Change Python model default materialization to table'
time: 2022-12-13T11:26:20.550017-08:00
custom:
Author: aranke
Issue: "5989"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Repair a regression which prevented basic logging before the logging subsystem
is completely configured.
time: 2022-12-14T15:53:07.396512-05:00
custom:
Author: peterallenwebb
Issue: "6434"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Put black config in explicit config
time: 2022-09-27T19:42:59.241433-07:00
custom:
Author: max-sixty
Issue: "5946"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Added flat_graph attribute the Manifest class's deepcopy() coverage
time: 2022-09-29T13:44:06.275941-04:00
custom:
Author: peterallenwebb
Issue: "5809"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Add mypy configs so `mypy` passes from CLI
time: 2022-10-05T12:03:10.061263-07:00
custom:
Author: max-sixty
Issue: "5983"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Exception message cleanup.
time: 2022-10-07T09:46:27.682872-05:00
custom:
Author: emmyoop
Issue: "6023"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Add dmypy cache to gitignore
time: 2022-10-07T14:00:44.227644-07:00
custom:
Author: max-sixty
Issue: "6028"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Provide useful errors when the value of 'materialized' is invalid
time: 2022-10-13T18:19:12.167548-04:00
custom:
Author: peterallenwebb
Issue: "5229"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Fixed extra whitespace in strings introduced by black.
time: 2022-10-17T15:15:11.499246-05:00
custom:
Author: luke-bassett
Issue: "1350"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Clean up string formatting
time: 2022-10-17T15:58:44.676549-04:00
custom:
Author: eve-johns
Issue: "6068"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Remove the 'root_path' field from most nodes
time: 2022-10-28T10:48:37.687886-04:00
custom:
Author: gshank
Issue: "6171"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Combine certain logging events with different levels
time: 2022-10-28T11:03:44.887836-04:00
custom:
Author: gshank
Issue: "6173"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Convert threading tests to pytest
time: 2022-11-08T07:45:50.589147-06:00
custom:
Author: stu-k
Issue: "5942"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Convert postgres index tests to pytest
time: 2022-11-08T11:56:33.743042-06:00
custom:
Author: stu-k
Issue: "5770"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Convert use color tests to pytest
time: 2022-11-08T13:31:04.788547-06:00
custom:
Author: stu-k
Issue: "5771"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Add github actions workflow to generate high level CLI API docs
time: 2022-11-16T13:00:37.916202-06:00
custom:
Author: stu-k
Issue: "5942"

View File

@@ -1,8 +0,0 @@
kind: Under the Hood
body: Functionality-neutral refactor of event logging system to improve encapsulation
and modularity.
time: 2022-11-18T14:57:17.792622-05:00
custom:
Author: peterallenwebb
Issue: "6139"
PR: "6291"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Consolidate ParsedNode and CompiledNode classes
time: 2022-12-05T16:49:48.563583-05:00
custom:
Author: gshank
Issue: "6383"
PR: "6384"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Prevent doc gen workflow from running on forks
time: 2022-12-06T09:40:15.301984-06:00
custom:
Author: stu-k
Issue: "6386"
PR: "6390"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Fix intermittent database connection failure in Windows CI test
time: 2022-12-06T11:30:53.166009-07:00
custom:
Author: MichelleArk dbeatty10
Issue: "6394"
PR: "6395"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Refactor and clean up manifest nodes
time: 2022-12-11T21:42:40.560074-05:00
custom:
Author: gshank
Issue: "6426"
PR: "6427"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Restore important legacy logging behaviors, following refactor which removed
them
time: 2022-12-13T21:41:06.815133-05:00
custom:
Author: peterallenwebb
Issue: "6437"

View File

@@ -1,9 +0,0 @@
kind: Breaking Changes
body: Cleaned up exceptions to directly raise in code. Also updated the existing
exception to meet PEP guidelines.Removed use of all exception
functions in the code base and marked them all as deprecated to be removed next
minor release.
time: 2022-12-05T14:19:37.863032-06:00
custom:
Author: emmyoop
Issue: 6339 6393 6460

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Update agate requirement from <1.6.4,>=1.6 to >=1.6,<1.7.1 in /core"
time: 2023-01-04T00:03:06.00000Z
custom:
Author: dependabot[bot]
PR: 6506

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Updated minor typos encountered when skipping profile setup
time: 2023-01-02T17:02:16.66596191-05:00
custom:
Author: nshuman1
Issue: 6529

View File

@@ -1,7 +0,0 @@
kind: Features
body: Making timestamp optional for metrics
time: 2022-12-07T09:17:22.571877-06:00
custom:
Author: callum-mcdata
Issue: "6398"
PR: "9400"

View File

@@ -1,7 +0,0 @@
kind: Features
body: The meta configuration field is now included in the node_info property of structured
logs.
time: 2022-12-30T10:48:20.486416-06:00
custom:
Author: tmastny
Issue: "6216"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Adds buildable selection mode
time: 2023-01-02T09:13:35.663627-05:00
custom:
Author: agpapa
Issue: "6365"

View File

@@ -1,7 +0,0 @@
kind: Features
body: '--warn-error-options: Treat warnings as errors for specific events, based on
user configuration'
time: 2023-01-04T18:10:03.203142-05:00
custom:
Author: MichelleArk
Issue: "6165"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: fix missing f-strings, convert old .format() messages to f-strings for consistency
time: 2022-11-13T10:41:50.009727-05:00
custom:
Author: justbldwn
Issue: "6241"
PR: "6243"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix typo in util.py
time: 2022-11-17T22:03:20.4836855+09:00
custom:
Author: eltociear
Issue: "4904"
PR: "6037"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: add pre-commit install to make dev script in Makefile
time: 2022-12-12T11:59:12.175136-05:00
custom:
Author: justbldwn
Issue: "6269"
PR: "6417"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Late-rendering for `pre_` and `post_hook`s in `dbt_project.yml`
time: 2022-12-13T09:26:55.11397-07:00
custom:
Author: dbeatty10
Issue: "6411"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: '[CT-1591] Don''t parse empty Python files'
time: 2022-12-13T11:39:15.818464-08:00
custom:
Author: aranke
Issue: "6345"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: fix docs generate --defer by adding defer_to_manifest to before_run
time: 2022-12-26T01:02:11.630614+01:00
custom:
Author: mivanicova
Issue: "6488"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Bug when partial parsing with an empty schema file
time: 2023-01-01T22:34:05.97322-05:00
custom:
Author: gshank
Issue: "4850"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Fix DBT_FAVOR_STATE env var
time: 2023-01-04T14:10:47.637495-08:00
custom:
Author: NiallRees
Issue: "5859"
PR: "6392"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Restore historical behavior of certain disabled test messages, so that they
are at the less obtrusive debug level, rather than the warning level.
time: 2023-01-09T16:12:54.064875-05:00
custom:
Author: peterallenwebb
Issue: "6501"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Bump mashumuro version to get regression fix and add unit test to verify that
fix.
time: 2023-01-10T12:41:32.339631-05:00
custom:
Author: peterallenwebb
Issue: "6428"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Respect quoting config for dbt.ref() + dbt.source() in dbt-py models
time: 2023-01-16T12:36:45.63092+01:00
custom:
Author: jtcohen6
Issue: "6103"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Respect quoting config for dbt.this() in dbt-py models
time: 2023-01-16T12:37:09.000659+01:00
custom:
Author: jtcohen6
Issue: "6619"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Provide backward compatibility for `get_merge_sql` arguments
time: 2023-01-17T10:13:42.118336-06:00
custom:
Author: dave-connors-3
Issue: "6625"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Treat dense text blobs as binary for `git grep`
time: 2022-12-19T19:34:35.890275-07:00
custom:
Author: dbeatty10
Issue: "6294"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Prune partial parsing logging events
time: 2022-12-21T12:19:04.7402-05:00
custom:
Author: gshank
Issue: "6313"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Updating the deprecation warning in the metric attributes renamed event
time: 2023-01-04T15:52:57.916398-06:00
custom:
Author: callum-mcdata
Issue: "6507"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: '[CT-1693] Port severity test to Pytest'
time: 2023-01-06T11:28:55.800547-08:00
custom:
Author: aranke
Issue: "6466"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: '[CT-1694] Deprecate event tracking tests'
time: 2023-01-06T11:44:12.210821-08:00
custom:
Author: aranke
Issue: "6467"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Reorganize structured logging events to have two top keys
time: 2023-01-09T09:59:07.842187-05:00
custom:
Author: gshank
Issue: "6311"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Combine some logging events
time: 2023-01-10T11:42:33.580756-05:00
custom:
Author: gshank
Issue: 1716 1717 1719

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Check length of escaped strings in the adapter test
time: 2023-01-10T14:56:48.044198-07:00
custom:
Author: dbeatty10
Issue: "6566"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix use of ConnectionReused logging event
time: 2023-01-13T13:25:13.023168-05:00
custom:
Author: gshank
Issue: "6168"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Port docs tests to pytest
time: 2023-01-13T15:07:00.477038-05:00
custom:
Author: peterallenwebb
Issue: "6573"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update deprecated github action command
time: 2023-01-17T11:17:37.046095-06:00
custom:
Author: davidbloss
Issue: "6153"

View File

@@ -9,4 +9,4 @@ ignore =
E203 # makes Flake8 work like black
E741
E501 # long line checking is done in black
exclude = test
exclude = test/

20
.github/_README.md vendored
View File

@@ -63,12 +63,12 @@ permissions:
contents: read
pull-requests: write
```
### Secrets
- When to use a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) vs the [GITHUB_TOKEN](https://docs.github.com/en/actions/security-guides/automatic-token-authentication) generated for the action?
The `GITHUB_TOKEN` is used by default. In most cases it is sufficient for what you need.
If you expect the workflow to result in a commit to that should retrigger workflows, you will need to use a Personal Access Token for the bot to commit the file. When using the GITHUB_TOKEN, the resulting commit will not trigger another GitHub Actions Workflow run. This is due to limitations set by GitHub. See [the docs](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#using-the-github_token-in-a-workflow) for a more detailed explanation.
For example, we must use a PAT in our workflow to commit a new changelog yaml file for bot PRs. Once the file has been committed to the branch, it should retrigger the check to validate that a changelog exists on the PR. Otherwise, it would stay in a failed state since the check would never retrigger.
@@ -105,7 +105,7 @@ Some triggers of note that we use:
```
# **what?**
# Describe what the action does.
# Describe what the action does.
# **why?**
# Why does this action exist?
@@ -138,7 +138,7 @@ Some triggers of note that we use:
id: fp
run: |
FILEPATH=.changes/unreleased/Dependencies-${{ steps.filename_time.outputs.time }}.yaml
echo "::set-output name=FILEPATH::$FILEPATH"
echo "FILEPATH=$FILEPATH" >> $GITHUB_OUTPUT
```
- Print out all variables you will reference as the first step of a job. This allows for easier debugging. The first job should log all inputs. Subsequent jobs should reference outputs of other jobs, if present.
@@ -158,14 +158,14 @@ Some triggers of note that we use:
echo "The build_script_path: ${{ inputs.build_script_path }}"
echo "The s3_bucket_name: ${{ inputs.s3_bucket_name }}"
echo "The package_test_command: ${{ inputs.package_test_command }}"
# collect all the variables that need to be used in subsequent jobs
- name: Set Variables
id: variables
run: |
echo "::set-output name=important_path::'performance/runner/Cargo.toml'"
echo "::set-output name=release_id::${{github.event.inputs.release_id}}"
echo "::set-output name=open_prs::${{github.event.inputs.open_prs}}"
echo "important_path='performance/runner/Cargo.toml'" >> $GITHUB_OUTPUT
echo "release_id=${{github.event.inputs.release_id}}" >> $GITHUB_OUTPUT
echo "open_prs=${{github.event.inputs.open_prs}}" >> $GITHUB_OUTPUT
job2:
needs: [job1]
@@ -190,7 +190,7 @@ ___
### Actions from the Marketplace
- Dont use external actions for things that can easily be accomplished manually.
- Always read through what an external action does before using it! Often an action in the GitHub Actions Marketplace can be replaced with a few lines in bash. This is much more maintainable (and wont change under us) and clear as to whats actually happening. It also prevents any
- Pin actions _we don't control_ to tags.
- Pin actions _we don't control_ to tags.
### Connecting to AWS
- Authenticate with the aws managed workflow
@@ -208,7 +208,7 @@ ___
```yaml
- name: Copy Artifacts from S3 via CLI
run: aws s3 cp ${{ env.s3_bucket }} . --recursive
run: aws s3 cp ${{ env.s3_bucket }} . --recursive
```
### Testing

View File

@@ -28,11 +28,12 @@ if __name__ == "__main__":
if package_request.status_code == 404:
if halt_on_missing:
sys.exit(1)
else:
# everything is the latest if the package doesn't exist
print(f"::set-output name=latest::{True}")
print(f"::set-output name=minor_latest::{True}")
sys.exit(0)
# everything is the latest if the package doesn't exist
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write("latest=True")
gh_output.write("minor_latest=True")
sys.exit(0)
# TODO: verify package meta is "correct"
# https://github.com/dbt-labs/dbt-core/issues/4640
@@ -91,5 +92,7 @@ if __name__ == "__main__":
latest = is_latest(pre_rel, new_version, current_latest)
minor_latest = is_latest(pre_rel, new_version, current_minor_latest)
print(f"::set-output name=latest::{latest}")
print(f"::set-output name=minor_latest::{minor_latest}")
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"latest={latest}")
gh_output.write(f"minor_latest={minor_latest}")

View File

@@ -101,7 +101,9 @@ jobs:
- name: Get current date
if: always()
id: date
run: echo "::set-output name=date::$(date +'%Y-%m-%dT%H_%M_%S')" #no colons allowed for artifacts
run: |
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v2
if: always()
@@ -168,7 +170,9 @@ jobs:
- name: Get current date
if: always()
id: date
run: echo "::set-output name=date::$(date +'%Y_%m_%dT%H_%M_%S')" #no colons allowed for artifacts
run: |
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v2
if: always()

View File

@@ -39,7 +39,7 @@ jobs:
max-parallel: 1
fail-fast: false
matrix:
branch: [1.0.latest, 1.1.latest, 1.2.latest, 1.3.latest, main]
branch: [1.0.latest, 1.1.latest, 1.2.latest, 1.3.latest, 1.4.latest, main]
steps:
- name: Call CI workflow for ${{ matrix.branch }} branch

View File

@@ -41,9 +41,9 @@ jobs:
id: version
run: |
IFS="." read -r MAJOR MINOR PATCH <<< ${{ github.event.inputs.version_number }}
echo "::set-output name=major::$MAJOR"
echo "::set-output name=minor::$MINOR"
echo "::set-output name=patch::$PATCH"
echo "major=$MAJOR" >> $GITHUB_OUTPUT
echo "minor=$MINOR" >> $GITHUB_OUTPUT
echo "patch=$PATCH" >> $GITHUB_OUTPUT
- name: Is pkg 'latest'
id: latest
@@ -70,8 +70,10 @@ jobs:
- name: Get docker build arg
id: build_arg
run: |
echo "::set-output name=build_arg_name::"$(echo ${{ github.event.inputs.package }} | sed 's/\-/_/g')
echo "::set-output name=build_arg_value::"$(echo ${{ github.event.inputs.package }} | sed 's/postgres/core/g')
BUILD_ARG_NAME=$(echo ${{ github.event.inputs.package }} | sed 's/\-/_/g')
BUILD_ARG_VALUE=$(echo ${{ github.event.inputs.package }} | sed 's/postgres/core/g')
echo "build_arg_name=$BUILD_ARG_NAME" >> $GITHUB_OUTPUT
echo "build_arg_value=$BUILD_ARG_VALUE" >> $GITHUB_OUTPUT
- name: Log in to the GHCR
uses: docker/login-action@v1

View File

@@ -165,7 +165,7 @@ jobs:
env:
IS_PRERELEASE: ${{ contains(github.event.inputs.version_number, 'rc') || contains(github.event.inputs.version_number, 'b') }}
run: |
echo ::set-output name=isPrerelease::$IS_PRERELEASE
echo "isPrerelease=$IS_PRERELEASE" >> $GITHUB_OUTPUT
- name: Creating GitHub Release
uses: softprops/action-gh-release@v1

View File

@@ -65,7 +65,7 @@ jobs:
- name: Set branch value
id: variables
run: |
echo "::set-output name=BRANCH_NAME::prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID"
echo "BRANCH_NAME=prep-release/${{ github.event.inputs.version_number }}_$GITHUB_RUN_ID" >> $GITHUB_OUTPUT
- name: Create PR branch
run: |

View File

@@ -5,101 +5,12 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.4.0-b1 - December 15, 2022
### Features
- Added favor-state flag to optionally favor state nodes even if unselected node exists ([#2968](https://github.com/dbt-labs/dbt-core/issues/2968))
- Update structured logging. Convert to using protobuf messages. Ensure events are enriched with node_info. ([#5610](https://github.com/dbt-labs/dbt-core/issues/5610))
- Friendlier error messages when packages.yml is malformed ([#5486](https://github.com/dbt-labs/dbt-core/issues/5486))
- Allow partitions in external tables to be supplied as a list ([#5929](https://github.com/dbt-labs/dbt-core/issues/5929))
- extend -f flag shorthand for seed command ([#5990](https://github.com/dbt-labs/dbt-core/issues/5990))
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201))
- Adding tarball install method for packages. Allowing package tarball to be specified via url in the packages.yaml. ([#4205](https://github.com/dbt-labs/dbt-core/issues/4205))
- Added an md5 function to the base context ([#6246](https://github.com/dbt-labs/dbt-core/issues/6246))
- Exposures support metrics in lineage ([#6057](https://github.com/dbt-labs/dbt-core/issues/6057))
- Add support for Python 3.11 ([#6147](https://github.com/dbt-labs/dbt-core/issues/6147))
- incremental predicates ([#5680](https://github.com/dbt-labs/dbt-core/issues/5680))
### Fixes
- Account for disabled flags on models in schema files more completely ([#3992](https://github.com/dbt-labs/dbt-core/issues/3992))
- Add validation of enabled config for metrics, exposures and sources ([#6030](https://github.com/dbt-labs/dbt-core/issues/6030))
- check length of args of python model function before accessing it ([#6041](https://github.com/dbt-labs/dbt-core/issues/6041))
- Add functors to ensure event types with str-type attributes are initialized to spec, even when provided non-str type params. ([#5436](https://github.com/dbt-labs/dbt-core/issues/5436))
- Allow hooks to fail without halting execution flow ([#5625](https://github.com/dbt-labs/dbt-core/issues/5625))
- Clarify Error Message for how many models are allowed in a Python file ([#6245](https://github.com/dbt-labs/dbt-core/issues/6245))
- After this, will be possible to use default values for dbt.config.get ([#6309](https://github.com/dbt-labs/dbt-core/issues/6309))
- Use full path for writing manifest ([#6055](https://github.com/dbt-labs/dbt-core/issues/6055))
- [CT-1284] Change Python model default materialization to table ([#6345](https://github.com/dbt-labs/dbt-core/issues/6345))
- Repair a regression which prevented basic logging before the logging subsystem is completely configured. ([#6434](https://github.com/dbt-labs/dbt-core/issues/6434))
### Docs
- minor doc correction ([dbt-docs/#5791](https://github.com/dbt-labs/dbt-docs/issues/5791))
- Generate API docs for new CLI interface ([dbt-docs/#5528](https://github.com/dbt-labs/dbt-docs/issues/5528))
- ([dbt-docs/#5880](https://github.com/dbt-labs/dbt-docs/issues/5880))
- Fix rendering of sample code for metrics ([dbt-docs/#323](https://github.com/dbt-labs/dbt-docs/issues/323))
- Alphabetize `core/dbt/README.md` ([dbt-docs/#6368](https://github.com/dbt-labs/dbt-docs/issues/6368))
### Under the Hood
- Put black config in explicit config ([#5946](https://github.com/dbt-labs/dbt-core/issues/5946))
- Added flat_graph attribute the Manifest class's deepcopy() coverage ([#5809](https://github.com/dbt-labs/dbt-core/issues/5809))
- Add mypy configs so `mypy` passes from CLI ([#5983](https://github.com/dbt-labs/dbt-core/issues/5983))
- Exception message cleanup. ([#6023](https://github.com/dbt-labs/dbt-core/issues/6023))
- Add dmypy cache to gitignore ([#6028](https://github.com/dbt-labs/dbt-core/issues/6028))
- Provide useful errors when the value of 'materialized' is invalid ([#5229](https://github.com/dbt-labs/dbt-core/issues/5229))
- Clean up string formatting ([#6068](https://github.com/dbt-labs/dbt-core/issues/6068))
- Fixed extra whitespace in strings introduced by black. ([#1350](https://github.com/dbt-labs/dbt-core/issues/1350))
- Remove the 'root_path' field from most nodes ([#6171](https://github.com/dbt-labs/dbt-core/issues/6171))
- Combine certain logging events with different levels ([#6173](https://github.com/dbt-labs/dbt-core/issues/6173))
- Convert threading tests to pytest ([#5942](https://github.com/dbt-labs/dbt-core/issues/5942))
- Convert postgres index tests to pytest ([#5770](https://github.com/dbt-labs/dbt-core/issues/5770))
- Convert use color tests to pytest ([#5771](https://github.com/dbt-labs/dbt-core/issues/5771))
- Add github actions workflow to generate high level CLI API docs ([#5942](https://github.com/dbt-labs/dbt-core/issues/5942))
- Functionality-neutral refactor of event logging system to improve encapsulation and modularity. ([#6139](https://github.com/dbt-labs/dbt-core/issues/6139))
- Consolidate ParsedNode and CompiledNode classes ([#6383](https://github.com/dbt-labs/dbt-core/issues/6383))
- Prevent doc gen workflow from running on forks ([#6386](https://github.com/dbt-labs/dbt-core/issues/6386))
- Fix intermittent database connection failure in Windows CI test ([#6394](https://github.com/dbt-labs/dbt-core/issues/6394))
- Refactor and clean up manifest nodes ([#6426](https://github.com/dbt-labs/dbt-core/issues/6426))
- Restore important legacy logging behaviors, following refactor which removed them ([#6437](https://github.com/dbt-labs/dbt-core/issues/6437))
### Dependencies
- Update pathspec requirement from ~=0.9.0 to >=0.9,<0.11 in /core ([#5917](https://github.com/dbt-labs/dbt-core/pull/5917))
- Bump black from 22.8.0 to 22.10.0 ([#6019](https://github.com/dbt-labs/dbt-core/pull/6019))
- Bump mashumaro[msgpack] from 3.0.4 to 3.1.1 in /core ([#6108](https://github.com/dbt-labs/dbt-core/pull/6108))
- Update colorama requirement from <0.4.6,>=0.3.9 to >=0.3.9,<0.4.7 in /core ([#6144](https://github.com/dbt-labs/dbt-core/pull/6144))
- Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core ([#4904](https://github.com/dbt-labs/dbt-core/issues/4904))
### Contributors
- [@andy-clapson](https://github.com/andy-clapson) ([dbt-docs/#5791](https://github.com/dbt-labs/dbt-docs/issues/5791))
- [@chamini2](https://github.com/chamini2) ([#6041](https://github.com/dbt-labs/dbt-core/issues/6041))
- [@daniel-murray](https://github.com/daniel-murray) ([#2968](https://github.com/dbt-labs/dbt-core/issues/2968))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#5990](https://github.com/dbt-labs/dbt-core/issues/5990))
- [@dbeatty10](https://github.com/dbeatty10) ([dbt-docs/#6368](https://github.com/dbt-labs/dbt-docs/issues/6368), [#6394](https://github.com/dbt-labs/dbt-core/issues/6394))
- [@devmessias](https://github.com/devmessias) ([#6309](https://github.com/dbt-labs/dbt-core/issues/6309))
- [@eve-johns](https://github.com/eve-johns) ([#6068](https://github.com/dbt-labs/dbt-core/issues/6068))
- [@haritamar](https://github.com/haritamar) ([#6246](https://github.com/dbt-labs/dbt-core/issues/6246))
- [@jared-rimmer](https://github.com/jared-rimmer) ([#5486](https://github.com/dbt-labs/dbt-core/issues/5486))
- [@josephberni](https://github.com/josephberni) ([#2968](https://github.com/dbt-labs/dbt-core/issues/2968))
- [@joshuataylor](https://github.com/joshuataylor) ([#6147](https://github.com/dbt-labs/dbt-core/issues/6147))
- [@justbldwn](https://github.com/justbldwn) ([#6245](https://github.com/dbt-labs/dbt-core/issues/6245))
- [@luke-bassett](https://github.com/luke-bassett) ([#1350](https://github.com/dbt-labs/dbt-core/issues/1350))
- [@max-sixty](https://github.com/max-sixty) ([#5946](https://github.com/dbt-labs/dbt-core/issues/5946), [#5983](https://github.com/dbt-labs/dbt-core/issues/5983), [#6028](https://github.com/dbt-labs/dbt-core/issues/6028))
- [@paulbenschmidt](https://github.com/paulbenschmidt) ([dbt-docs/#5880](https://github.com/dbt-labs/dbt-docs/issues/5880))
- [@pgoslatara](https://github.com/pgoslatara) ([#5929](https://github.com/dbt-labs/dbt-core/issues/5929))
- [@racheldaniel](https://github.com/racheldaniel) ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201))
- [@timle2](https://github.com/timle2) ([#4205](https://github.com/dbt-labs/dbt-core/issues/4205))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#5680](https://github.com/dbt-labs/dbt-core/issues/5680))
## Previous Releases
For information on prior major and minor releases, see their changelogs:
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)

View File

@@ -142,44 +142,44 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
)
def set_connection_name(self, name: Optional[str] = None) -> Connection:
conn_name: str
if name is None:
# if a name isn't specified, we'll re-use a single handle
# named 'master'
conn_name = "master"
else:
if not isinstance(name, str):
raise dbt.exceptions.CompilerException(
f"For connection name, got {name} - not a string!"
)
assert isinstance(name, str)
conn_name = name
"""Called by 'acquire_connection' in BaseAdapter, which is called by
'connection_named', called by 'connection_for(node)'.
Creates a connection for this thread if one doesn't already
exist, and will rename an existing connection."""
conn_name: str = "master" if name is None else name
# Get a connection for this thread
conn = self.get_if_exists()
if conn and conn.name == conn_name and conn.state == "open":
# Found a connection and nothing to do, so just return it
return conn
if conn is None:
# Create a new connection
conn = Connection(
type=Identifier(self.TYPE),
name=None,
name=conn_name,
state=ConnectionState.INIT,
transaction_open=False,
handle=None,
credentials=self.profile.credentials,
)
self.set_thread_connection(conn)
if conn.name == conn_name and conn.state == "open":
return conn
fire_event(
NewConnection(conn_name=conn_name, conn_type=self.TYPE, node_info=get_node_info())
)
if conn.state == "open":
fire_event(ConnectionReused(conn_name=conn_name))
else:
conn.handle = LazyHandle(self.open)
# Add the connection to thread_connections for this thread
self.set_thread_connection(conn)
fire_event(
NewConnection(conn_name=conn_name, conn_type=self.TYPE, node_info=get_node_info())
)
else: # existing connection either wasn't open or didn't have the right name
if conn.state != "open":
conn.handle = LazyHandle(self.open)
if conn.name != conn_name:
orig_conn_name: str = conn.name or ""
conn.name = conn_name
fire_event(ConnectionReused(orig_conn_name=orig_conn_name, conn_name=conn_name))
conn.name = conn_name
return conn
@classmethod

View File

@@ -269,6 +269,22 @@ class BaseAdapter(metaclass=AdapterMeta):
"""
return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch)
@available.parse(lambda *a, **k: [])
def get_column_schema_from_query(
self, sql: str, auto_begin: bool = False, fetch: bool = False
) -> Tuple[AdapterResponse, agate.Table]:
"""Execute the given SQL. This is a thin wrapper around
ConnectionManager.execute.
:param str sql: The sql to execute.
:param bool auto_begin: If set, and dbt is not currently inside a
transaction, automatically begin one.
:param bool fetch: If set, fetch results.
:return: A tuple of the query status and results (empty if fetch=False).
:rtype: List[(column_name: str, data_type: str]
"""
return self.connections.get_column_schema_from_query(sql=sql)
@available.parse(lambda *a, **k: ("", empty_table()))
def get_partitions_metadata(self, table: str) -> Tuple[agate.Table]:
"""Obtain partitions metadata for a BigQuery partitioned table.

View File

@@ -128,6 +128,31 @@ class SQLConnectionManager(BaseConnectionManager):
return dbt.clients.agate_helper.table_from_data_flat(data, column_names)
@classmethod
def data_type_code_to_name(cls, int) -> str:
"""Get the string representation of the data type from the type_code."""
# https://peps.python.org/pep-0249/#type-objects
raise dbt.exceptions.NotImplementedError(
"`data_type_code_to_name` is not implemented for this adapter!"
)
@classmethod
def get_column_schema_from_cursor(cls, cursor: Any) -> List[Tuple[str, str]]:
# (column_name, data_type)
columns: List[Tuple[str, str]] = []
if cursor.description is not None:
# https://peps.python.org/pep-0249/#description
columns = [
# TODO: ignoring size, precision, scale for now
# (though it is part of DB-API standard, and our Column class does have these attributes)
# IMO user-defined contracts shouldn't have to match an exact size/precision/scale
(col[0], cls.data_type_code_to_name(col[1]))
for col in cursor.description
]
return columns
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False
) -> Tuple[AdapterResponse, agate.Table]:
@@ -140,6 +165,20 @@ class SQLConnectionManager(BaseConnectionManager):
table = dbt.clients.agate_helper.empty_table()
return response, table
# TODO: do we need to care about auto_begin here?
def get_column_schema_from_query(self, sql: str) -> List[Tuple[str, str]]:
sql = self._add_query_comment(sql)
_, cursor = self.add_query(sql)
return self.get_column_schema_from_cursor(cursor)
# For dbt-bigquery
# def get_column_schema_from_query(cls, sql: str) -> List[Tuple[str, str]]:
# sql = self._add_query_comment(sql)
# # auto_begin is ignored on bigquery, and only included for consistency
# query_job, iterator = self.raw_execute(sql)
# columns = [(field.name, field.field_type) for field in resp.iterator]
# return columns
def add_begin_query(self):
return self.add_query("BEGIN", auto_begin=False)

View File

@@ -351,13 +351,6 @@ class Compiler:
)
if node.language == ModelLanguage.python:
# TODO could we also 'minify' this code at all? just aesthetic, not functional
# quoating seems like something very specific to sql so far
# for all python implementations we are seeing there's no quating.
# TODO try to find better way to do this, given that
original_quoting = self.config.quoting
self.config.quoting = {key: False for key in original_quoting.keys()}
context = self._create_node_context(node, manifest, extra_context)
postfix = jinja.get_rendered(
@@ -367,8 +360,6 @@ class Compiler:
)
# we should NOT jinja render the python model's 'raw code'
node.compiled_code = f"{node.raw_code}\n\n{postfix}"
# restore quoting settings in the end since context is lazy evaluated
self.config.quoting = original_quoting
else:
context = self._create_node_context(node, manifest, extra_context)

View File

@@ -9,7 +9,7 @@ from dbt.config import Profile, Project, read_user_config
from dbt.config.renderer import DbtProjectYamlRenderer, ProfileRenderer
from dbt.events.functions import fire_event
from dbt.events.types import InvalidOptionYAML
from dbt.exceptions import DbtValidationError, OptionNotYamlDict
from dbt.exceptions import DbtValidationError, OptionNotYamlDictError
def parse_cli_vars(var_string: str) -> Dict[str, Any]:
@@ -23,7 +23,7 @@ def parse_cli_yaml_string(var_string: str, cli_option_name: str) -> Dict[str, An
if var_type is dict:
return cli_vars
else:
raise OptionNotYamlDict(var_type, cli_option_name)
raise OptionNotYamlDictError(var_type, cli_option_name)
except DbtValidationError:
fire_event(InvalidOptionYAML(option_name=cli_option_name))
raise

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More