Compare commits

..

1 Commits

Author SHA1 Message Date
Callum McCann
2da925aa25 Adding entity node to core (#6648)
* first draft

* finishing first commit

* adding testing project

* adding changie

* cleaning

* removing blocks

* fixing proto error message

* updates to events

* fixing issues

* adding test dimension

* updating schemas

* updating manfiest.json

* removing old versions from compatability

* updating

* fixes

* fixing more bugs caught by tests

* updating tests
2023-01-31 09:03:06 -06:00
1395 changed files with 63955 additions and 216135 deletions

View File

@@ -1,19 +1,13 @@
[bumpversion]
current_version = 1.11.0b3
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
(?P<prerelease> # optional pre-release - ex: a1, b2, rc25
(?P<prekind>a|b|rc) # pre-release type
(?P<num>[\d]+) # pre-release version number
current_version = 1.5.0a1
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
((?P<prekind>a|b|rc)
(?P<pre>\d+) # pre-release version num
)?
( # optional nightly release indicator
\.(?P<nightly>dev[0-9]+) # ex: .dev02142023
)? # expected matches: `1.15.0`, `1.5.0a11`, `1.5.0a1.dev123`, `1.5.0.dev123457`, expected failures: `1`, `1.5`, `1.5.2-a1`, `text1.5.0`
serialize =
{major}.{minor}.{patch}{prekind}{num}.{nightly}
{major}.{minor}.{patch}.{nightly}
{major}.{minor}.{patch}{prekind}{num}
{major}.{minor}.{patch}{prekind}{pre}
{major}.{minor}.{patch}
commit = False
tag = False
@@ -27,11 +21,19 @@ values =
rc
final
[bumpversion:part:num]
[bumpversion:part:pre]
first_value = 1
[bumpversion:part:nightly]
[bumpversion:file:core/setup.py]
[bumpversion:file:core/dbt/version.py]
[bumpversion:file:plugins/postgres/setup.py]
[bumpversion:file:plugins/postgres/dbt/adapters/postgres/__version__.py]
[bumpversion:file:docker/Dockerfile]
[bumpversion:file:tests/adapter/setup.py]
[bumpversion:file:tests/adapter/dbt/tests/adapter/__version__.py]

View File

@@ -3,12 +3,6 @@
For information on prior major and minor releases, see their changelogs:
* [1.10](https://github.com/dbt-labs/dbt-core/blob/1.10.latest/CHANGELOG.md)
* [1.9](https://github.com/dbt-labs/dbt-core/blob/1.9.latest/CHANGELOG.md)
* [1.8](https://github.com/dbt-labs/dbt-core/blob/1.8.latest/CHANGELOG.md)
* [1.7](https://github.com/dbt-labs/dbt-core/blob/1.7.latest/CHANGELOG.md)
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)

View File

@@ -1,89 +0,0 @@
## dbt-core 1.11.0-b1 - October 01, 2025
### Features
- Add file_format to catalog integration config ([#11695](https://github.com/dbt-labs/dbt-core/issues/11695))
- 11561 ([#deprecate](https://github.com/dbt-labs/dbt-core/issues/deprecate), [#--models,--model,](https://github.com/dbt-labs/dbt-core/issues/--models,--model,), [#and](https://github.com/dbt-labs/dbt-core/issues/and), [#-m](https://github.com/dbt-labs/dbt-core/issues/-m), [#flags](https://github.com/dbt-labs/dbt-core/issues/flags))
- Update jsonschemas with builtin data test properties and exposure configs in dbt_project.yml for more accurate deprecations ([#11335](https://github.com/dbt-labs/dbt-core/issues/11335))
- Support loaded_at_query and loaded_at_field on source and table configs ([#11659](https://github.com/dbt-labs/dbt-core/issues/11659))
- Begin validating configs from model sql files ([#11727](https://github.com/dbt-labs/dbt-core/issues/11727))
- Deprecate `overrides` property for sources ([#11566](https://github.com/dbt-labs/dbt-core/issues/11566))
- Create constrained namespace for dbt engine env vars ([#11340](https://github.com/dbt-labs/dbt-core/issues/11340))
- Gate jsonschema validations by adapter ([#11680](https://github.com/dbt-labs/dbt-core/issues/11680))
- Deprecate top-level argument properties in generic tests ([#11847](https://github.com/dbt-labs/dbt-core/issues/11847))
- Deprecate {{ modules.itertools }} usage ([#11725](https://github.com/dbt-labs/dbt-core/issues/11725))
- Default require_generic_test_arguments_property flag to True - The 'arguments' property will be parsed as keyword arguments to data tests, if provided ([#11911](https://github.com/dbt-labs/dbt-core/issues/11911))
- Support Nested Key Traversal in dbt ls json output ([#11919](https://github.com/dbt-labs/dbt-core/issues/11919))
- No-op when project-level `quoting.snowflake_ignore_case` is set. ([#11882](https://github.com/dbt-labs/dbt-core/issues/11882))
- Support UDFs by allowing user definition of `function` nodes ([#11923](https://github.com/dbt-labs/dbt-core/issues/11923))
- Support listing functions via `list` command ([#11967](https://github.com/dbt-labs/dbt-core/issues/11967))
- Support selecting funciton nodes via: name, file path, and resource type ([#11962](https://github.com/dbt-labs/dbt-core/issues/11962), [#11958](https://github.com/dbt-labs/dbt-core/issues/11958), [#11961](https://github.com/dbt-labs/dbt-core/issues/11961))
- Parse catalogs.yml during parse, seed, and test commands ([#12002](https://github.com/dbt-labs/dbt-core/issues/12002))
- Handle creation of function nodes during DAG execution ([#11965](https://github.com/dbt-labs/dbt-core/issues/11965))
- Support configuring model.config.freshness.build_after.updates_on without period or count ([#12019](https://github.com/dbt-labs/dbt-core/issues/12019))
- Add `function` macro to jinja context ([#11972](https://github.com/dbt-labs/dbt-core/issues/11972))
- Adding run_started_at to manifest.json metadata ([#12047](https://github.com/dbt-labs/dbt-core/issues/12047))
- Validate {{ config }} in SQL for models that don't statically parse ([#12046](https://github.com/dbt-labs/dbt-core/issues/12046))
- Add `type` property to `function` nodes ([#12042](https://github.com/dbt-labs/dbt-core/issues/12042), [#12037](https://github.com/dbt-labs/dbt-core/issues/12037))
### Fixes
- Don't warn for metricflow_time_spine with non-day grain ([#11690](https://github.com/dbt-labs/dbt-core/issues/11690))
- Fix source freshness set via config to handle explicit nulls ([#11685](https://github.com/dbt-labs/dbt-core/issues/11685))
- Ensure build_after is present in model freshness in parsing, otherwise skip freshness definition ([#11709](https://github.com/dbt-labs/dbt-core/issues/11709))
- Ensure source node `.freshness` is equal to node's `.config.freshness` ([#11717](https://github.com/dbt-labs/dbt-core/issues/11717))
- ignore invalid model freshness configs in inline model configs ([#11728](https://github.com/dbt-labs/dbt-core/issues/11728))
- Fix store_failures hierarachical config parsing ([#10165](https://github.com/dbt-labs/dbt-core/issues/10165))
- Remove model freshness property support in favor of config level support ([#11713](https://github.com/dbt-labs/dbt-core/issues/11713))
- Bump dbt-common to 1.25.0 to access WarnErrorOptionsV2 ([#11755](https://github.com/dbt-labs/dbt-core/issues/11755))
- ensure consistent casing in column names while processing user unit tests ([#11770](https://github.com/dbt-labs/dbt-core/issues/11770))
- Update jsonschema definitions with nested config defs, cloud info, and dropping source overrides ([#N/A](https://github.com/dbt-labs/dbt-core/issues/N/A))
- Make `GenericJSONSchemaValidationDeprecation` a "preview" deprecation ([#11814](https://github.com/dbt-labs/dbt-core/issues/11814))
- Correct JSONSchema Semantic Layer node issues ([#11818](https://github.com/dbt-labs/dbt-core/issues/11818))
- Improve SL JSONSchema definitions ([#N/A](https://github.com/dbt-labs/dbt-core/issues/N/A))
- raise MissingPlusPrefixDeprecation instead of GenericJSONSchemaValidationDeprecation when config missing plus prefix in dbt_project.yml ([#11826](https://github.com/dbt-labs/dbt-core/issues/11826))
- Propagate config.meta and config.tags to top-level on source nodes ([#11839](https://github.com/dbt-labs/dbt-core/issues/11839))
- Safe handling of malformed config.tags on sources/tables ([#11855](https://github.com/dbt-labs/dbt-core/issues/11855))
- Quoting the event_time field when the configuration says so ([#11858](https://github.com/dbt-labs/dbt-core/issues/11858))
- Raise PropertyMovedToConfigDeprecation instead of CustomTopLevelKeyDeprecation when additional attribute is a valid node config ([#11879](https://github.com/dbt-labs/dbt-core/issues/11879))
- Avoid redundant node patch removal during partial parsing ([#11886](https://github.com/dbt-labs/dbt-core/issues/11886))
- Comply with strict `str` type when `block.contents` is `None` ([#11492](https://github.com/dbt-labs/dbt-core/issues/11492))
- Remove duplicative PropertyMovedToConfigDeprecation for source freshness ([#11880](https://github.com/dbt-labs/dbt-core/issues/11880))
- Add path to MissingArgumentsPropertyInGenericTestDeprecation message ([#11940](https://github.com/dbt-labs/dbt-core/issues/11940))
- Unhide sample mode CLI flag ([#11959](https://github.com/dbt-labs/dbt-core/issues/11959))
- Implement checked_agg_time_dimension_for_simple_metric to satisfy dbt-semantic-interfaces>0.9.0 ([#11998](https://github.com/dbt-labs/dbt-core/issues/11998))
- Propagate column meta/tags from config to tests ([#11984](https://github.com/dbt-labs/dbt-core/issues/11984))
- Skip initial render of loaded_at_query when specified as source or table config ([#11973](https://github.com/dbt-labs/dbt-core/issues/11973))
- Guarantee instantiation result and thread_exception prior to access to avoid thread hangs ([#12013](https://github.com/dbt-labs/dbt-core/issues/12013))
- Fixes a bug in the logic for legacy time spine deprecation warnings. ([#11690](https://github.com/dbt-labs/dbt-core/issues/11690))
- Fix errors in partial parsing when working with versioned models ([#11869](https://github.com/dbt-labs/dbt-core/issues/11869))
### Under the Hood
- Prevent overcounting PropertyMovedToConfigDeprecation for source freshness ([#11660](https://github.com/dbt-labs/dbt-core/issues/11660))
- call adapter.add_catalog_integration during parse_manifest ([#11889](https://github.com/dbt-labs/dbt-core/issues/11889))
- Fix docker os dependency install issue ([#11934](https://github.com/dbt-labs/dbt-core/issues/11934))
- Ensure dbt-core modules aren't importing versioned artifact resources directly ([#11951](https://github.com/dbt-labs/dbt-core/issues/11951))
- Update jsonschemas used for schema-based deprecations ([#11987](https://github.com/dbt-labs/dbt-core/issues/11987))
- Introduce dbt_version-based manifest json upgrade framework to avoid state:modified false positives on minor evolutions ([#12006](https://github.com/dbt-labs/dbt-core/issues/12006))
### Dependencies
- Bump minimum jsonschema version to `4.19.1` ([#11740](https://github.com/dbt-labs/dbt-core/issues/11740))
- Allow for either pydantic v1 and v2 ([#11634](https://github.com/dbt-labs/dbt-core/issues/11634))
- Bump dbt-common minimum to 1.25.1 ([#11789](https://github.com/dbt-labs/dbt-core/issues/11789))
- Upgrade to dbt-semantic-interfaces==0.9.0 for more robust saved query support. ([#11809](https://github.com/dbt-labs/dbt-core/issues/11809))
- upgrade protobuf to 6.0 ([#11916](https://github.com/dbt-labs/dbt-core/issues/11916))
- Bump dbt-adapters minimum to 1.16.5 ([#11932](https://github.com/dbt-labs/dbt-core/issues/11932))
- Bump actions/setup-python from 5 to 6 ([#11993](https://github.com/dbt-labs/dbt-core/issues/11993))
- Loosen dbt-semantic-interfaces lower pin to >=0.9.0 ([#12005](https://github.com/dbt-labs/dbt-core/issues/12005))
### Contributors
- [@aksestok](https://github.com/aksestok) ([#11882](https://github.com/dbt-labs/dbt-core/issues/11882))
- [@aranke](https://github.com/aranke) ([#11660](https://github.com/dbt-labs/dbt-core/issues/11660))
- [@courtneyholcomb](https://github.com/courtneyholcomb) ([#11690](https://github.com/dbt-labs/dbt-core/issues/11690), [#11690](https://github.com/dbt-labs/dbt-core/issues/11690), [#11809](https://github.com/dbt-labs/dbt-core/issues/11809))
- [@michellark](https://github.com/michellark) ([#11987](https://github.com/dbt-labs/dbt-core/issues/11987))
- [@pablomc87](https://github.com/pablomc87) ([#11858](https://github.com/dbt-labs/dbt-core/issues/11858))
- [@peterallenwebb](https://github.com/peterallenwebb) ([#11566](https://github.com/dbt-labs/dbt-core/issues/11566))
- [@trouze](https://github.com/trouze) ([#11919](https://github.com/dbt-labs/dbt-core/issues/11919))
- [@wircho](https://github.com/wircho) ([#11886](https://github.com/dbt-labs/dbt-core/issues/11886), [#11492](https://github.com/dbt-labs/dbt-core/issues/11492))

View File

@@ -1,6 +0,0 @@
## dbt-core 1.11.0-b2 - October 02, 2025
### Fixes
- Fix property names of function nodes (arguments.data_type, returns, returns.data_type) ([#12064](https://github.com/dbt-labs/dbt-core/issues/12064))
- Exclude `functions` from being filtered by `empty` and `event_time` ([#12066](https://github.com/dbt-labs/dbt-core/issues/12066))

View File

@@ -1,10 +0,0 @@
## dbt-core 1.11.0-b3 - October 07, 2025
### Features
- Support function nodes for unit tested models ([#12024](https://github.com/dbt-labs/dbt-core/issues/12024))
### Fixes
- Fix case of successful function status in logs ([#12075](https://github.com/dbt-labs/dbt-core/issues/12075))
- Fix `ref` support in function nodes ([#12076](https://github.com/dbt-labs/dbt-core/issues/12076))

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Bump minimum jsonschema version to `4.19.1`
time: 2025-06-16T14:44:08.512306-05:00
custom:
Author: QMalcolm
Issue: "11740"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Allow for either pydantic v1 and v2
time: 2025-06-20T12:36:00.196384-05:00
custom:
Author: QMalcolm
Issue: "11634"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Bump dbt-common minimum to 1.25.1
time: 2025-07-02T14:47:20.772002-05:00
custom:
Author: QMalcolm
Issue: "11789"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Upgrade to dbt-semantic-interfaces==0.9.0 for more robust saved query support.
time: 2025-07-09T13:22:13.688162-07:00
custom:
Author: courtneyholcomb
Issue: "11809"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: upgrade protobuf to 6.0
time: 2025-08-13T14:48:32.380929-07:00
custom:
Author: colin-rogers-dbt
Issue: "11916"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Bump dbt-adapters minimum to 1.16.5
time: 2025-08-19T08:57:14.396843-05:00
custom:
Author: QMalcolm
Issue: "11932"

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Bump actions/setup-python from 5 to 6"
time: 2025-09-08T00:06:55.00000Z
custom:
Author: dependabot[bot]
Issue: 11993

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Loosen dbt-semantic-interfaces lower pin to >=0.9.0
time: 2025-09-11T09:57:31.09927-04:00
custom:
Author: michelleark
Issue: "12005"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add file_format to catalog integration config
time: 2025-05-29T08:53:11.64904-07:00
custom:
Author: colin-rogers-dbt
Issue: "11695"

View File

@@ -1,6 +0,0 @@
kind: Features
body: "11561"
time: 2025-06-11T16:02:17.334525-04:00
custom:
Author: michelleark
Issue: deprecate --models,--model, and -m flags

View File

@@ -1,7 +0,0 @@
kind: Features
body: Update jsonschemas with builtin data test properties and exposure configs in
dbt_project.yml for more accurate deprecations
time: 2025-06-17T14:25:16.976867-04:00
custom:
Author: michelleark
Issue: "11335"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support loaded_at_query and loaded_at_field on source and table configs
time: 2025-06-23T11:31:30.897805-04:00
custom:
Author: michelleark
Issue: "11659"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Begin validating configs from model sql files
time: 2025-06-25T15:18:18.164-05:00
custom:
Author: QMalcolm
Issue: "11727"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Deprecate `overrides` property for sources
time: 2025-07-01T16:49:57.979871-05:00
custom:
Author: peterallenwebb
Issue: "11566"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Create constrained namespace for dbt engine env vars
time: 2025-07-03T17:53:41.406701-05:00
custom:
Author: QMalcolm
Issue: "11340"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Gate jsonschema validations by adapter
time: 2025-07-14T23:25:24.475471-05:00
custom:
Author: QMalcolm
Issue: "11680"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Deprecate top-level argument properties in generic tests
time: 2025-07-21T17:31:00.960402-04:00
custom:
Author: michelleark
Issue: "11847"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Deprecate {{ modules.itertools }} usage
time: 2025-07-28T11:54:43.28275-04:00
custom:
Author: michelleark
Issue: "11725"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Default require_generic_test_arguments_property flag to True - The 'arguments' property will be parsed as keyword arguments to data tests, if provided
time: 2025-08-11T15:03:47.369377+02:00
custom:
Author: michelleark
Issue: "11911"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support Nested Key Traversal in dbt ls json output
time: 2025-08-15T13:14:58.162067-05:00
custom:
Author: trouze
Issue: "11919"

View File

@@ -1,6 +0,0 @@
kind: Features
body: No-op when project-level `quoting.snowflake_ignore_case` is set.
time: 2025-08-21T09:25:32.049328307+02:00
custom:
Author: aksestok
Issue: "11882"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support UDFs by allowing user definition of `function` nodes
time: 2025-08-26T13:27:00.607953-05:00
custom:
Author: QMalcolm
Issue: "11923"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support listing functions via `list` command
time: 2025-08-27T16:34:00.216146-05:00
custom:
Author: QMalcolm
Issue: "11967"

View File

@@ -1,6 +0,0 @@
kind: Features
body: 'Support selecting funciton nodes via: name, file path, and resource type'
time: 2025-08-27T17:31:03.518839-05:00
custom:
Author: QMalcolm
Issue: 11962 11958 11961

View File

@@ -1,6 +0,0 @@
kind: Features
body: Parse catalogs.yml during parse, seed, and test commands
time: 2025-09-10T14:45:54.065327-04:00
custom:
Author: michelleark
Issue: "12002"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Handle creation of function nodes during DAG execution
time: 2025-09-11T11:38:47.780819-07:00
custom:
Author: QMalcolm
Issue: "11965"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support configuring model.config.freshness.build_after.updates_on without period or count
time: 2025-09-17T16:49:41.528385-04:00
custom:
Author: michelleark
Issue: "12019"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add `function` macro to jinja context
time: 2025-09-18T16:58:21.799588-05:00
custom:
Author: QMalcolm
Issue: "11972"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Adding run_started_at to manifest.json metadata
time: 2025-09-24T13:09:23.50087-04:00
custom:
Author: michelleark
Issue: "12047"

View File

@@ -1,6 +0,0 @@
kind: Features
body: 'Validate {{ config }} in SQL for models that don''t statically parse '
time: 2025-09-25T17:16:52.76822-04:00
custom:
Author: michelleark
Issue: "12046"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add `type` property to `function` nodes
time: 2025-09-26T14:03:15.728495-05:00
custom:
Author: QMalcolm
Issue: 12042 12037

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support function nodes for unit tested models
time: 2025-10-02T18:29:11.316884-05:00
custom:
Author: QMalcolm
Issue: "12024"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Don't warn for metricflow_time_spine with non-day grain
time: 2025-05-28T09:20:55.866514-07:00
custom:
Author: courtneyholcomb
Issue: "11690"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix source freshness set via config to handle explicit nulls
time: 2025-05-30T00:58:04.94133-05:00
custom:
Author: QMalcolm
Issue: "11685"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Ensure build_after is present in model freshness in parsing, otherwise skip
freshness definition
time: 2025-06-05T11:06:45.329942-07:00
custom:
Author: QMalcolm
Issue: "11709"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Ensure source node `.freshness` is equal to node's `.config.freshness`
time: 2025-06-09T17:52:39.978403-05:00
custom:
Author: QMalcolm
Issue: "11717"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ignore invalid model freshness configs in inline model configs
time: 2025-06-10T21:12:41.972614-07:00
custom:
Author: colin-rogers-dbt
Issue: "11728"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix store_failures hierarachical config parsing
time: 2025-06-12T14:51:59.358498-04:00
custom:
Author: michelleark
Issue: "10165"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: "Remove model freshness property support in favor of config level support"
time: 2025-06-16T08:56:00.641553-05:00
custom:
Author: QMalcolm
Issue: "11713"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Bump dbt-common to 1.25.0 to access WarnErrorOptionsV2
time: 2025-06-24T09:12:58.904713-04:00
custom:
Author: michelleark
Issue: "11755"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ensure consistent casing in column names while processing user unit tests
time: 2025-06-24T15:20:47.589212-07:00
custom:
Author: colin-rogers-dbt
Issue: "11770"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Update jsonschema definitions with nested config defs, cloud info, and dropping
source overrides
time: 2025-07-07T10:34:18.982725-05:00
custom:
Author: QMalcolm
Issue: N/A

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Make `GenericJSONSchemaValidationDeprecation` a "preview" deprecation
time: 2025-07-10T17:01:48.903582-05:00
custom:
Author: QMalcolm
Issue: "11814"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Correct JSONSchema Semantic Layer node issues
time: 2025-07-11T09:54:39.86192-05:00
custom:
Author: MichelleArk
Issue: "11818"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Improve SL JSONSchema definitions
time: 2025-07-14T12:50:00.3541-05:00
custom:
Author: QMalcolm
Issue: N/A

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: raise MissingPlusPrefixDeprecation instead of GenericJSONSchemaValidationDeprecation when config missing plus prefix in dbt_project.yml
time: 2025-07-14T18:41:31.322137-04:00
custom:
Author: michelleark
Issue: "11826"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Propagate config.meta and config.tags to top-level on source nodes
time: 2025-07-16T16:45:35.683199-04:00
custom:
Author: michelleark
Issue: "11839"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Safe handling of malformed config.tags on sources/tables
time: 2025-07-24T10:44:10.828775-04:00
custom:
Author: michelleark
Issue: "11855"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Quoting the event_time field when the configuration says so
time: 2025-07-25T11:29:03.342884+02:00
custom:
Author: pablomc87
Issue: "11858"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Raise PropertyMovedToConfigDeprecation instead of CustomTopLevelKeyDeprecation when additional attribute is a valid node config
time: 2025-07-31T16:21:42.938703-04:00
custom:
Author: michelleark
Issue: "11879"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Avoid redundant node patch removal during partial parsing
time: 2025-08-04T07:40:06.993913-06:00
custom:
Author: wircho
Issue: "11886"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Comply with strict `str` type when `block.contents` is `None`
time: 2025-08-04T07:42:54.612616-06:00
custom:
Author: wircho
Issue: "11492"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Remove duplicative PropertyMovedToConfigDeprecation for source freshness
time: 2025-08-05T10:43:09.502585-04:00
custom:
Author: michelleark
Issue: "11880"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Add path to MissingArgumentsPropertyInGenericTestDeprecation message
time: 2025-08-20T19:53:39.282829+02:00
custom:
Author: michelleark
Issue: "11940"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Unhide sample mode CLI flag
time: 2025-08-26T14:48:59.685423-05:00
custom:
Author: QMalcolm
Issue: "11959"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Implement checked_agg_time_dimension_for_simple_metric to satisfy dbt-semantic-interfaces>0.9.0
time: 2025-09-09T17:11:32.936696-04:00
custom:
Author: michelleark
Issue: "11998"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Propagate column meta/tags from config to tests
time: 2025-09-10T16:39:05.129379-04:00
custom:
Author: michelleark
Issue: "11984"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Skip initial render of loaded_at_query when specified as source or table config
time: 2025-09-11T17:16:48.843045-04:00
custom:
Author: michelleark
Issue: '11973'

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Guarantee instantiation result and thread_exception prior to access to avoid thread hangs
time: 2025-09-12T14:51:13.355232-04:00
custom:
Author: michelleark
Issue: "12013"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fixes a bug in the logic for legacy time spine deprecation warnings.
time: 2025-09-15T15:22:27.791819-07:00
custom:
Author: courtneyholcomb
Issue: "11690"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix errors in partial parsing when working with versioned models
time: 2025-09-25T21:27:28.591187-04:00
custom:
Author: gshank
Issue: "11869"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix property names of function nodes (arguments.data_type, returns, returns.data_type)
time: 2025-10-02T12:14:27.305546-05:00
custom:
Author: QMalcolm
Issue: "12064"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Exclude `functions` from being filtered by `empty` and `event_time`
time: 2025-10-02T16:06:55.689911-05:00
custom:
Author: QMalcolm
Issue: "12066"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix case of successful function status in logs
time: 2025-10-06T17:35:03.327207-05:00
custom:
Author: QMalcolm
Issue: "12075"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix `ref` support in function nodes
time: 2025-10-06T17:58:13.853195-05:00
custom:
Author: QMalcolm
Issue: "12076"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Prevent overcounting PropertyMovedToConfigDeprecation for source freshness
time: 2025-05-27T16:21:36.551426+01:00
custom:
Author: aranke
Issue: "11660"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: call adapter.add_catalog_integration during parse_manifest
time: 2025-08-04T21:13:46.077488-07:00
custom:
Author: colin-rogers-dbt
Issue: "11889"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Fix docker os dependency install issue
time: 2025-08-19T12:34:48.402158-05:00
custom:
Author: QMalcolm
Issue: "11934"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Ensure dbt-core modules aren't importing versioned artifact resources directly
time: 2025-08-22T16:27:52.403177-05:00
custom:
Author: QMalcolm
Issue: "11951"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Update jsonschemas used for schema-based deprecations
time: 2025-09-03T16:18:33.931722-04:00
custom:
Author: michellark
Issue: "11987"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Introduce dbt_version-based manifest json upgrade framework to avoid state:modified false positives on minor evolutions
time: 2025-09-11T11:01:54.180698-04:00
custom:
Author: michelleark
Issue: "12006"

View File

@@ -1,6 +1,6 @@
# dbt Core Changelog
- This file provides a full account of all changes to `dbt-core`
- This file provides a full account of all changes to `dbt-core` and `dbt-postgres`
- Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)

View File

@@ -0,0 +1,6 @@
kind: Features
body: Adding the entity node
time: 2023-01-18T13:48:04.487817-06:00
custom:
Author: callum-mcdata
Issue: "6627"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Allow for the specification of function volatility
time: 2025-10-21T13:57:35.169466-05:00
custom:
Author: "12030"
Issue: QMalcolm

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Move function node `type` into it's config
time: 2025-10-21T14:30:01.07439-05:00
custom:
Author: QMalcolm
Issue: "12101"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Support setting function node configs from dbt_project.yml
time: 2025-10-22T13:25:07.824387-05:00
custom:
Author: QMalcolm
Issue: "12096"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix use of ConnectionReused logging event
time: 2023-01-13T13:25:13.023168-05:00
custom:
Author: gshank
Issue: "6168"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update deprecated github action command
time: 2023-01-17T11:17:37.046095-06:00
custom:
Author: davidbloss
Issue: "6153"

View File

@@ -4,7 +4,6 @@ headerPath: header.tpl.md
versionHeaderPath: ""
changelogPath: CHANGELOG.md
versionExt: md
envPrefix: "CHANGIE_"
versionFormat: '## dbt-core {{.Version}} - {{.Time.Format "January 02, 2006"}}'
kindFormat: '### {{.Kind}}'
changeFormat: |-
@@ -31,7 +30,43 @@ kinds:
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
@@ -52,36 +87,32 @@ custom:
footerFormat: |
{{- $contributorDict := dict }}
{{- /* ensure all names in this list are all lowercase for later matching purposes */}}
{{- $core_team := splitList " " .Env.CORE_TEAM }}
{{- /* ensure we always skip snyk and dependabot in addition to the core team */}}
{{- $maintainers := list "dependabot[bot]" "snyk-bot"}}
{{- range $team_member := $core_team }}
{{- $team_member_lower := lower $team_member }}
{{- $maintainers = append $maintainers $team_member_lower }}
{{- end }}
{{- /* any names added to this list should be all lowercase for later matching purposes */}}
{{- $core_team := list "michelleark" "peterallenwebb" "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "aranke" "dependabot[bot]" "snyk-bot" "colin-rogers-dbt" }}
{{- range $change := .Changes }}
{{- $authorList := splitList " " $change.Custom.Author }}
{{- /* loop through all authors for a single changelog */}}
{{- range $author := $authorList }}
{{- $authorLower := lower $author }}
{{- /* we only want to include non-core team contributors */}}
{{- if not (has $authorLower $maintainers)}}
{{- if not (has $authorLower $core_team)}}
{{- $changeList := splitList " " $change.Custom.Author }}
{{- $IssueList := list }}
{{- /* Docs kind link back to dbt-docs instead of dbt-core issues */}}
{{- $changeLink := $change.Kind }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }}
{{- $changeLink = "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $change.Custom.PR }}
{{- else if eq $change.Kind "Docs"}}
{{- $changeLink = "[dbt-docs/#nbr](https://github.com/dbt-labs/dbt-docs/issues/nbr)" | replace "nbr" $change.Custom.Issue }}
{{- else }}
{{- $changeLink = "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $change.Custom.Issue }}
{{- end }}
{{- /* check if this contributor has other changes associated with them already */}}
{{- if hasKey $contributorDict $author }}
{{- $contributionList := get $contributorDict $author }}
{{- $contributionList = concat $contributionList $IssueList }}
{{- $contributionList = append $contributionList $changeLink }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- else }}
{{- $contributionList := $IssueList }}
{{- $contributionList := list $changeLink }}
{{- $contributorDict := set $contributorDict $author $contributionList }}
{{- end }}
{{- end}}

View File

@@ -7,9 +7,6 @@ ignore =
W503 # makes Flake8 work like black
W504
E203 # makes Flake8 work like black
E704 # makes Flake8 work like black
E741
E501 # long line checking is done in black
exclude = test/
per-file-ignores =
*/__init__.py: F401

6
.gitattributes vendored
View File

@@ -1,6 +1,2 @@
core/dbt/task/docs/index.html binary
core/dbt/include/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary
core/dbt/docs/build/html/searchindex.js binary
core/dbt/docs/build/html/index.html binary
performance/runner/Cargo.lock binary
core/dbt/events/types_pb2.py binary

63
.github/CODEOWNERS vendored
View File

@@ -11,8 +11,65 @@
# As a default for areas with no assignment,
# the core team as a whole will be assigned
* @dbt-labs/core-team
* @dbt-labs/core
### ARTIFACTS
# Changes to GitHub configurations including Actions
/.github/ @leahwicz
/schemas/dbt @dbt-labs/cloud-artifacts
### LANGUAGE
# Language core modules
/core/dbt/config/ @dbt-labs/core-language
/core/dbt/context/ @dbt-labs/core-language
/core/dbt/contracts/ @dbt-labs/core-language
/core/dbt/deps/ @dbt-labs/core-language
/core/dbt/events/ @dbt-labs/core-language # structured logging
/core/dbt/parser/ @dbt-labs/core-language
# Language misc files
/core/dbt/dataclass_schema.py @dbt-labs/core-language
/core/dbt/hooks.py @dbt-labs/core-language
/core/dbt/node_types.py @dbt-labs/core-language
/core/dbt/semver.py @dbt-labs/core-language
### EXECUTION
# Execution core modules
/core/dbt/graph/ @dbt-labs/core-execution
/core/dbt/task/ @dbt-labs/core-execution
# Execution misc files
/core/dbt/compilation.py @dbt-labs/core-execution
/core/dbt/flags.py @dbt-labs/core-execution
/core/dbt/lib.py @dbt-labs/core-execution
/core/dbt/main.py @dbt-labs/core-execution
/core/dbt/profiler.py @dbt-labs/core-execution
/core/dbt/selected_resources.py @dbt-labs/core-execution
/core/dbt/tracking.py @dbt-labs/core-execution
/core/dbt/version.py @dbt-labs/core-execution
### ADAPTERS
# Adapter interface ("base" + "sql" adapter defaults, cache)
/core/dbt/adapters @dbt-labs/core-adapters
# Global project (default macros + materializations), starter project
/core/dbt/include @dbt-labs/core-adapters
# Postgres plugin
/plugins/ @dbt-labs/core-adapters
# Functional tests for adapter plugins
/tests/adapter @dbt-labs/core-adapters
### TESTS
# Overlapping ownership for vast majority of unit + functional tests
# Perf regression testing framework
# This excludes the test project files itself since those aren't specific
# framework changes (excluded by not setting an owner next to it- no owner)
/performance @nathaniel-may
/performance/projects

View File

@@ -61,7 +61,7 @@ body:
label: Environment
description: |
examples:
- **OS**: Ubuntu 24.04
- **OS**: Ubuntu 20.04
- **Python**: 3.9.12 (`python3 --version`)
- **dbt-core**: 1.1.1 (`dbt --version`)
value: |

View File

@@ -1,18 +0,0 @@
name: 📄 Code docs
description: Report an issue for markdown files within this repo, such as README, ARCHITECTURE, etc.
title: "[Code docs] <title>"
labels: ["triage"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this code docs issue!
- type: textarea
attributes:
label: Please describe the issue and your proposals.
description: |
Links? References? Anything that will give us more context about the issue you are encountering!
Tip: You can attach images by clicking this area to highlight it and then dragging files in.
validations:
required: false

View File

@@ -1,8 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Documentation
url: https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose
about: Problems and issues with dbt product documentation hosted on docs.getdbt.com. Issues for markdown files within this repo, such as README, should be opened using the "Code docs" template.
- name: Ask the community for help
url: https://github.com/dbt-labs/docs.getdbt.com/discussions
about: Need help troubleshooting? Check out our guide on how to ask
@@ -12,6 +9,15 @@ contact_links:
- name: Participate in Discussions
url: https://github.com/dbt-labs/dbt-core/discussions
about: Do you have a Big Idea for dbt? Read open discussions, or start a new one
- name: Create an issue for adapters
url: https://github.com/dbt-labs/dbt-adapters/issues/new/choose
about: Report a bug or request a feature for an adapter
- name: Create an issue for dbt-redshift
url: https://github.com/dbt-labs/dbt-redshift/issues/new/choose
about: Report a bug or request a feature for dbt-redshift
- name: Create an issue for dbt-bigquery
url: https://github.com/dbt-labs/dbt-bigquery/issues/new/choose
about: Report a bug or request a feature for dbt-bigquery
- name: Create an issue for dbt-snowflake
url: https://github.com/dbt-labs/dbt-snowflake/issues/new/choose
about: Report a bug or request a feature for dbt-snowflake
- name: Create an issue for dbt-spark
url: https://github.com/dbt-labs/dbt-spark/issues/new/choose
about: Report a bug or request a feature for dbt-spark

View File

@@ -1,67 +0,0 @@
name: 🛠️ Implementation
description: This is an implementation ticket intended for use by the maintainers of dbt-core
title: "[<project>] <title>"
labels: ["user docs"]
body:
- type: markdown
attributes:
value: This is an implementation ticket intended for use by the maintainers of dbt-core
- type: checkboxes
attributes:
label: Housekeeping
description: >
A couple friendly reminders:
1. Remove the `user docs` label if the scope of this work does not require changes to https://docs.getdbt.com/docs: no end-user interface (e.g. yml spec, CLI, error messages, etc) or functional changes
2. Link any blocking issues in the "Blocked on" field under the "Core devs & maintainers" project.
options:
- label: I am a maintainer of dbt-core
required: true
- type: textarea
attributes:
label: Short description
description: |
Describe the scope of the ticket, a high-level implementation approach and any tradeoffs to consider
validations:
required: true
- type: textarea
attributes:
label: Acceptance criteria
description: |
What is the definition of done for this ticket? Include any relevant edge cases and/or test cases
validations:
required: true
- type: textarea
attributes:
label: Suggested Tests
description: |
Provide scenarios to test. Link to existing similar tests if appropriate.
placeholder: |
1. Test with no version specified in the schema file and use selection logic on a versioned model for a specific version. Expect pass.
2. Test with a version specified in the schema file that is no valid. Expect ParsingError.
validations:
required: true
- type: textarea
attributes:
label: Impact to Other Teams
description: |
Will this change impact other teams? Include details of the kinds of changes required (new tests, code changes, related tickets) and _add the relevant `Impact:[team]` label_.
placeholder: |
Example: This change impacts `dbt-redshift` because the tests will need to be modified. The `Impact:[Adapter]` label has been added.
validations:
required: true
- type: textarea
attributes:
label: Will backports be required?
description: |
Will this change need to be backported to previous versions? Add details, possible blockers to backporting and _add the relevant backport labels `backport 1.x.latest`_
placeholder: |
Example: Backport to 1.6.latest, 1.5.latest and 1.4.latest. Since 1.4 isn't using click, the backport may be complicated. The `backport 1.6.latest`, `backport 1.5.latest` and `backport 1.4.latest` labels have been added.
validations:
required: true
- type: textarea
attributes:
label: Context
description: |
Provide the "why", motivation, and alternative approaches considered -- linking to previous refinement issues, spikes and documentation as appropriate
validations:
required: false

View File

@@ -55,7 +55,7 @@ body:
label: Environment
description: |
examples:
- **OS**: Ubuntu 24.04
- **OS**: Ubuntu 20.04
- **Python**: 3.9.12 (`python3 --version`)
- **dbt-core (working version)**: 1.1.1 (`dbt --version`)
- **dbt-core (regression version)**: 1.2.0 (`dbt --version`)

13
.github/_README.md vendored
View File

@@ -47,8 +47,7 @@ ___
### How to re-run jobs
- From the UI you can rerun from failure
- You can retrigger the cla check by commenting on the PR with `@cla-bot check`
- Some actions cannot be rerun in the GitHub UI. Namely the snyk checks and the cla check. Snyk checks are rerun by closing and reopening the PR. You can retrigger the cla check by commenting on the PR with `@cla-bot check`
___
@@ -120,7 +119,7 @@ Some triggers of note that we use:
```yaml
jobs:
dependency_changelog:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- name: Get File Name Timestamp
@@ -188,12 +187,6 @@ ___
- The [GitHub CLI](https://cli.github.com/) is available in the default runners
- Actions run in your context. ie, using an action from the marketplace that uses the GITHUB_TOKEN uses the GITHUB_TOKEN generated by your workflow run.
### Runners
- We dynamically set runners based on repository vars. Admins can view repository vars and reset them. Current values are the following but are subject to change:
- `vars.UBUNTU_LATEST` -> `ubuntu-latest`
- `vars.WINDOWS_LATEST` -> `windows-latest`
- `vars.MACOS_LATEST` -> `macos-14`
### Actions from the Marketplace
- Dont use external actions for things that can easily be accomplished manually.
- Always read through what an external action does before using it! Often an action in the GitHub Actions Marketplace can be replaced with a few lines in bash. This is much more maintainable (and wont change under us) and clear as to whats actually happening. It also prevents any
@@ -204,7 +197,7 @@ ___
```yaml
- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v2
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

View File

@@ -33,9 +33,9 @@ on:
jobs:
build:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v1
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler

View File

@@ -1,21 +1,20 @@
name: "GitHub package `latest` tag wrangler for containers"
description: "Determines if the published image should include `latest` tags"
name: "Github package 'latest' tag wrangler for containers"
description: "Determines wether or not a given dbt container should be given a bare 'latest' tag (I.E. dbt-core:latest)"
inputs:
package_name:
description: "Package being published (i.e. `dbt-core`, `dbt-redshift`, etc.)"
description: "Package to check (I.E. dbt-core, dbt-redshift, etc)"
required: true
new_version:
description: "SemVer of the package being published (i.e. 1.7.2, 1.8.0a1, etc.)"
description: "Semver of the container being built (I.E. 1.0.4)"
required: true
github_token:
description: "Auth token for GitHub (must have view packages scope)"
gh_token:
description: "Auth token for github (must have view packages scope)"
required: true
outputs:
tags:
description: "A list of tags to associate with this version"
latest:
description: "Wether or not built container should be tagged latest (bool)"
minor_latest:
description: "Wether or not built container should be tagged minor.latest (bool)"
runs:
using: "docker"
image: "Dockerfile"

View File

@@ -3,24 +3,24 @@ on:
workflow_dispatch:
inputs:
package:
description: The package to publish
required: true
description: The package to publish
required: true
version_number:
description: The version number
required: true
description: The version number
required: true
jobs:
build:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler
with:
package: ${{ github.event.inputs.package }}
new_version: ${{ github.event.inputs.new_version }}
gh_token: ${{ secrets.GITHUB_TOKEN }}
- name: Print the results
run: |
echo "Is it latest? Survey says: ${{ steps.is_latest.outputs.latest }} !"
- uses: actions/checkout@v1
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler
with:
package: ${{ github.event.inputs.package }}
new_version: ${{ github.event.inputs.new_version }}
gh_token: ${{ secrets.GITHUB_TOKEN }}
- name: Print the results
run: |
echo "Is it latest? Survey says: ${{ steps.is_latest.outputs.latest }} !"

View File

@@ -1,71 +1,98 @@
import os
from packaging.version import Version, parse
import requests
import sys
from typing import List
def main():
package_name: str = os.environ["INPUT_PACKAGE_NAME"]
new_version: Version = parse(os.environ["INPUT_NEW_VERSION"])
github_token: str = os.environ["INPUT_GITHUB_TOKEN"]
response = _package_metadata(package_name, github_token)
published_versions = _published_versions(response)
new_version_tags = _new_version_tags(new_version, published_versions)
_register_tags(new_version_tags, package_name)
def _package_metadata(package_name: str, github_token: str) -> requests.Response:
url = f"https://api.github.com/orgs/dbt-labs/packages/container/{package_name}/versions"
return requests.get(url, auth=("", github_token))
def _published_versions(response: requests.Response) -> List[Version]:
package_metadata = response.json()
return [
parse(tag)
for version in package_metadata
for tag in version["metadata"]["container"]["tags"]
if "latest" not in tag
]
def _new_version_tags(new_version: Version, published_versions: List[Version]) -> List[str]:
# the package version is always a tag
tags = [str(new_version)]
# pre-releases don't get tagged with `latest`
if new_version.is_prerelease:
return tags
if new_version > max(published_versions):
tags.append("latest")
published_patches = [
version
for version in published_versions
if version.major == new_version.major and version.minor == new_version.minor
]
if new_version > max(published_patches):
tags.append(f"{new_version.major}.{new_version.minor}.latest")
return tags
def _register_tags(tags: List[str], package_name: str) -> None:
fully_qualified_tags = ",".join([f"ghcr.io/dbt-labs/{package_name}:{tag}" for tag in tags])
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"fully_qualified_tags={fully_qualified_tags}")
def _validate_response(response: requests.Response) -> None:
message = response["message"]
if response.status_code != 200:
print(f"Call to GitHub API failed: {response.status_code} - {message}")
sys.exit(1)
import requests
from distutils.util import strtobool
from typing import Union
from packaging.version import parse, Version
if __name__ == "__main__":
main()
# get inputs
package = os.environ["INPUT_PACKAGE"]
new_version = parse(os.environ["INPUT_NEW_VERSION"])
gh_token = os.environ["INPUT_GH_TOKEN"]
halt_on_missing = strtobool(os.environ.get("INPUT_HALT_ON_MISSING", "False"))
# get package metadata from github
package_request = requests.get(
f"https://api.github.com/orgs/dbt-labs/packages/container/{package}/versions",
auth=("", gh_token),
)
package_meta = package_request.json()
# Log info if we don't get a 200
if package_request.status_code != 200:
print(f"Call to GH API failed: {package_request.status_code} {package_meta['message']}")
# Make an early exit if there is no matching package in github
if package_request.status_code == 404:
if halt_on_missing:
sys.exit(1)
# everything is the latest if the package doesn't exist
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write("latest=True")
gh_output.write("minor_latest=True")
sys.exit(0)
# TODO: verify package meta is "correct"
# https://github.com/dbt-labs/dbt-core/issues/4640
# map versions and tags
version_tag_map = {
version["id"]: version["metadata"]["container"]["tags"] for version in package_meta
}
# is pre-release
pre_rel = True if any(x in str(new_version) for x in ["a", "b", "rc"]) else False
# semver of current latest
for version, tags in version_tag_map.items():
if "latest" in tags:
# N.B. This seems counterintuitive, but we expect any version tagged
# 'latest' to have exactly three associated tags:
# latest, major.minor.latest, and major.minor.patch.
# Subtracting everything that contains the string 'latest' gets us
# the major.minor.patch which is what's needed for comparison.
current_latest = parse([tag for tag in tags if "latest" not in tag][0])
else:
current_latest = False
# semver of current_minor_latest
for version, tags in version_tag_map.items():
if f"{new_version.major}.{new_version.minor}.latest" in tags:
# Similar to above, only now we expect exactly two tags:
# major.minor.patch and major.minor.latest
current_minor_latest = parse([tag for tag in tags if "latest" not in tag][0])
else:
current_minor_latest = False
def is_latest(
pre_rel: bool, new_version: Version, remote_latest: Union[bool, Version]
) -> bool:
"""Determine if a given contaier should be tagged 'latest' based on:
- it's pre-release status
- it's version
- the version of a previously identified container tagged 'latest'
:param pre_rel: Wether or not the version of the new container is a pre-release
:param new_version: The version of the new container
:param remote_latest: The version of the previously identified container that's
already tagged latest or False
"""
# is a pre-release = not latest
if pre_rel:
return False
# + no latest tag found = is latest
if not remote_latest:
return True
# + if remote version is lower than current = is latest, else not latest
return True if remote_latest <= new_version else False
latest = is_latest(pre_rel, new_version, current_latest)
minor_latest = is_latest(pre_rel, new_version, current_minor_latest)
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"latest={latest}")
gh_output.write(f"minor_latest={minor_latest}")

View File

@@ -0,0 +1,10 @@
name: "Set up postgres (linux)"
description: "Set up postgres service on linux vm for dbt integration tests"
runs:
using: "composite"
steps:
- shell: bash
run: |
sudo systemctl start postgresql.service
pg_isready
sudo -u postgres bash ${{ github.action_path }}/setup_db.sh

View File

@@ -0,0 +1 @@
../../../test/setup_db.sh

View File

@@ -0,0 +1,24 @@
name: "Set up postgres (macos)"
description: "Set up postgres service on macos vm for dbt integration tests"
runs:
using: "composite"
steps:
- shell: bash
run: |
brew services start postgresql
echo "Check PostgreSQL service is running"
i=10
COMMAND='pg_isready'
while [ $i -gt -1 ]; do
if [ $i == 0 ]; then
echo "PostgreSQL service not ready, all attempts exhausted"
exit 1
fi
echo "Check PostgreSQL service status"
eval $COMMAND && break
echo "PostgreSQL service not ready, wait 10 more sec, attempts left: $i"
sleep 10
((i--))
done
createuser -s postgres
bash ${{ github.action_path }}/setup_db.sh

Some files were not shown because too many files have changed in this diff Show More