Compare commits

..

65 Commits

Author SHA1 Message Date
Mike Alfare
817a809dd3 updated "skip" to "continue", throw an error on non-implemented macro defaults 2023-06-03 23:38:41 -04:00
Mike Alfare
9fe6cc613a updated "skip" to "continue", throw an error on non-implemented macro defaults 2023-06-02 10:34:05 -04:00
Mike Alfare
feff055c2e updated "skip" to "continue", throw an error on non-implemented macro defaults 2023-06-02 01:52:50 -04:00
Mike Alfare
856704b1cd updated "skip" to "continue", throw an error on non-implemented macro defaults 2023-06-02 01:51:10 -04:00
Mike Alfare
fdb59f81c5 modelled database objects to support monitoring all configuration changes 2023-06-02 01:25:03 -04:00
Mike Alfare
c2ba88763d reverted the change to create a config object from the option object, using just the option object now 2023-05-30 21:18:38 -04:00
Mike Alfare
c50e873535 reverted the change to create a config object from the option object, using just the option object now 2023-05-30 20:56:12 -04:00
Mike Alfare
d2142bfe1c reflected update in tests and materialization logic 2023-05-30 20:13:16 -04:00
Mike Alfare
5280b06af8 reflected update in tests and materialization logic 2023-05-30 19:43:58 -04:00
Mike Alfare
7cb83029b9 updated on_configuration_change to use existing patterns 2023-05-30 19:17:59 -04:00
Mike Alfare
8eb95af9eb updated on_configuration_change to use existing patterns 2023-05-30 19:04:29 -04:00
Mike Alfare
83041f4ff2 fixed all the broken index stuff 2023-05-23 22:11:06 -04:00
Mike Alfare
c51059e795 fixed type to be more generic to accommodate adapters with their own relation types 2023-05-23 14:13:24 -04:00
Mike Alfare
c9c1a2eb41 generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility 2023-05-23 14:06:11 -04:00
Mike Alfare
d57495fbbd remove the "apply" from the tests and put it on the base as the default 2023-05-23 13:39:45 -04:00
Mike Alfare
d1c0a4ffcb made test more readable 2023-05-23 13:28:50 -04:00
Mike Alfare
df21475c8c fixed index issue, removed some indirection in testing 2023-05-23 09:57:00 -04:00
Mike Alfare
c1ae265657 removed assumed models in method calls, removed odd insert records and replaced with get row count 2023-05-22 23:50:58 -04:00
Mike Alfare
9b54d85e81 removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports 2023-05-22 22:59:11 -04:00
Mike Alfare
720da381f7 added FailFastError back, the right way 2023-05-22 21:57:47 -04:00
Mike Alfare
01d42ff08b reverting exceptions_jinja.py 2023-05-22 21:55:37 -04:00
Mike Alfare
1b4a231a71 removed call to skip status 2023-05-22 21:34:41 -04:00
Mike Alfare
e884abd5a1 typo 2023-05-22 21:34:26 -04:00
Mike Alfare
f158a81fa4 update skip and fail scenarios with more descriptive results 2023-05-22 20:44:09 -04:00
Mike Alfare
249f5f7fe6 Merge branch 'main' into feature/materialized-views/ADAP-2 2023-05-19 17:05:18 -04:00
Mike Alfare
cca6246cca break out a large test suite as a separate run 2023-05-19 14:02:57 -04:00
Mike Alfare
59de331e63 rewrite index updates to use a better source in Postgres 2023-05-19 13:02:50 -04:00
Mike Alfare
22a5f1229e still trying to make the test runner happy 2023-05-17 13:26:35 -04:00
Mike Alfare
96ba1b1803 updated naming convention for proper test collection with the test runner 2023-05-17 11:52:34 -04:00
Mike Alfare
0070877124 fixed test import paths 2023-05-17 11:16:36 -04:00
Mike Alfare
dac6088e11 updated import order to standard 2023-05-16 22:12:26 -04:00
Mike Alfare
39b8b0ad8b fixed overwrite issue, simplified assertion method 2023-05-16 22:05:12 -04:00
Mike Alfare
4f72934c89 moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures 2023-05-16 21:51:32 -04:00
Matthew McKnight
76eba9cbc2 removing returns from tests to stop logs from printing 2023-05-16 17:13:30 -05:00
Matthew McKnight
3c95e0ce9b Merge branch 'feature/materialized-views/ADAP-2' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-05-16 12:28:26 -05:00
Mike Alfare
5ae82386da ADAP-395: Implement native materialized view DDL (#7336)
* changie

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* added materialized view stubs and test

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* updated postgres to use the new macros structure

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* init copy of pr 387 to begin 391 implementation

* updating to feature main

* updating to feature main

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* remove postgres-specific materialization in favor of core default materialization

* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies

* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integretion tests

* add on_configuration_change to expected_manifest to pass functuional tests

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* revert to current approach for materializations

* added tests

* move materialized view logic into the `/materializations` directory in line with `dbt-core`

* moved default macros in `dbt-core` into `dbt-postgres`

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* typo

* moved materialized view tests into adapter test framework

* updated materialized view tests to use adapter test framework

* add alter materialized view macro

* add alter materialized view macro

* added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update

* change class name, and config setup

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* added in `on_configuration_change` setting

* updated default option for on_configuration_change on NodeConfig

* updated default option for on_configuration_change on NodeConfig

* fixed list being passed as string bug

* fixed list being passed as string bug

* removed explicit calls to enum values

* removed explicit calls to enum values

* removed unneeded test class

* fixed on_configuration_change to be picked up appropriately

* add test setup for testing fail config option

* remove breakpoint, uncomment tests

* update skip scenario to use empty strings

* update skip scenario to avoid using sql at all, remove extra whitespace in some templates

* push up initial addition of indexes for mv macro

* push slight change up

* reverting alt macro and moving the do create_index call to be more in line with other materializations

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402

# Conflicts:
#	core/dbt/contracts/graph/model_config.py
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/base.py
#	tests/functional/materializations/test_materialized_view.py

* merge feature branch into story branch

* merge feature branch into story branch

* added indexes into the workflow

* fix error in jinja that caused print error

* working on test messaging and skipping tests that might not fit quite into current system

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395

# Conflicts:
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	plugins/postgres/dbt/include/postgres/macros/adapters.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py
#	tests/functional/materializations/test_materialized_view.py

* moved postgres implemention into plugin directory

* update index methods to align with the configuration update macro

* added native ddl to postgres macros

* removed extra docstring

* updated references to View, now references MaterializedView

* decomposed materialization into macros

* refactor index create statement parser, add exceptions for unexpected formats

* swapped conditional to check for positive state

* removed skipped test now that materialized view is being used

* return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test

* add check for indexes in particular for apply on configuration scenario

* removed extra argument

* add materialized views to get_relations / list_relations

* typos in index change logic

* moved full refresh check inside the build sql step

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
2023-05-16 12:26:51 -05:00
Mike Alfare
76c0b007bd Merge branch 'main' into feature/materialized-views/ADAP-2 2023-05-16 11:31:24 -04:00
Matthew McKnight
a20a3b514b Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-05-15 15:27:21 -05:00
Mike Alfare
f7aefd187b Merge branch 'main' into feature/materialized-views/ADAP-2 2023-05-15 15:52:05 -04:00
Mike Alfare
96d9439301 Merge branch 'main' into feature/materialized-views/ADAP-2 2023-05-11 13:26:28 -04:00
Matthew McKnight
92627745a6 Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-05-11 12:26:08 -05:00
Matthew McKnight
ed65ce6c9c Merge branch 'feature/materialized-views/ADAP-2' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-05-11 12:25:54 -05:00
Mike Alfare
0d341dc3c1 ADAP-402: Add configuration change option - postgres (#7334)
* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* added materialized view stubs and test

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* updated postgres to use the new macros structure

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* init copy of pr 387 to begin 391 implementation

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* remove postgres-specific materialization in favor of core default materialization

* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies

* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integretion tests

* add on_configuration_change to expected_manifest to pass functuional tests

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* revert to current approach for materializations

* added tests

* move materialized view logic into the `/materializations` directory in line with `dbt-core`

* moved default macros in `dbt-core` into `dbt-postgres`

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* moved materialized view tests into adapter test framework

* updated materialized view tests to use adapter test framework

* add alter materialized view macro

* add alter materialized view macro

* change class name, and config setup

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* added in `on_configuration_change` setting

* updated default option for on_configuration_change on NodeConfig

* updated default option for on_configuration_change on NodeConfig

* fixed list being passed as string bug

* removed explicit calls to enum values

* removed unneeded test class

* fixed on_configuration_change to be picked up appropriately

* add test setup for testing fail config option

* remove breakpoint, uncomment tests

* update skip scenario to use empty strings

* update skip scenario to avoid using sql at all, remove extra whitespace in some templates

* push up initial addition of indexes for mv macro

* push slight change up

* reverting alt macro and moving the do create_index call to be more in line with other materializations

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402

# Conflicts:
#	core/dbt/contracts/graph/model_config.py
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/base.py
#	tests/functional/materializations/test_materialized_view.py

* merge feature branch into story branch

* merge feature branch into story branch

* added indexes into the workflow

* fix error in jinja that caused print error

* working on test messaging and skipping tests that might not fit quite into current system

* add drop and show macros for indexes

* add drop and show macros for indexes

* add logic to determine the indexes to create or drop

* pulled index updates through the workflow properly

* convert configuration changes to fixtures, implement index changes into tests

* created Model dataclass for readability, added column to swap index columns for testing

* fixed typo

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
2023-05-11 13:25:20 -04:00
Mike Alfare
88c4a973f6 Merge branch 'main' into feature/materialized-views/ADAP-2 2023-05-09 12:46:50 -04:00
Matthew McKnight
205b57bbb6 Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-05-09 11:27:38 -05:00
Matthew McKnight
bba3a6c278 Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-04-28 10:32:51 -05:00
Mike Alfare
f4aab05b25 ADAP-388: Stub materialized view as a materialization - postgres (#7244)
* move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
2023-04-27 22:38:54 -04:00
Mike Alfare
9b8e8cb819 Merge branch 'main' into feature/materialized-views/ADAP-2 2023-04-27 18:45:29 -04:00
Matthew McKnight
3f46e7d2f9 ADAP-391: Add configuration change option (#7272)
* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* move up pre-existing drop of backup

* change relation type to view to be consistent

* add base test case

* fix jinja exception message expression, basic test passing

* align names between postgres and default to same convention

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integration tests

* add on_configuration_change to expected_manifest to pass functional tests

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* moved materialized view tests into adapter test framework

* add alter materialized view macro

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* updated default option for on_configuration_change on NodeConfig

* removed explicit calls to enum values

* add test setup for testing fail config option

* updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name

* setup configuration change framework

* skipped tests that are expected to fail without adapter implementation

* cleaned up log checks

---------

Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
2023-04-27 17:55:33 -04:00
Matthew McKnight
269888c047 Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-04-27 13:50:05 -05:00
Mike Alfare
aabd97e5c5 Merge branch 'main' into feature/materialized-views/ADAP-2 2023-04-25 16:13:24 -04:00
Matthew McKnight
7747f8b4ca Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-04-21 14:02:37 -05:00
Mike Alfare
e6d59e499f Merge branch 'main' into feature/materialized-views/ADAP-2 2023-04-19 15:36:21 -04:00
Matthew McKnight
8e4b1a71e6 Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-04-17 10:50:21 -05:00
Matthew McKnight
9fae72bb8b ADAP-387: reverting db_api implementation (#7322)
* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* updating to feature main

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* added tests

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* typo

* moved materialized view tests into adapter test framework

* add enum to relation for comparison in jinja

---------

Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
2023-04-12 16:32:13 -04:00
Mike Alfare
8f3cc58b78 Merge branch 'main' into feature/materialized-views/ADAP-2 2023-04-12 15:57:58 -04:00
Mike Alfare
2fe362e039 Merge branch 'main' into feature/materialized-views/ADAP-2 2023-04-11 19:57:26 -04:00
Matthew McKnight
619c138190 Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-04-11 11:21:26 -05:00
Mike Alfare
30b73f5b74 Merge branch 'main' into feature/materialized-views/ADAP-2 2023-04-10 12:12:43 -04:00
Matthew McKnight
1efadbd83f Merge branch 'main' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-04-10 10:21:15 -05:00
Mike Alfare
6a460ef5ad ADAP-387: Stub materialized view as a materialization (#7211)
* changie

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* updating to feature main

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
2023-04-07 13:20:33 -05:00
Matthew McKnight
454d85f349 updating to main 2023-04-04 14:32:23 -05:00
Matthew McKnight
486a7bb836 Merge branch 'feature/materialized-views/ADAP-2' of github.com:dbt-labs/dbt into feature/materialized-views/ADAP-2 2023-04-04 14:32:06 -05:00
Matthew McKnight
2ee2f1ea91 updating to main 2023-04-04 14:31:34 -05:00
Mike Alfare
6e2476e84e changie 2023-03-29 12:03:31 -04:00
5269 changed files with 122818 additions and 159234 deletions

47
.bumpversion.cfg Normal file
View File

@@ -0,0 +1,47 @@
[bumpversion]
current_version = 1.6.0b1
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
(?P<prerelease> # optional pre-release - ex: a1, b2, rc25
(?P<prekind>a|b|rc) # pre-release type
(?P<num>[\d]+) # pre-release version number
)?
( # optional nightly release indicator
\.(?P<nightly>dev[0-9]+) # ex: .dev02142023
)? # expected matches: `1.15.0`, `1.5.0a11`, `1.5.0a1.dev123`, `1.5.0.dev123457`, expected failures: `1`, `1.5`, `1.5.2-a1`, `text1.5.0`
serialize =
{major}.{minor}.{patch}{prekind}{num}.{nightly}
{major}.{minor}.{patch}.{nightly}
{major}.{minor}.{patch}{prekind}{num}
{major}.{minor}.{patch}
commit = False
tag = False
[bumpversion:part:prekind]
first_value = a
optional_value = final
values =
a
b
rc
final
[bumpversion:part:num]
first_value = 1
[bumpversion:part:nightly]
[bumpversion:file:core/setup.py]
[bumpversion:file:core/dbt/version.py]
[bumpversion:file:plugins/postgres/setup.py]
[bumpversion:file:plugins/postgres/dbt/adapters/postgres/__version__.py]
[bumpversion:file:docker/Dockerfile]
[bumpversion:file:tests/adapter/setup.py]
[bumpversion:file:tests/adapter/dbt/tests/adapter/__version__.py]

View File

@@ -3,12 +3,6 @@
For information on prior major and minor releases, see their changelogs:
* [1.10](https://github.com/dbt-labs/dbt-core/blob/1.10.latest/CHANGELOG.md)
* [1.9](https://github.com/dbt-labs/dbt-core/blob/1.9.latest/CHANGELOG.md)
* [1.8](https://github.com/dbt-labs/dbt-core/blob/1.8.latest/CHANGELOG.md)
* [1.7](https://github.com/dbt-labs/dbt-core/blob/1.7.latest/CHANGELOG.md)
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)

1
.changes/1.6.0-a1.md Normal file
View File

@@ -0,0 +1 @@
## dbt-core 1.6.0-a1 - April 17, 2023

50
.changes/1.6.0-b1.md Normal file
View File

@@ -0,0 +1,50 @@
## dbt-core 1.6.0-b1 - May 12, 2023
### Features
- Skip catalog generation ([#6980](https://github.com/dbt-labs/dbt-core/issues/6980))
- Publication artifacts and cross-project ref ([#7227](https://github.com/dbt-labs/dbt-core/issues/7227))
- Add graph structure summaries to target path output ([#7357](https://github.com/dbt-labs/dbt-core/issues/7357))
- Allow duplicate manifest node (models, seeds, analyses, snapshots) names across packages ([#7446](https://github.com/dbt-labs/dbt-core/issues/7446))
- Detect breaking changes to enforced constraints ([#7065](https://github.com/dbt-labs/dbt-core/issues/7065))
- Check for project dependency cycles ([#7468](https://github.com/dbt-labs/dbt-core/issues/7468))
### Fixes
- Persist timing info in run results for failed nodes ([#5476](https://github.com/dbt-labs/dbt-core/issues/5476))
- fix typo in unpacking statically parsed ref ([#7364](https://github.com/dbt-labs/dbt-core/issues/7364))
- safe version attribute access in _check_resource_uniqueness ([#7375](https://github.com/dbt-labs/dbt-core/issues/7375))
- Fix dbt command missing target-path param ([# 7411](https://github.com/dbt-labs/dbt-core/issues/ 7411))
- Fix v0 ref resolution ([#7408](https://github.com/dbt-labs/dbt-core/issues/7408))
- Add --target-path to dbt snapshot command. ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- dbt build selection of tests' descendants ([#7289](https://github.com/dbt-labs/dbt-core/issues/7289))
- fix groupable node partial parsing, raise DbtReferenceError at runtime for safety ([#7437](https://github.com/dbt-labs/dbt-core/issues/7437))
- Fix partial parsing of latest_version changes for downstream references ([#7369](https://github.com/dbt-labs/dbt-core/issues/7369))
- Use "add_node" to update depends_on.nodes ([#7453](https://github.com/dbt-labs/dbt-core/issues/7453))
- Fix var precedence in configs: root vars override package vars ([#6705](https://github.com/dbt-labs/dbt-core/issues/6705))
- Fix inverted `--print/--no-print` flag ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
- Back-compat for previous return type of 'collect_freshness' macro ([#7489](https://github.com/dbt-labs/dbt-core/issues/7489))
- print model version in dbt show if specified ([#7407](https://github.com/dbt-labs/dbt-core/issues/7407))
- enable dbt show for seeds ([#7273](https://github.com/dbt-labs/dbt-core/issues/7273))
- push down limit filtering to adapter ([#7390](https://github.com/dbt-labs/dbt-core/issues/7390))
- Allow missing `profiles.yml` for `dbt deps` and `dbt init` ([#7511](https://github.com/dbt-labs/dbt-core/issues/7511))
- `run_results.json` is now written after every node completes. ([#7302](https://github.com/dbt-labs/dbt-core/issues/7302))
- Do not rewrite manifest.json during 'docs serve' command ([#7553](https://github.com/dbt-labs/dbt-core/issues/7553))
- Pin protobuf to greater than 4.0.0 ([#7565](https://github.com/dbt-labs/dbt-core/issues/7565))
- inject sql header in query for show ([#7413](https://github.com/dbt-labs/dbt-core/issues/7413))
- Pin urllib3 to ~=1.0 ([#7573](https://github.com/dbt-labs/dbt-core/issues/7573))
- Throw error for duplicated versioned and unversioned models ([#7487](https://github.com/dbt-labs/dbt-core/issues/7487))
### Under the Hood
- Update docs link in ContractBreakingChangeError message ([#7366](https://github.com/dbt-labs/dbt-core/issues/7366))
- Reduce memory footprint of cached statement results. ([#7281](https://github.com/dbt-labs/dbt-core/issues/7281))
- Remove noisy parsing events: GenericTestFileParse, MacroFileParse, Note events for static model parsing ([#6671](https://github.com/dbt-labs/dbt-core/issues/6671))
- Update --help text for cache-related parameters ([#7381](https://github.com/dbt-labs/dbt-core/issues/7381))
- Small UX improvements to model versions: Support defining latest_version in unsuffixed file by default. Notify on unpinned ref when a prerelease version is available. ([#7443](https://github.com/dbt-labs/dbt-core/issues/7443))
### Contributors
- [@AndyBys](https://github.com/AndyBys) ([#6980](https://github.com/dbt-labs/dbt-core/issues/6980))
- [@b-luu](https://github.com/b-luu) ([#7289](https://github.com/dbt-labs/dbt-core/issues/7289))
- [@dwreeves](https://github.com/dwreeves) ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- [@thomasgjerdekog](https://github.com/thomasgjerdekog) ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))

View File

@@ -0,0 +1,6 @@
kind: Features
body: Skip catalog generation
time: 2023-03-21T21:33:38.513443Z
custom:
Author: AndyBys
Issue: "6980"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Publication artifacts and cross-project ref
time: 2023-04-20T12:47:56.92683-04:00
custom:
Author: gshank
Issue: "7227"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add graph structure summaries to target path output
time: 2023-04-25T14:25:22.269051-04:00
custom:
Author: peterallenwebb
Issue: "7357"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Allow duplicate manifest node (models, seeds, analyses, snapshots) names across
packages
time: 2023-04-29T15:50:57.757832-04:00
custom:
Author: MichelleArk
Issue: "7446"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Detect breaking changes to enforced constraints
time: 2023-05-03T10:11:00.617936-05:00
custom:
Author: emmyoop
Issue: "7065"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Check for project dependency cycles
time: 2023-05-09T09:41:47.2-04:00
custom:
Author: gshank
Issue: "7468"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Persist timing info in run results for failed nodes
time: 2023-04-13T13:31:57.938847-05:00
custom:
Author: stu-k
Issue: "5476"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix typo in unpacking statically parsed ref
time: 2023-04-14T16:36:42.279838-04:00
custom:
Author: MichelleArk
Issue: "7364"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: safe version attribute access in _check_resource_uniqueness
time: 2023-04-18T13:52:57.367108-04:00
custom:
Author: MichelleArk
Issue: "7375"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix dbt command missing target-path param
time: 2023-04-19T14:21:50.959786-07:00
custom:
Author: ChenyuLInx
Issue: "\t7411"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix v0 ref resolution
time: 2023-04-19T22:09:10.155137-04:00
custom:
Author: MichelleArk
Issue: "7408"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add --target-path to dbt snapshot command.
time: 2023-04-20T10:42:54.17972-04:00
custom:
Author: dwreeves
Issue: "7418"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: dbt build selection of tests' descendants
time: 2023-04-21T17:24:28.335866975+02:00
custom:
Author: b-luu
Issue: "7289"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix groupable node partial parsing, raise DbtReferenceError at runtime for safety
time: 2023-04-24T16:18:43.130637-04:00
custom:
Author: MichelleArk
Issue: "7437"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix partial parsing of latest_version changes for downstream references
time: 2023-04-24T16:46:49.721231-04:00
custom:
Author: MichelleArk
Issue: "7369"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Use "add_node" to update depends_on.nodes
time: 2023-04-24T17:34:04.37479-04:00
custom:
Author: gshank
Issue: "7453"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Fix var precedence in configs: root vars override package vars'
time: 2023-04-27T23:07:14.992529-04:00
custom:
Author: MichelleArk
Issue: "6705"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix inverted `--print/--no-print` flag
time: 2023-05-05T13:25:45.949997-06:00
custom:
Author: dbeatty10 thomasgjerdekog
Issue: "7517"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Back-compat for previous return type of 'collect_freshness' macro
time: 2023-05-06T17:33:15.104953+02:00
custom:
Author: jtcohen6
Issue: "7489"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: print model version in dbt show if specified
time: 2023-05-08T04:27:07.9965-07:00
custom:
Author: aranke
Issue: "7407"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: enable dbt show for seeds
time: 2023-05-08T04:49:22.82093-07:00
custom:
Author: aranke
Issue: "7273"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: push down limit filtering to adapter
time: 2023-05-08T06:09:26.455524-07:00
custom:
Author: aranke
Issue: "7390"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Allow missing `profiles.yml` for `dbt deps` and `dbt init`
time: 2023-05-08T07:29:29.873793-06:00
custom:
Author: dbeatty10
Issue: "7511"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: '`run_results.json` is now written after every node completes.'
time: 2023-05-08T09:37:32.809356-05:00
custom:
Author: iknox-fa
Issue: "7302"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Do not rewrite manifest.json during 'docs serve' command
time: 2023-05-08T14:25:18.376379-04:00
custom:
Author: jtcohen6
Issue: "7553"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: inject sql header in query for show
time: 2023-05-09T10:29:32.12947-07:00
custom:
Author: aranke
Issue: "7413"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Pin protobuf to greater than 4.0.0
time: 2023-05-09T12:31:59.019718-04:00
custom:
Author: gshank
Issue: "7565"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Pin urllib3 to ~=1.0
time: 2023-05-09T15:35:30.504674-04:00
custom:
Author: mikealfare
Issue: "7573"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Throw error for duplicated versioned and unversioned models
time: 2023-05-09T16:50:07.530882-04:00
custom:
Author: gshank
Issue: "7487"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update docs link in ContractBreakingChangeError message
time: 2023-04-17T11:45:01.005104+02:00
custom:
Author: jtcohen6
Issue: "7366"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Reduce memory footprint of cached statement results.
time: 2023-04-17T12:27:21.972268-05:00
custom:
Author: iknox-fa
Issue: "7281"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: 'Remove noisy parsing events: GenericTestFileParse, MacroFileParse, Note events
for static model parsing'
time: 2023-04-18T12:12:36.928665+02:00
custom:
Author: jtcohen6
Issue: "6671"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update --help text for cache-related parameters
time: 2023-04-18T12:23:23.276693+02:00
custom:
Author: jtcohen6
Issue: "7381"

View File

@@ -0,0 +1,8 @@
kind: Under the Hood
body: 'Small UX improvements to model versions: Support defining latest_version in
unsuffixed file by default. Notify on unpinned ref when a prerelease version is
available. '
time: 2023-04-24T13:53:00.484916+02:00
custom:
Author: jtcohen6
Issue: "7443"

View File

@@ -1,6 +1,6 @@
# dbt Core Changelog
- This file provides a full account of all changes to `dbt-core`
- This file provides a full account of all changes to `dbt-core` and `dbt-postgres`
- Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Use EventCatcher from dbt-common instead of maintaining a local copy
time: 2025-11-18T15:53:54.284561+05:30
custom:
Author: 3loka
Issue: "12124"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add support for materialized views
time: 2023-03-29T12:03:13.862041-04:00
custom:
Author: mikealfare McKnight-42
Issue: "6911"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Update drop_relation macro to allow for configuration of drop statement separately
from object name
time: 2023-05-15T12:23:04.177141-04:00
custom:
Author: mikealfare
Issue: "7625"

View File

@@ -0,0 +1,6 @@
kind: Features
body: accept publications in dbt.invoke
time: 2023-05-17T18:56:27.585784-04:00
custom:
Author: MichelleArk
Issue: "7372"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support partial parsing for function nodes
time: 2025-10-06T14:03:52.258104-05:00
custom:
Author: QMalcolm
Issue: "12072"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Allow for defining funciton arguments with default values
time: 2025-11-17T14:10:53.860178-06:00
custom:
Author: QMalcolm
Issue: "12044"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Raise jsonschema-based deprecation warnings by default
time: 2025-12-01T16:52:09.354436-05:00
custom:
Author: michelleark
Issue: 12240

View File

@@ -1,6 +0,0 @@
kind: Features
body: ':bug: :snowman: Disable unit tests whose model is disabled'
time: 2025-12-03T12:29:26.209248-05:00
custom:
Author: michelleark
Issue: "10540"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Implement config.meta_get and config.meta_require
time: 2025-12-10T20:20:01.354288-05:00
custom:
Author: gshank
Issue: "12012"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Fix: Relative project paths weren''t working with deps'
time: 2023-05-15T12:36:54.807413-05:00
custom:
Author: iknox-fa
Issue: "7491"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Exclude password fields from Jinja rendering.
time: 2023-05-15T14:28:51.400321-04:00
custom:
Author: peterallenwebb
Issue: "7629"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add --target-path to more CLI subcommands
time: 2023-05-16T15:26:44.557072-04:00
custom:
Author: dwreeves
Issue: "7646"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Address Click 8.2+ deprecation warning
time: 2025-09-22T15:17:26.983151-06:00
custom:
Author: edgarrmondragon
Issue: "12038"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Include macros in unit test parsing
time: 2025-11-17T14:06:49.518566-05:00
custom:
Author: michelleark nathanskone
Issue: "10157"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Allow dbt deps to run when vars lack defaults in dbt_project.yml
time: 2025-11-17T18:50:25.759091+05:30
custom:
Author: 3loka
Issue: "8913"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Restore DuplicateResourceNameError for intra-project node name duplication, behind behavior flag `require_unique_project_resource_names`
time: 2025-11-18T17:11:06.454784-05:00
custom:
Author: michelleark
Issue: "12152"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Allow the usage of `function` with `--exclude-resource-type` flag
time: 2025-11-19T19:50:34.703236-06:00
custom:
Author: QMalcolm
Issue: "12143"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix bug where schemas of functions weren't guaranteed to exist
time: 2025-11-24T15:56:29.467004-06:00
custom:
Author: QMalcolm
Issue: "12142"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix generation of deprecations summary
time: 2025-11-24T15:57:56.544123-08:00
custom:
Author: asiunov
Issue: "12146"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Correctly reference foreign key references when --defer and --state provided'
time: 2025-11-24T17:08:55.387946-05:00
custom:
Author: michellark
Issue: "11885"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Add exception when using --state and referring to a removed
test'
time: 2025-11-25T12:02:46.635026-05:00
custom:
Author: emmyoop
Issue: "10630"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Stop emitting `NoNodesForSelectionCriteria` three times during `build` command'
time: 2025-11-25T12:20:20.132379-06:00
custom:
Author: QMalcolm
Issue: "11627"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ":bug: :snowman: Fix long Python stack traces appearing when package dependencies have incompatible version requirements"
time: 2025-11-27T14:13:08.082542-05:00
custom:
Author: emmyoop
Issue: "12049"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Fixed issue where changing data type size/precision/scale (e.g.,
varchar(3) to varchar(10)) incorrectly triggered a breaking change error fo'
time: 2025-11-27T14:59:29.256274-05:00
custom:
Author: emmyoop
Issue: "11186"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Support unit testing models that depend on sources with the same name'
time: 2025-11-27T17:01:24.193516-05:00
custom:
Author: michelleark
Issue: 11975 10433

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix bug in partial parsing when updating a model with a schema file that is referenced by a singular test
time: 2025-11-28T10:21:29.911147Z
custom:
Author: mattogburke
Issue: "12223"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Avoid retrying successful run-operation commands'
time: 2025-11-28T12:28:38.546261-05:00
custom:
Author: michelleark
Issue: "11850"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Fix `dbt deps --add-package` crash when packages.yml contains `warn-unpinned:
false`'
time: 2025-11-28T16:19:37.608722-05:00
custom:
Author: emmyoop
Issue: "9104"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Improve `dbt deps --add-package` duplicate detection with better
cross-source matching and word boundaries'
time: 2025-11-28T16:31:44.344099-05:00
custom:
Author: emmyoop
Issue: "12239"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ':bug: :snowman: Fix false positive deprecation warning of pre/post-hook SQL configs'
time: 2025-12-02T13:37:05.012112-05:00
custom:
Author: michelleark
Issue: "12244"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Ensure recent deprecation warnings include event name in message
time: 2025-12-09T17:50:31.334618-06:00
custom:
Author: QMalcolm
Issue: "12264"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Improve error message clarity when detecting nodes with space in name
time: 2025-12-10T14:39:35.107841-08:00
custom:
Author: michelleark
Issue: "11835"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add ability to instantiate Flags class from dict
time: 2023-05-15T09:51:16.291352-05:00
custom:
Author: stu-k
Issue: "7607"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add other relation to reffable nodes
time: 2023-05-15T15:21:07.808892-05:00
custom:
Author: stu-k
Issue: "7550"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move node patch method to schema parser patch_node_properties and refactor schema
parsing
time: 2023-05-16T09:42:41.793503-04:00
custom:
Author: gshank
Issue: "7430"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Break up integration tests as a short term fix for Windows CI runs
time: 2023-05-19T15:30:59.138043-04:00
custom:
Author: mikealfare
Issue: "7668"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Update jsonschemas for schema.yml and dbt_project.yml deprecations
time: 2025-11-19T11:01:10.616676-05:00
custom:
Author: michelleark
Issue: "12180"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Replace setuptools and tox with hatch for build, test, and environment management.
time: 2025-11-21T14:05:15.838252-05:00
custom:
Author: emmyoop
Issue: "12151"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Add add_catalog_integration call even if we have a pre-existing manifest
time: 2025-12-09T13:18:57.043254-08:00
custom:
Author: colin-rogers-dbt
Issue: "12262"

View File

@@ -31,7 +31,43 @@ kinds:
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
@@ -41,33 +77,47 @@ newlines:
endOfVersion: 1
custom:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: Issue
label: GitHub Issue Number (separated by a single space if multiple)
type: string
minLength: 1
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: Issue
label: GitHub Issue Number (separated by a single space if multiple)
type: string
minLength: 1
footerFormat: |
{{- $contributorDict := dict }}
{{- /* ensure we always skip snyk and dependabot */}}
{{- $bots := list "dependabot[bot]" "snyk-bot"}}
{{- /* ensure all names in this list are all lowercase for later matching purposes */}}
{{- $core_team := splitList " " .Env.CORE_TEAM }}
{{- /* ensure we always skip snyk and dependabot in addition to the core team */}}
{{- $maintainers := list "dependabot[bot]" "snyk-bot"}}
{{- range $team_member := $core_team }}
{{- $team_member_lower := lower $team_member }}
{{- $maintainers = append $maintainers $team_member_lower }}
{{- end }}
{{- range $change := .Changes }}
{{- $authorList := splitList " " $change.Custom.Author }}
{{- /* loop through all authors for a single changelog */}}
{{- range $author := $authorList }}
{{- $authorLower := lower $author }}
{{- /* we only want to include non-bot contributors */}}
{{- if not (has $authorLower $bots)}}
{{- /* we only want to include non-core team contributors */}}
{{- if not (has $authorLower $maintainers)}}
{{- $changeList := splitList " " $change.Custom.Author }}
{{- $IssueList := list }}
{{- $changeLink := $change.Kind }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }}
{{- $changes := splitList " " $change.Custom.PR }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- else }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- end }}
{{- /* check if this contributor has other changes associated with them already */}}
{{- if hasKey $contributorDict $author }}

View File

@@ -7,8 +7,6 @@ ignore =
W503 # makes Flake8 work like black
W504
E203 # makes Flake8 work like black
E704 # makes Flake8 work like black
E741
E501 # long line checking is done in black
per-file-ignores =
*/__init__.py: F401
exclude = test/

2
.gitattributes vendored
View File

@@ -1,4 +1,4 @@
core/dbt/task/docs/index.html binary
core/dbt/include/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary
core/dbt/docs/build/html/searchindex.js binary
core/dbt/docs/build/html/index.html binary

79
.github/CODEOWNERS vendored
View File

@@ -11,7 +11,84 @@
# As a default for areas with no assignment,
# the core team as a whole will be assigned
* @dbt-labs/core-team
* @dbt-labs/core
### OSS Tooling Guild
/.github/ @dbt-labs/guild-oss-tooling
.bumpversion.cfg @dbt-labs/guild-oss-tooling
/.changes/ @dbt-labs/guild-oss-tooling
.changie.yaml @dbt-labs/guild-oss-tooling
pre-commit-config.yaml @dbt-labs/guild-oss-tooling
pytest.ini @dbt-labs/guild-oss-tooling
tox.ini @dbt-labs/guild-oss-tooling
pyproject.toml @dbt-labs/guild-oss-tooling
requirements.txt @dbt-labs/guild-oss-tooling
dev_requirements.txt @dbt-labs/guild-oss-tooling
/core/setup.py @dbt-labs/guild-oss-tooling
/core/MANIFEST.in @dbt-labs/guild-oss-tooling
### LANGUAGE
# Language core modules
/core/dbt/config/ @dbt-labs/core-language
/core/dbt/context/ @dbt-labs/core-language
/core/dbt/contracts/ @dbt-labs/core-language
/core/dbt/deps/ @dbt-labs/core-language
/core/dbt/events/ @dbt-labs/core-language # structured logging
/core/dbt/parser/ @dbt-labs/core-language
# Language misc files
/core/dbt/dataclass_schema.py @dbt-labs/core-language
/core/dbt/hooks.py @dbt-labs/core-language
/core/dbt/node_types.py @dbt-labs/core-language
/core/dbt/semver.py @dbt-labs/core-language
### EXECUTION
# Execution core modules
/core/dbt/graph/ @dbt-labs/core-execution
/core/dbt/task/ @dbt-labs/core-execution
# Execution misc files
/core/dbt/compilation.py @dbt-labs/core-execution
/core/dbt/flags.py @dbt-labs/core-execution
/core/dbt/lib.py @dbt-labs/core-execution
/core/dbt/main.py @dbt-labs/core-execution
/core/dbt/profiler.py @dbt-labs/core-execution
/core/dbt/selected_resources.py @dbt-labs/core-execution
/core/dbt/tracking.py @dbt-labs/core-execution
/core/dbt/version.py @dbt-labs/core-execution
### ADAPTERS
# Adapter interface ("base" + "sql" adapter defaults, cache)
/core/dbt/adapters @dbt-labs/core-adapters
# Global project (default macros + materializations), starter project
/core/dbt/include @dbt-labs/core-adapters
# Postgres plugin
/plugins/ @dbt-labs/core-adapters
/plugins/postgres/setup.py @dbt-labs/core-adapters @dbt-labs/guild-oss-tooling
# Functional tests for adapter plugins
/tests/adapter @dbt-labs/core-adapters
### TESTS
# Overlapping ownership for vast majority of unit + functional tests
# Perf regression testing framework
# This excludes the test project files itself since those aren't specific
# framework changes (excluded by not setting an owner next to it- no owner)
/performance @nathaniel-may
/performance/projects
### ARTIFACTS

View File

@@ -61,8 +61,8 @@ body:
label: Environment
description: |
examples:
- **OS**: Ubuntu 24.04
- **Python**: 3.10.12 (`python3 --version`)
- **OS**: Ubuntu 20.04
- **Python**: 3.9.12 (`python3 --version`)
- **dbt-core**: 1.1.1 (`dbt --version`)
value: |
- OS:

View File

@@ -1,18 +0,0 @@
name: 📄 Code docs
description: Report an issue for markdown files within this repo, such as README, ARCHITECTURE, etc.
title: "[Code docs] <title>"
labels: ["triage"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this code docs issue!
- type: textarea
attributes:
label: Please describe the issue and your proposals.
description: |
Links? References? Anything that will give us more context about the issue you are encountering!
Tip: You can attach images by clicking this area to highlight it and then dragging files in.
validations:
required: false

View File

@@ -1,8 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Documentation
url: https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose
about: Problems and issues with dbt product documentation hosted on docs.getdbt.com. Issues for markdown files within this repo, such as README, should be opened using the "Code docs" template.
- name: Ask the community for help
url: https://github.com/dbt-labs/docs.getdbt.com/discussions
about: Need help troubleshooting? Check out our guide on how to ask
@@ -12,6 +9,15 @@ contact_links:
- name: Participate in Discussions
url: https://github.com/dbt-labs/dbt-core/discussions
about: Do you have a Big Idea for dbt? Read open discussions, or start a new one
- name: Create an issue for adapters
url: https://github.com/dbt-labs/dbt-adapters/issues/new/choose
about: Report a bug or request a feature for an adapter
- name: Create an issue for dbt-redshift
url: https://github.com/dbt-labs/dbt-redshift/issues/new/choose
about: Report a bug or request a feature for dbt-redshift
- name: Create an issue for dbt-bigquery
url: https://github.com/dbt-labs/dbt-bigquery/issues/new/choose
about: Report a bug or request a feature for dbt-bigquery
- name: Create an issue for dbt-snowflake
url: https://github.com/dbt-labs/dbt-snowflake/issues/new/choose
about: Report a bug or request a feature for dbt-snowflake
- name: Create an issue for dbt-spark
url: https://github.com/dbt-labs/dbt-spark/issues/new/choose
about: Report a bug or request a feature for dbt-spark

View File

@@ -1,67 +0,0 @@
name: 🛠️ Implementation
description: This is an implementation ticket intended for use by the maintainers of dbt-core
title: "[<project>] <title>"
labels: ["user docs"]
body:
- type: markdown
attributes:
value: This is an implementation ticket intended for use by the maintainers of dbt-core
- type: checkboxes
attributes:
label: Housekeeping
description: >
A couple friendly reminders:
1. Remove the `user docs` label if the scope of this work does not require changes to https://docs.getdbt.com/docs: no end-user interface (e.g. yml spec, CLI, error messages, etc) or functional changes
2. Link any blocking issues in the "Blocked on" field under the "Core devs & maintainers" project.
options:
- label: I am a maintainer of dbt-core
required: true
- type: textarea
attributes:
label: Short description
description: |
Describe the scope of the ticket, a high-level implementation approach and any tradeoffs to consider
validations:
required: true
- type: textarea
attributes:
label: Acceptance criteria
description: |
What is the definition of done for this ticket? Include any relevant edge cases and/or test cases
validations:
required: true
- type: textarea
attributes:
label: Suggested Tests
description: |
Provide scenarios to test. Link to existing similar tests if appropriate.
placeholder: |
1. Test with no version specified in the schema file and use selection logic on a versioned model for a specific version. Expect pass.
2. Test with a version specified in the schema file that is no valid. Expect ParsingError.
validations:
required: true
- type: textarea
attributes:
label: Impact to Other Teams
description: |
Will this change impact other teams? Include details of the kinds of changes required (new tests, code changes, related tickets) and _add the relevant `Impact:[team]` label_.
placeholder: |
Example: This change impacts `dbt-redshift` because the tests will need to be modified. The `Impact:[Adapter]` label has been added.
validations:
required: true
- type: textarea
attributes:
label: Will backports be required?
description: |
Will this change need to be backported to previous versions? Add details, possible blockers to backporting and _add the relevant backport labels `backport 1.x.latest`_
placeholder: |
Example: Backport to 1.6.latest, 1.5.latest and 1.4.latest. Since 1.4 isn't using click, the backport may be complicated. The `backport 1.6.latest`, `backport 1.5.latest` and `backport 1.4.latest` labels have been added.
validations:
required: true
- type: textarea
attributes:
label: Context
description: |
Provide the "why", motivation, and alternative approaches considered -- linking to previous refinement issues, spikes and documentation as appropriate
validations:
required: false

View File

@@ -55,8 +55,8 @@ body:
label: Environment
description: |
examples:
- **OS**: Ubuntu 24.04
- **Python**: 3.10.12 (`python3 --version`)
- **OS**: Ubuntu 20.04
- **Python**: 3.9.12 (`python3 --version`)
- **dbt-core (working version)**: 1.1.1 (`dbt --version`)
- **dbt-core (regression version)**: 1.2.0 (`dbt --version`)
value: |

11
.github/_README.md vendored
View File

@@ -47,8 +47,7 @@ ___
### How to re-run jobs
- From the UI you can rerun from failure
- You can retrigger the cla check by commenting on the PR with `@cla-bot check`
- Some actions cannot be rerun in the GitHub UI. Namely the snyk checks and the cla check. Snyk checks are rerun by closing and reopening the PR. You can retrigger the cla check by commenting on the PR with `@cla-bot check`
___
@@ -120,7 +119,7 @@ Some triggers of note that we use:
```yaml
jobs:
dependency_changelog:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- name: Get File Name Timestamp
@@ -188,12 +187,6 @@ ___
- The [GitHub CLI](https://cli.github.com/) is available in the default runners
- Actions run in your context. ie, using an action from the marketplace that uses the GITHUB_TOKEN uses the GITHUB_TOKEN generated by your workflow run.
### Runners
- We dynamically set runners based on repository vars. Admins can view repository vars and reset them. Current values are the following but are subject to change:
- `vars.UBUNTU_LATEST` -> `ubuntu-latest`
- `vars.WINDOWS_LATEST` -> `windows-latest`
- `vars.MACOS_LATEST` -> `macos-14`
### Actions from the Marketplace
- Dont use external actions for things that can easily be accomplished manually.
- Always read through what an external action does before using it! Often an action in the GitHub Actions Marketplace can be replaced with a few lines in bash. This is much more maintainable (and wont change under us) and clear as to whats actually happening. It also prevents any

View File

@@ -33,7 +33,7 @@ on:
jobs:
build:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Wrangle latest tag

View File

@@ -1,21 +1,20 @@
name: "GitHub package `latest` tag wrangler for containers"
description: "Determines if the published image should include `latest` tags"
name: "Github package 'latest' tag wrangler for containers"
description: "Determines wether or not a given dbt container should be given a bare 'latest' tag (I.E. dbt-core:latest)"
inputs:
package_name:
description: "Package being published (i.e. `dbt-core`, `dbt-redshift`, etc.)"
description: "Package to check (I.E. dbt-core, dbt-redshift, etc)"
required: true
new_version:
description: "SemVer of the package being published (i.e. 1.7.2, 1.8.0a1, etc.)"
description: "Semver of the container being built (I.E. 1.0.4)"
required: true
github_token:
description: "Auth token for GitHub (must have view packages scope)"
gh_token:
description: "Auth token for github (must have view packages scope)"
required: true
outputs:
tags:
description: "A list of tags to associate with this version"
latest:
description: "Wether or not built container should be tagged latest (bool)"
minor_latest:
description: "Wether or not built container should be tagged minor.latest (bool)"
runs:
using: "docker"
image: "Dockerfile"

View File

@@ -3,24 +3,24 @@ on:
workflow_dispatch:
inputs:
package:
description: The package to publish
required: true
description: The package to publish
required: true
version_number:
description: The version number
required: true
description: The version number
required: true
jobs:
build:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler
with:
package: ${{ github.event.inputs.package }}
new_version: ${{ github.event.inputs.new_version }}
gh_token: ${{ secrets.GITHUB_TOKEN }}
- name: Print the results
run: |
echo "Is it latest? Survey says: ${{ steps.is_latest.outputs.latest }} !"
- uses: actions/checkout@v3
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler
with:
package: ${{ github.event.inputs.package }}
new_version: ${{ github.event.inputs.new_version }}
gh_token: ${{ secrets.GITHUB_TOKEN }}
- name: Print the results
run: |
echo "Is it latest? Survey says: ${{ steps.is_latest.outputs.latest }} !"

View File

@@ -1,72 +1,98 @@
import os
import sys
from typing import List
import requests
from packaging.version import Version, parse
def main():
package_name: str = os.environ["INPUT_PACKAGE_NAME"]
new_version: Version = parse(os.environ["INPUT_NEW_VERSION"])
github_token: str = os.environ["INPUT_GITHUB_TOKEN"]
response = _package_metadata(package_name, github_token)
published_versions = _published_versions(response)
new_version_tags = _new_version_tags(new_version, published_versions)
_register_tags(new_version_tags, package_name)
def _package_metadata(package_name: str, github_token: str) -> requests.Response:
url = f"https://api.github.com/orgs/dbt-labs/packages/container/{package_name}/versions"
return requests.get(url, auth=("", github_token))
def _published_versions(response: requests.Response) -> List[Version]:
package_metadata = response.json()
return [
parse(tag)
for version in package_metadata
for tag in version["metadata"]["container"]["tags"]
if "latest" not in tag
]
def _new_version_tags(new_version: Version, published_versions: List[Version]) -> List[str]:
# the package version is always a tag
tags = [str(new_version)]
# pre-releases don't get tagged with `latest`
if new_version.is_prerelease:
return tags
if new_version > max(published_versions):
tags.append("latest")
published_patches = [
version
for version in published_versions
if version.major == new_version.major and version.minor == new_version.minor
]
if new_version > max(published_patches):
tags.append(f"{new_version.major}.{new_version.minor}.latest")
return tags
def _register_tags(tags: List[str], package_name: str) -> None:
fully_qualified_tags = ",".join([f"ghcr.io/dbt-labs/{package_name}:{tag}" for tag in tags])
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"fully_qualified_tags={fully_qualified_tags}")
def _validate_response(response: requests.Response) -> None:
message = response["message"]
if response.status_code != 200:
print(f"Call to GitHub API failed: {response.status_code} - {message}")
sys.exit(1)
from distutils.util import strtobool
from typing import Union
from packaging.version import parse, Version
if __name__ == "__main__":
main()
# get inputs
package = os.environ["INPUT_PACKAGE"]
new_version = parse(os.environ["INPUT_NEW_VERSION"])
gh_token = os.environ["INPUT_GH_TOKEN"]
halt_on_missing = strtobool(os.environ.get("INPUT_HALT_ON_MISSING", "False"))
# get package metadata from github
package_request = requests.get(
f"https://api.github.com/orgs/dbt-labs/packages/container/{package}/versions",
auth=("", gh_token),
)
package_meta = package_request.json()
# Log info if we don't get a 200
if package_request.status_code != 200:
print(f"Call to GH API failed: {package_request.status_code} {package_meta['message']}")
# Make an early exit if there is no matching package in github
if package_request.status_code == 404:
if halt_on_missing:
sys.exit(1)
# everything is the latest if the package doesn't exist
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write("latest=True")
gh_output.write("minor_latest=True")
sys.exit(0)
# TODO: verify package meta is "correct"
# https://github.com/dbt-labs/dbt-core/issues/4640
# map versions and tags
version_tag_map = {
version["id"]: version["metadata"]["container"]["tags"] for version in package_meta
}
# is pre-release
pre_rel = True if any(x in str(new_version) for x in ["a", "b", "rc"]) else False
# semver of current latest
for version, tags in version_tag_map.items():
if "latest" in tags:
# N.B. This seems counterintuitive, but we expect any version tagged
# 'latest' to have exactly three associated tags:
# latest, major.minor.latest, and major.minor.patch.
# Subtracting everything that contains the string 'latest' gets us
# the major.minor.patch which is what's needed for comparison.
current_latest = parse([tag for tag in tags if "latest" not in tag][0])
else:
current_latest = False
# semver of current_minor_latest
for version, tags in version_tag_map.items():
if f"{new_version.major}.{new_version.minor}.latest" in tags:
# Similar to above, only now we expect exactly two tags:
# major.minor.patch and major.minor.latest
current_minor_latest = parse([tag for tag in tags if "latest" not in tag][0])
else:
current_minor_latest = False
def is_latest(
pre_rel: bool, new_version: Version, remote_latest: Union[bool, Version]
) -> bool:
"""Determine if a given contaier should be tagged 'latest' based on:
- it's pre-release status
- it's version
- the version of a previously identified container tagged 'latest'
:param pre_rel: Wether or not the version of the new container is a pre-release
:param new_version: The version of the new container
:param remote_latest: The version of the previously identified container that's
already tagged latest or False
"""
# is a pre-release = not latest
if pre_rel:
return False
# + no latest tag found = is latest
if not remote_latest:
return True
# + if remote version is lower than current = is latest, else not latest
return True if remote_latest <= new_version else False
latest = is_latest(pre_rel, new_version, current_latest)
minor_latest = is_latest(pre_rel, new_version, current_minor_latest)
github_output = os.environ.get("GITHUB_OUTPUT")
with open(github_output, "at", encoding="utf-8") as gh_output:
gh_output.write(f"latest={latest}")
gh_output.write(f"minor_latest={minor_latest}")

View File

@@ -0,0 +1,10 @@
name: "Set up postgres (linux)"
description: "Set up postgres service on linux vm for dbt integration tests"
runs:
using: "composite"
steps:
- shell: bash
run: |
sudo systemctl start postgresql.service
pg_isready
sudo -u postgres bash ${{ github.action_path }}/setup_db.sh

View File

@@ -0,0 +1 @@
../../../test/setup_db.sh

View File

@@ -0,0 +1,24 @@
name: "Set up postgres (macos)"
description: "Set up postgres service on macos vm for dbt integration tests"
runs:
using: "composite"
steps:
- shell: bash
run: |
brew services start postgresql
echo "Check PostgreSQL service is running"
i=10
COMMAND='pg_isready'
while [ $i -gt -1 ]; do
if [ $i == 0 ]; then
echo "PostgreSQL service not ready, all attempts exhausted"
exit 1
fi
echo "Check PostgreSQL service status"
eval $COMMAND && break
echo "PostgreSQL service not ready, wait 10 more sec, attempts left: $i"
sleep 10
((i--))
done
createuser -s postgres
bash ${{ github.action_path }}/setup_db.sh

View File

@@ -0,0 +1 @@
../../../test/setup_db.sh

View File

@@ -5,22 +5,8 @@ runs:
steps:
- shell: pwsh
run: |
Write-Host -Object "Installing PostgreSQL 16 as windows service..."
$installerArgs = @("--install_runtimes 0", "--superpassword root", "--enable_acledit 1", "--unattendedmodeui none", "--mode unattended")
$filePath = Invoke-DownloadWithRetry -Url "https://get.enterprisedb.com/postgresql/postgresql-16.1-1-windows-x64.exe" -Path "$env:PGROOT/postgresql-16.1-1-windows-x64.exe"
Start-Process -FilePath $filePath -ArgumentList $installerArgs -Wait -PassThru
Write-Host -Object "Validating PostgreSQL 16 Install..."
Get-Service -Name postgresql*
$pgReady = Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
$exitCode = $pgReady.ExitCode
if ($exitCode -ne 0) {
Write-Host -Object "PostgreSQL is not ready. Exitcode: $exitCode"
exit $exitCode
}
Write-Host -Object "Starting PostgreSQL 16 Service..."
$pgService = Get-Service -Name postgresql-x64-16
$pgService = Get-Service -Name postgresql*
Set-Service -InputObject $pgService -Status running -StartupType automatic
Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
$env:Path += ";$env:PGBIN"
bash ${{ github.action_path }}/setup_db.sh

View File

@@ -1 +1 @@
../../../scripts/setup_db.sh
../../../test/setup_db.sh

View File

@@ -1,169 +0,0 @@
# **what?**
# Runs all tests in dbt-postgres with this branch of dbt-core to ensure nothing is broken
# **why?**
# Ensure dbt-core changes do not break dbt-postgres, as a basic proxy for other adapters
# **when?**
# This will run when trying to merge a PR into main.
# It can also be manually triggered.
# This workflow can be skipped by adding the "Skip Postgres Testing" label to the PR. This is
# useful when making a change in both `dbt-postgres` and `dbt-core` where the changes are dependant
# and cause the other repository to break.
name: "dbt-postgres Tests"
run-name: >-
${{ (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call')
&& format('dbt-postgres@{0} with dbt-core@{1}', inputs.dbt-postgres-ref, inputs.dbt-core-ref)
|| 'dbt-postgres@main with dbt-core branch' }}
on:
push:
branches:
- "main"
- "*.latest"
- "releases/*"
pull_request:
merge_group:
types: [checks_requested]
workflow_dispatch:
inputs:
dbt-postgres-ref:
description: "The branch of dbt-postgres to test against"
default: "main"
dbt-core-ref:
description: "The branch of dbt-core to test against"
default: "main"
workflow_call:
inputs:
dbt-postgres-ref:
description: "The branch of dbt-postgres to test against"
type: string
required: true
default: "main"
dbt-core-ref:
description: "The branch of dbt-core to test against"
type: string
required: true
default: "main"
permissions: read-all
# will cancel previous workflows triggered by the same event
# and for the same ref for PRs/merges or same SHA otherwise
# and for the same inputs on workflow_dispatch or workflow_call
concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ contains(fromJson('["pull_request", "merge_group"]'), github.event_name) && github.event.pull_request.head.ref || github.sha }}-${{ contains(fromJson('["workflow_call", "workflow_dispatch"]'), github.event_name) && github.event.inputs.dbt-postgres-ref && github.event.inputs.dbt-core-ref || github.sha }}
cancel-in-progress: true
defaults:
run:
shell: bash
jobs:
job-prep:
# This allow us to run the workflow on pull_requests as well so we can always run unit tests
# and only run integration tests on merge for time purposes
name: Setup Repo Refs
runs-on: ubuntu-latest
outputs:
dbt-postgres-ref: ${{ steps.core-ref.outputs.ref }}
dbt-core-ref: ${{ steps.common-ref.outputs.ref }}
steps:
- name: "Input Refs"
id: job-inputs
run: |
echo "inputs.dbt-postgres-ref=${{ inputs.dbt-postgres-ref }}"
echo "inputs.dbt-core-ref=${{ inputs.dbt-core-ref }}"
- name: "Determine dbt-postgres ref"
id: core-ref
run: |
if [[ -z "${{ inputs.dbt-postgres-ref }}" ]]; then
REF="main"
else
REF=${{ inputs.dbt-postgres-ref }}
fi
echo "ref=$REF" >> $GITHUB_OUTPUT
- name: "Determine dbt-core ref"
id: common-ref
run: |
if [[ -z "${{ inputs.dbt-core-ref }}" ]]; then
# these will be commits instead of branches
if [[ "${{ github.event_name }}" == "merge_group" ]]; then
REF=${{ github.event.merge_group.head_sha }}
else
REF=${{ github.event.pull_request.base.sha }}
fi
else
REF=${{ inputs.dbt-core-ref }}
fi
echo "ref=$REF" >> $GITHUB_OUTPUT
- name: "Final Refs"
run: |
echo "dbt-postgres-ref=${{ steps.core-ref.outputs.ref }}"
echo "dbt-core-ref=${{ steps.common-ref.outputs.ref }}"
integration-tests-postgres:
name: "dbt-postgres integration tests"
needs: [job-prep]
runs-on: ubuntu-latest
defaults:
run:
working-directory: "./dbt-postgres"
environment:
name: "dbt-postgres"
env:
POSTGRES_TEST_HOST: ${{ vars.POSTGRES_TEST_HOST }}
POSTGRES_TEST_PORT: ${{ vars.POSTGRES_TEST_PORT }}
POSTGRES_TEST_USER: ${{ vars.POSTGRES_TEST_USER }}
POSTGRES_TEST_PASS: ${{ secrets.POSTGRES_TEST_PASS }}
POSTGRES_TEST_DATABASE: ${{ vars.POSTGRES_TEST_DATABASE }}
POSTGRES_TEST_THREADS: ${{ vars.POSTGRES_TEST_THREADS }}
services:
postgres:
image: postgres
env:
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- ${{ vars.POSTGRES_TEST_PORT }}:5432
steps:
- name: "Check out dbt-adapters@${{ needs.job-prep.outputs.dbt-postgres-ref }}"
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # actions/checkout@v4
with:
repository: dbt-labs/dbt-adapters
ref: ${{ needs.job-prep.outputs.dbt-postgres-ref }}
- name: "Set up Python"
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # actions/setup-python@v5
with:
python-version: ${{ inputs.python-version }}
- name: "Set environment variables"
run: |
echo "HATCH_PYTHON=${{ inputs.python-version }}" >> $GITHUB_ENV
echo "PIP_ONLY_BINARY=psycopg2-binary" >> $GITHUB_ENV
- name: "Setup test database"
run: psql -f ./scripts/setup_test_database.sql
env:
PGHOST: ${{ vars.POSTGRES_TEST_HOST }}
PGPORT: ${{ vars.POSTGRES_TEST_PORT }}
PGUSER: postgres
PGPASSWORD: postgres
PGDATABASE: postgres
- name: "Install hatch"
uses: pypa/hatch@257e27e51a6a5616ed08a39a408a21c35c9931bc # pypa/hatch@install
- name: "Run integration tests"
run: hatch run ${{ inputs.hatch-env }}:integration-tests

View File

@@ -11,6 +11,11 @@ updates:
schedule:
interval: "daily"
rebase-strategy: "disabled"
- package-ecosystem: "pip"
directory: "/plugins/postgres"
schedule:
interval: "daily"
rebase-strategy: "disabled"
# docker dependencies
- package-ecosystem: "docker"
@@ -23,10 +28,3 @@ updates:
schedule:
interval: "weekly"
rebase-strategy: "disabled"
# github dependencies
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
rebase-strategy: "disabled"

Some files were not shown because too many files have changed in this diff Show More