Compare commits

...

93 Commits

Author SHA1 Message Date
Gerda Shank
e983ee4a75 Update sqslparse to 0.4.4 2023-06-22 17:37:23 -04:00
FishtownBuildBot
533988233e [Automated] Merged prep-release/1.6.0b5_5349512955 into target main during release process 2023-06-22 15:07:48 -05:00
Peter Webb
8bc0e77a1d CT-2719: Rename the semantic_nodes collection on the manifest to semantic_models (#7927) 2023-06-22 15:59:54 -04:00
Github Build Bot
1c93c9bb58 Bumping version to 1.6.0b5 and generate changelog 2023-06-22 19:27:29 +00:00
Michelle Ark
6d7b32977c Fix: safe remove of external nodes from nodes.depends_on (#7923) 2023-06-22 13:42:06 -04:00
Michelle Ark
bf15466bec UninstalledPackagesFoundError references correct packages specified path (#7886) 2023-06-22 11:40:59 -04:00
Gerda Shank
fb1ebe48f0 Resolve SemanticModel refs in the same way as other refs (#7895) 2023-06-22 11:26:08 -04:00
Peter Webb
de65697ff9 Further Integrate Semantic Models (#7917)
* CT-2651: Add Semantic Models to the manifest and various pieces of graph linking code

* CT-2651: Finish integrating semantic models into the partial parsing system

* CT-2651: More semantic model details for partial parsing

* CT-2651: Remove merged references to project_dependencies

* CT-2651: Revise changelog entry

* CT-2651: Disable unit test until partial parsing of semantic models is complete.

* CT-2651: Temporarily disable an apparently-flaky test.
2023-06-21 18:31:54 -04:00
Michelle Ark
ecf90d689e Refactor/unify public and model nodes (#7891) 2023-06-21 17:12:15 -04:00
Mila Page
4cdeff11cd Remove --config-dir instead of fixing #7774 (#7793)
* Stringfy the dir always to solve the bug.

* Add changelog

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-06-21 11:36:47 -07:00
Gerda Shank
9ff2f6e430 Do not jinja render packages from dependencies yml (#7910)
* Add some comments to methods constructing Project/RuntimeConfig

* Save flag that packages dict came from dependencies.yml

* Test for not rendering packages_dict

* Changie

* Ensure packages_yml_dict and dependencies_yml_dict are dictionaries

* Ensure "packages" passed to render_packages is a dict
2023-06-21 14:33:23 -04:00
Jeremy Cohen
73a0dc6d14 Reorganize, annotate, and revise dependency pins (#7368)
* Reorganize + annotate dependencies

* Loosen pins on other dbt-labs packages

* Minor pins, no patch pins

* Rm support for py37

* Fix protobuf pin

* Bump networkx upper bound #6551
2023-06-21 07:08:43 +02:00
Kshitij Aranke
0a1c73e0fd Fixes #7753: Fix regression in run-operation to not require the name of the package to run (#7811) 2023-06-20 11:19:16 -05:00
Quigley Malcolm
8653ffc5a4 Upgrade core to support dbt-semantic-interfaces 0.1.0dev7 (#7903)
* Bump DSI dependency version to 0.1.0dev7

* Cleaner DSI type enum importing

Previoulsy we had to use individual import paths for each type enum
that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces
has been updated to allow for importing all the type enums from a
singular path.

* Cleaner DSI protocol importing

Previoulsy we had to use individual import paths for each protocol
that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces
has been updated to allow for importing all the protocols from a
singular path.

* Add semantic protocol satisifcation test for metric type params

* Replace `metric.type_params.measures` with `metric.type_params.input_measures`

In DSI 0.1.0dev7 `measures` on metric type params became `input_measures`.
Additionally `input_measures` should not be user specified but something
we compile at parse time, thus we've removed it from `UnparsedMetricTypeParams`.
Finally, actually populating `input_measures` is somewhat complicated due
to the existance of derived metrics, thus that work is being pushed
off to CT-2707.

* Update metric numerator/denominator to be `MetricInput`s

In DSI 0.1.0dev7 `metric.type_params.numerator` and `metric.type_params.denominator`
switched from being `MetricInputMeasure`s to `MetricInput`s. This
commit reflects that change. Additionally, some helper functions on
metric type params were removed related to the numerator and denominator.
Thus we've removed them respectively in this commit.

* Add protocol satisfaction tests for `MetricInput` and `MetricInputMeasure`

* Add `post_aggregation_reference` to `MetricInput` and fix typo in `MetricInputMeasure`

DSI 0.1.0dev7 added `post_aggregation_reference` to the `MetricInput` protocol,
thus we've added it to our implementation in core. Additionally, we had a typo
in a method name in our implementation of `MetricInputMeasure`, ironically
a similar function to the one we've added for `MetricInput`

* Changie doc for upgraded to DSI 0.1.0dev7

* Fix parsing of metric numerator and denominator in schema_yaml_readers

Previously numerator and denominator of a metric were `MetricInputMeasure`s,
now they're `MetricInput`s. Changing the typing isn't enough though.
We have parsing functions in `schema_yaml_readers` which were specifically
parsing the numerator and denominator as if they were `MetricInputMeasure`s.
Thus we had to updating the schema_yaml_readers to parse them as `MetricInput`s.
During this we had some logic in a parsing function `_get_metric_inputs` which
could be abstracted to newly added functions.
2023-06-20 08:08:57 -07:00
Quigley Malcolm
86583a350f Ct-2690 Support dbt-semantic-interfaces 0.1.0dev5 (#7888)
* Upgrade to dbt-semantic-interfaces v0.1.0dev5

This is a fairly simple upgrade. Literally it's just pointing at the
the new versions. The v3 schemas are directly compatible with v5 because
there were no protocol level changes from v3 to v5. All the changers were
updates to tools MetricFlow uses from DSI, not tools that we ourselves
are using in core (yet).

* Add changie doc for DSI version bump
2023-06-16 12:47:55 -07:00
Quigley Malcolm
fafab5d557 [CT-2696] Skip jinia parsing of metric filters (#7885)
* Update metric filters in testing fixtures

I incorrectly wrote the tests such that they didn't include curly
braces, `{{..}}`, around things like `dimension(..)` for filters.
This updates the tests fixtures to have proper filter specifications

* Skip jinja rendering of `filter` key of metrics

Note that `filter` can show up in multiple places: as a root key
on a metric (`metric.filter`), on a metric input (`metric.type_params.metrics[x].filter`),
denominator (`metric.type_params.denominator.filter`), numerator
(`metric.type_params.numerator.filter`), and a metric input measure
(`metric.type_params.measure.filter` and `metric.type_params.measures[x].filter`).
In this commit we skip all of them :)

* Add changie doc for skipping jinja parsing for metric filters

* Update yaml renderer test for metrics
2023-06-15 15:41:31 -07:00
Gerda Shank
39e0c22353 Allow setting packages in dependencies.yml and move dependencies to runtime config (#7857) 2023-06-15 14:41:57 -04:00
colin-rogers-dbt
f767943fb2 Add AdapterRegistered event log message (#7862)
* Add AdapterRegistered event log message

* Add AdapterRegistered to unit test

* make versioning and logging consistent

* make versioning and logging consistent

* add to_version_string

* remove extra equals

* format fire_event
2023-06-14 14:56:47 -07:00
dependabot[bot]
ae97831ebf Bump mypy from 0.981 to 1.0.1 (#7027)
* Bump mypy from 0.981 to 1.0.1

Bumps [mypy](https://github.com/python/mypy) from 0.981 to 1.0.1.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.981...v1.0.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

* upgrade mypy and fix all errors

* fixing some duplicate imports from conflict resolution

* fix mypy errors from merging in main

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-06-14 14:31:57 -05:00
Jeremy Cohen
f16bae0ab9 Fix: dbt show --inline with private models (#7838)
* Add functional test

* Check resource_type before DbtReferenceError

* Changelog entry
2023-06-14 12:23:20 -04:00
FishtownBuildBot
b947b2bc7e [Automated] Merged prep-release/1.6.0b4_5260249343 into target main during release process 2023-06-13 16:40:37 -05:00
Github Build Bot
7068688181 Bumping version to 1.6.0b4 and generate changelog 2023-06-13 20:54:08 +00:00
Quigley Malcolm
38c0600982 Update SemanticModel node to match DSI 0.1.0dev3 protocols (#7848)
* Add tests to ensure our semantic layer nodes satisfy the DSI protocols

These tests create runtime checkable versions of the protocols defined in
DSI. Thus we can instantiate instances of our semantic layer nodes and
use `isinstance` to check that they satisfy the protocol. These `runtime_checkable`
versions of the protocols should only exist in testing and should never
be used in the actual package code.

* Update the `Dimension` object of `SemanticModel` node to match DSI protocol

* Make `UnparsedDimension` more strict and update schema readers accordingly

* Update the `Entity` object of `SemanticModel` node to match DSI protocol

* Make `UnparsedEntity` more strict and update schema readers accordingly

* Update the `Measure` object of `SemanticModel` node to match DSI protocol

* Make `UnparsedMeasure` more strict and update schema readers accordingly

* Update the `SemanticModel` node to match DSI protocol

A lot of the additions are helper functions which we don't actually
use in core. This is a known issue. We're in the process of removing
a fair number of them from the DSI protocol spec. However, in the meantime
we need to implement them to satisfy the protocol unfortunately.

* Make `UnparsedSemanticModel` more strict and update schema readers accordingly

* Changie entry for updating SemanticModel node
2023-06-13 13:25:35 -07:00
Jeremy Cohen
83d163add5 Respect column quote config in model contracts (#7537) 2023-06-13 15:32:56 -04:00
Emily Rockman
d46e8855ef Allow ProjectDependency to have extra fields (#7834)
* Allow ProjectDependency to have extra fields

* changelog
2023-06-13 09:47:50 -05:00
mirnawong1
60524c0f8e update adapters url (#7779)
* update adapters url

in response to [docs.getedbt.com pr 3465](https://github.com/dbt-labs/docs.getdbt.com/issues/3465), updating this error message to point to the correct URL, which was recently changed. 

old URL: https://docs.getdbt.com/docs/supported-data-platforms#adapter-installation
new URL: https://docs.getdbt.com/docs/connect-adapters#install-using-the-cli

thank you @dbeatty10 for your 🦅 👀 !

* adding changie entry

* Update .changes/unreleased/Breaking Changes-20230612-161159.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-06-13 08:51:21 -05:00
Gerda Shank
ca73a2aa15 Use project directory in path selector instead of cwd (#7829)
* Use contextvar to store and get project_root for path selector method

* Changie

* Modify test to check Path selector with project-dir

* Don't set cv_project_root in base task if no config
2023-06-13 09:08:09 -04:00
Emily Rockman
4a833a4272 Move remaining unit tests to /tests directory (#7843)
* moved remaining unit tests to  directory

* fix path

* Delete profiles.yml

* rename remaining test refs
2023-06-13 07:08:28 -05:00
FishtownBuildBot
f9abeca231 Add new index.html and changelog yaml files from dbt-docs (#7836) 2023-06-12 11:55:42 -04:00
Jeremy Cohen
5f9e527768 Rm spaces from NodeType strings (#7842)
* Rm space from NodeType strings

* Add changelog entry
2023-06-12 11:55:07 -04:00
FishtownBuildBot
6f51de4cb5 [Automated] Merged prep-release/1.6.0b3_5215324721 into target main during release process 2023-06-08 15:52:09 -05:00
Github Build Bot
cb64682d33 Bumping version to 1.6.0b3 and generate changelog 2023-06-08 20:15:20 +00:00
Quigley Malcolm
98d1a94b60 Update spec MetricNode for dbt x MetricFlow integration and begin outputting semantic manifest artifact (#7812)
* Refactor MetricNode definition to satisfy DSI Metric protocol

* Fix tests involving metrics to have updated properties

* Update UnparsedMetricNode to match new metric yaml spec

* Update MetricParser for new unparsed and parsed MetricNodes

* Remove `rename_metric_attr`

We're intentionally breaking the spec. There will be a separate tool provided
for migrating from dbt-metrics to dbt x metricflow. This bit of code was renaming
things like `type` to `calculation_method`. This is problematic because `type` is
on the new spec, while `calculation_method` is not. Additionally, since we're
intentionally breaking the spec, this function, `rename_metric_attr`, shouldn't be
used for any property renaming.

* Fix tests for Metrics (1.6) changes

* Regenerated v10 manifest schema and associated functional test artifact state

* Remove no longer needed tests

* Skip / comment out tests for metrics functionality that we'll be implementing later

* Begin outputting semantic manifest artifact on every run

* Drop metrics during upgrade_manifest_json if manifest is v9 or before

* Update properties of `minimal_parsed_metric_dict` to match new metric spec

* Add changie entry for metric node breaking changes

* Add semantic model nodes to semantic manifest
2023-06-08 10:23:36 -07:00
Peter Webb
a89da7ca88 Add SemanticModel Node Type (#7769)
* Add dbt-semantic-interfaces as a dependency

With the integration with MetricFlow we're taking a dependency on
`dbt-semantic-interfaces` which acts as the source of truth for
protocols which MetricFlow and dbt-core need to agree on. Additionally
we're hard pinning to 0.1.0.dev3 for now. We plan on having a less
restrictive specification when dbt-core 1.6 hits GA.

* Add implementations of DSI Metadata protocol to nodes.py

* CT-2521: Initial work on adding new SemanticModel node

* CT-2521: Second rough draft of SemanticModels

* CT-2521: Update schema v10

* CT-2521: Update unit tests for new SemanticModel collection in manifest

* CT-2521: Add changelog entry

* CT-2521: Final touches on initial implementation of SemanticModel parsing

* Change name of Metadata class to reduce potential for confusion

* Remove "Replaceable" inheritance, per review

* CT-2521: Rename internal variables from semantic_models to semantic_nodes

* CT-2521: Update manifest schema to reflect change

---------

Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
2023-06-08 09:39:04 -04:00
Mike Alfare
2d237828ae ADAP-2: Materialized Views (#7239)
* changie

* ADAP-387: Stub materialized view as a materialization (#7211)

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* updating to feature main

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>

* ADAP-387: reverting db_api implementation (#7322)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* updating to feature main

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* added tests

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* typo

* moved materialized view tests into adapter test framework

* add enum to relation for comparison in jinja

---------

Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>

* ADAP-391: Add configuration change option (#7272)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* move up pre-existing drop of backup

* change relation type to view to be consistent

* add base test case

* fix jinja exception message expression, basic test passing

* align names between postgres and default to same convention

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integration tests

* add on_configuration_change to expected_manifest to pass functional tests

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* moved materialized view tests into adapter test framework

* add alter materialized view macro

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* updated default option for on_configuration_change on NodeConfig

* removed explicit calls to enum values

* add test setup for testing fail config option

* updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name

* setup configuration change framework

* skipped tests that are expected to fail without adapter implementation

* cleaned up log checks

---------

Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>

* ADAP-388: Stub materialized view as a materialization - postgres (#7244)

* move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>

* ADAP-402: Add configuration change option - postgres (#7334)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* added materialized view stubs and test

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* updated postgres to use the new macros structure

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* init copy of pr 387 to begin 391 implementation

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* remove postgres-specific materialization in favor of core default materialization

* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies

* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integretion tests

* add on_configuration_change to expected_manifest to pass functuional tests

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* revert to current approach for materializations

* added tests

* move materialized view logic into the `/materializations` directory in line with `dbt-core`

* moved default macros in `dbt-core` into `dbt-postgres`

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* moved materialized view tests into adapter test framework

* updated materialized view tests to use adapter test framework

* add alter materialized view macro

* add alter materialized view macro

* change class name, and config setup

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* added in `on_configuration_change` setting

* updated default option for on_configuration_change on NodeConfig

* updated default option for on_configuration_change on NodeConfig

* fixed list being passed as string bug

* removed explicit calls to enum values

* removed unneeded test class

* fixed on_configuration_change to be picked up appropriately

* add test setup for testing fail config option

* remove breakpoint, uncomment tests

* update skip scenario to use empty strings

* update skip scenario to avoid using sql at all, remove extra whitespace in some templates

* push up initial addition of indexes for mv macro

* push slight change up

* reverting alt macro and moving the do create_index call to be more in line with other materializations

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402

# Conflicts:
#	core/dbt/contracts/graph/model_config.py
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/base.py
#	tests/functional/materializations/test_materialized_view.py

* merge feature branch into story branch

* merge feature branch into story branch

* added indexes into the workflow

* fix error in jinja that caused print error

* working on test messaging and skipping tests that might not fit quite into current system

* add drop and show macros for indexes

* add drop and show macros for indexes

* add logic to determine the indexes to create or drop

* pulled index updates through the workflow properly

* convert configuration changes to fixtures, implement index changes into tests

* created Model dataclass for readability, added column to swap index columns for testing

* fixed typo

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>

* ADAP-395: Implement native materialized view DDL (#7336)

* changie

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* added materialized view stubs and test

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* updated postgres to use the new macros structure

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* init copy of pr 387 to begin 391 implementation

* updating to feature main

* updating to feature main

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* remove postgres-specific materialization in favor of core default materialization

* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies

* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integretion tests

* add on_configuration_change to expected_manifest to pass functuional tests

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* revert to current approach for materializations

* added tests

* move materialized view logic into the `/materializations` directory in line with `dbt-core`

* moved default macros in `dbt-core` into `dbt-postgres`

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* typo

* moved materialized view tests into adapter test framework

* updated materialized view tests to use adapter test framework

* add alter materialized view macro

* add alter materialized view macro

* added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update

* change class name, and config setup

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* added in `on_configuration_change` setting

* updated default option for on_configuration_change on NodeConfig

* updated default option for on_configuration_change on NodeConfig

* fixed list being passed as string bug

* fixed list being passed as string bug

* removed explicit calls to enum values

* removed explicit calls to enum values

* removed unneeded test class

* fixed on_configuration_change to be picked up appropriately

* add test setup for testing fail config option

* remove breakpoint, uncomment tests

* update skip scenario to use empty strings

* update skip scenario to avoid using sql at all, remove extra whitespace in some templates

* push up initial addition of indexes for mv macro

* push slight change up

* reverting alt macro and moving the do create_index call to be more in line with other materializations

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402

# Conflicts:
#	core/dbt/contracts/graph/model_config.py
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/base.py
#	tests/functional/materializations/test_materialized_view.py

* merge feature branch into story branch

* merge feature branch into story branch

* added indexes into the workflow

* fix error in jinja that caused print error

* working on test messaging and skipping tests that might not fit quite into current system

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395

# Conflicts:
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	plugins/postgres/dbt/include/postgres/macros/adapters.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py
#	tests/functional/materializations/test_materialized_view.py

* moved postgres implemention into plugin directory

* update index methods to align with the configuration update macro

* added native ddl to postgres macros

* removed extra docstring

* updated references to View, now references MaterializedView

* decomposed materialization into macros

* refactor index create statement parser, add exceptions for unexpected formats

* swapped conditional to check for positive state

* removed skipped test now that materialized view is being used

* return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test

* add check for indexes in particular for apply on configuration scenario

* removed extra argument

* add materialized views to get_relations / list_relations

* typos in index change logic

* moved full refresh check inside the build sql step

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>

* removing returns from tests to stop logs from printing

* moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures

* fixed overwrite issue, simplified assertion method

* updated import order to standard

* fixed test import paths

* updated naming convention for proper test collection with the test runner

* still trying to make the test runner happy

* rewrite index updates to use a better source in Postgres

* break out a large test suite as a separate run

* update `skip` and `fail` scenarios with more descriptive results

* typo

* removed call to skip status

* reverting `exceptions_jinja.py`

* added FailFastError back, the right way

* removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports

* removed assumed models in method calls, removed odd insert records and replaced with get row count

* fixed index issue, removed some indirection in testing

* made test more readable

* remove the "apply" from the tests and put it on the base as the default

* generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility

* fixed type to be more generic to accommodate adapters with their own relation types

* fixed all the broken index stuff

* updated on_configuration_change to use existing patterns

* updated on_configuration_change to use existing patterns

* reflected update in tests and materialization logic

* reflected update in tests and materialization logic

* reverted the change to create a config object from the option object, using just the option object now

* reverted the change to create a config object from the option object, using just the option object now

* modelled database objects to support monitoring all configuration changes

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* reverted centralized framework, retained a few reusable base classes

* updated names to be more consistent

* readability updates

* added readme specifying that `relation_configs` only supports materialized views for now

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
2023-06-07 19:19:09 -04:00
Michelle Ark
f4253da72a fix: removing dependency from dependencies.yml (#7743) 2023-06-07 13:48:39 -04:00
Mila Page
919822e583 Adap 496/add test connection mode to debug (#7741)
* --connection-flag

* Standardize the plugin functions used by DebugTask

* Cleanup redundant code and help logic along.

* Add more output tests to add logic coverage and formatting.

* Code review

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-06-07 09:56:35 -07:00
Michelle Ark
444c787729 fix error message for empty/None: --warn-error-options handling (#7735) 2023-06-07 12:23:40 -04:00
Michelle Ark
3b63dd9f11 Validate public models are not materialized as ephemeral (#7794) 2023-06-07 12:22:57 -04:00
Kshitij Aranke
84166bf457 Fixes #7551: Create add_from_artifact to populate state_relation field of nodes (#7796) 2023-06-06 15:25:12 -07:00
Michelle Ark
dd445e1fde generalize BaseModelConstraintsRuntimeEnforcement (#7805) 2023-06-06 16:30:50 -04:00
Michelle Ark
6a22ec1b2e Package-namespaced generate_x_name macro resolution (#7509) 2023-06-06 15:05:44 -04:00
Emily Rockman
587bbcbf0d Improve warnings for constraints and materialization types (#7696)
* first pass

* debugging

* regen proto types

* refactor to use warn_supported flag

* PR feedback
2023-06-06 12:50:58 -05:00
Doug Beatty
8e1c4ec116 Fix not equals comparison to be null-safe for adapters/utils tests (#7776)
* Fix names within functional test

* Changelog entry

* Test for implementation of null-safe equals comparison

* Remove duplicated where filter

* Fix null-safe equals comparison

* Fix tests for `concat` and `hash` by using empty strings () instead of `null`

* Remove macro namespace interpolation
2023-06-06 06:11:48 -06:00
Kshitij Aranke
dc35f56baa Fixes #7299: dbt retry (#7763) 2023-06-05 15:51:00 -07:00
Michelle Ark
60d116b5b5 log PublicationArtifactAvailable even when partially parsing & public models unchanged (#7783) 2023-06-05 14:59:38 -04:00
Emily Rockman
4dbc4a41c4 remove entire changes folder (#7766)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-06-05 09:51:30 -05:00
Michelle Ark
89541faec9 force dependency between test models (#7767) 2023-06-05 10:45:59 -04:00
Gerda Shank
79bd98560b Version 0 for model works for latest_version (#7712) 2023-06-05 10:21:39 -04:00
Michelle Ark
7917bd5033 add project_name to manifest metadata (#7754) 2023-06-02 16:16:14 -04:00
Michelle Ark
05b0ebb184 Fix constraint rendering for expressions and foreign key constraint types (#7512) 2023-06-02 15:05:09 -04:00
Quazi Irfan
e1d7a53325 Fix doc link in selector.py (#7755)
* Fix doc link in selector.py

* Ran changie to modify changelog entry
2023-06-02 09:00:58 -05:00
Michelle Ark
7a06d354aa pass optional sql_header to empty subquery sql rendering (#7734) 2023-06-01 14:08:02 -04:00
dave-connors-3
9dd5ab90bf add ability to select models by access (#7739)
* add ability to select models by access

* changie

* Update core/dbt/graph/selector_methods.py
2023-06-01 09:25:59 -04:00
Michelle Ark
45d614533f fix StopIteration error when publication not found (#7710) 2023-05-30 16:50:36 -04:00
Peter Webb
00a531d9d6 Template rendering optimization (#7451)
* CT-2478: Template rendering optimization

* CT-2478: Fix type annotation, and accomodate non-string unit test cases.
2023-05-30 12:48:47 -04:00
Sam Debruyn
fd301a38db Dropped support for Python 3.7 (#7623) 2023-05-30 12:12:57 -04:00
Jeremy Cohen
9c7e01dbca Readd exp_path for config deprecation warnings (#7536) 2023-05-30 12:04:49 -04:00
github-actions[bot]
1ac6df0996 Adding performance modeling for 1.2.0 to refs/heads/main (#7560)
* adding performance baseline for 1.2.0

* Adding newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-27 22:10:38 -04:00
Gerda Shank
38ca4fce25 Target path should be relative to project dir, rather than current working directory (#7706) 2023-05-26 18:50:38 -04:00
Kshitij Aranke
7e3a6eec96 fix #7300: Enable state for deferral to be separate from state for selectors (#7690) 2023-05-26 13:00:16 -07:00
Emily Rockman
ac16a55c64 Update to reusable workflow for branch testing (#7676)
* fix overlooked node12 case with abandonded marketplace action

* update slack notification

* remove spaces per formatting

* replace with cli dispatch

* move conditional

* add explicit token, temp comment out slack

* add checkout

* checkout teh right branch

* switch to PAT

* add back repo checkout

* manually check workflow status

* fix notification

* swap to reusable workflow

* fix path

* swap permissions

* remove trigger

* fix secrets

* point to main
2023-05-26 14:51:00 -05:00
Doug Beatty
620ca40b85 Add % to adapter suite test cases for persist_docs (#7699)
* Test table/view/column-level comments with `%` symbol

* Test docs block with `%` symbol

* Changelog entry
2023-05-26 12:48:08 -04:00
leahwicz
aa11cf2956 Adding link to 1.5 release notes (#7707) 2023-05-26 08:38:13 -04:00
FishtownBuildBot
feb06e2107 [Automated] Merged prep-release/1.6.0b2_5081502847 into target main during release process 2023-05-25 10:41:03 -05:00
Github Build Bot
a3d40e0abf Bumping version to 1.6.0b2 and generate changelog 2023-05-25 15:06:24 +00:00
Gerda Shank
7c1bd91d0a CT 2590 write pub artifact to log (#7686) 2023-05-24 13:54:58 -04:00
leahwicz
70a132d059 Updating CODEOWNERS to consolidate (#7693)
Consolidating to remove Language and Execution and instead default to the `core-team`
2023-05-24 09:12:25 -04:00
Anis Nasir
1fdebc660b Relaxed the pyyaml dependency to >=5.3. (#7681) 2023-05-24 08:41:38 -04:00
Gerda Shank
0516192d69 CT 2516 ensure that paths in Jinja context flags object are strings (#7678) 2023-05-24 08:24:36 -04:00
github-actions[bot]
f99be58217 Adding performance modeling for 1.4.6 to refs/heads/main (#7532)
* adding performance baseline for 1.4.6

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-24 08:17:19 -04:00
github-actions[bot]
3b6222e516 Adding performance modeling for 1.3.0 to refs/heads/main (#7530)
* adding performance baseline for 1.3.0

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-24 08:16:30 -04:00
github-actions[bot]
b88e60f8dd Adding performance modeling for 1.4.1 to refs/heads/main (#7527)
* adding performance baseline for 1.4.1

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-24 08:16:02 -04:00
github-actions[bot]
9373c4d1e4 Adding performance modeling for 1.3.4 to refs/heads/main (#7525)
* adding performance baseline for 1.3.4

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-24 08:15:26 -04:00
dependabot[bot]
0fe3ee8eca Bump ubuntu from 23.04 to 23.10 (#7675)
* Bump ubuntu from 23.04 to 23.10

Bumps ubuntu from 23.04 to 23.10.

---
updated-dependencies:
- dependency-name: ubuntu
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-05-24 08:01:10 -04:00
Sam Debruyn
0d71a32aa2 Include null checks in utils test base (#7672)
* Include null checks in utils test base

* Add tests for the schema test

* Add tests for this macro

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-05-23 19:49:48 -07:00
Doug Beatty
0f223663bb Honor --skip-profile-setup parameter when inside an existing project (#7609)
* Honor `--skip-profile-setup` parameter when inside an existing project

* Use project name as the profile name

* Use separate file connections for reading and writing

* Raise a custom exception when no adapters are installed

* Test skipping interactive profile setup when inside a dbt project

* Replace `assert_not_called()` since it does not work

* Verbose CLI argument for skipping profile setup

* Use separate file connections for reading and writing

* Check empty list in a Pythonic manner
2023-05-23 17:56:54 -06:00
Kshitij Aranke
c25d0c9f9c fix #7502: write run_results.json for run operation (#7655) 2023-05-23 14:56:23 -07:00
Peter Webb
4a4b7beeb9 Model Deprecation (#7562)
* CT-2461: Work toward model deprecation

* CT-2461: Remove unneeded conversions

* CT-2461: Fix up unit tests for new fields, correct a couple oversights

* CT-2461: Remaining implementation and tests for model/ref deprecation warnings

* CT-2461: Changelog entry for deprecation warnings

* CT-2461: Refine datetime handling and tests

* CT-2461: Fix up unit test data

* CT-2461: Fix some more unit test data.

* CT-2461: Fix merge issues

* CT-2461: Code review items.

* CT-2461: Improve version -> str conversion
2023-05-23 09:30:32 -04:00
Ian Knox
265e09dc93 Remove DelayedFileHandler (#7661)
* remove DelayedFileHandler

* Changelog

* set_path to no-op

* more no-ops for rpc

* Clearer comments
2023-05-22 16:42:48 -04:00
Mike Alfare
87ea28fe84 break out a large test suite as a separate execution to avoid memory issues on windows CI runs (#7669) 2023-05-19 17:04:35 -04:00
Michelle Ark
af0f786f2e Accept PublicationArtifacts in dbtRunner.invoke (#7656) 2023-05-18 16:42:50 -04:00
David Bloss
50528a009d update used gh actions ahead of node12 deprecation (#7651)
* update used gh actions ahead of node12 deprecation

* replace with valid tag

---------

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-05-17 16:20:26 -05:00
Stu Kilgore
f6e5582370 Add "other" relation to reffable node classes (#7645) 2023-05-17 12:14:16 -05:00
Peter Webb
dea3181d96 Exclude some profile fields from Jinja rendering when they are not valid Jinja. (#7630)
* CT-2583: Exclude some profile fields from Jinja rendering.

* CT-2583: Add functional test.

* CT-2583: Change approach to password jinja detection

* CT-2583: Extract string constant and add additional checks

* CT-2583: Improve unit test coverage
2023-05-17 11:38:48 -04:00
Gerda Shank
5f7ae2fd4c Move node patch method to schema parser patch_node_properties and refactor schema parsing (#7640) 2023-05-16 21:08:04 -04:00
Daniel Reeves
4f249b8652 Add target_path to more cli commands that use it (#7647) 2023-05-16 16:22:59 -05:00
Ian Knox
df23f68dd4 Missed PR fedback (#7642) 2023-05-16 13:50:28 -05:00
Stu Kilgore
4b091cee9e Instantiate Flags class from dict and command name (#7624) 2023-05-16 13:31:23 -05:00
Ian Knox
dcb5acdf29 bugfix: Deps hangs when using relative paths via --project-dir (#7628) 2023-05-16 10:00:23 -05:00
Emily Rockman
7fbeced315 updates for github deprecations (#7614)
* updates for github deprecations

* fix jira file name

* swap out abandonded action

* add quotes to env var

* revert main.yml
2023-05-15 15:28:52 -05:00
Mike Alfare
47e7b1cc80 Feature/drop relation/ct 2581 (#7626)
* changie
* move drop_relation macros into their own file, add scenarios for table, view, and materialized view
2023-05-15 15:51:19 -04:00
FishtownBuildBot
8f998c218e [Automated] Merged prep-release/1.6.0b1_4961250999 into target main during release process 2023-05-12 12:37:41 -05:00
330 changed files with 10652 additions and 4704 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.6.0b1
current_version = 1.6.0b5
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

View File

@@ -3,6 +3,7 @@
For information on prior major and minor releases, see their changelogs:
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)

38
.changes/1.6.0-b2.md Normal file
View File

@@ -0,0 +1,38 @@
## dbt-core 1.6.0-b2 - May 25, 2023
### Features
- Added warnings for model and ref deprecations ([#7433](https://github.com/dbt-labs/dbt-core/issues/7433))
- Update drop_relation macro to allow for configuration of drop statement separately from object name ([#7625](https://github.com/dbt-labs/dbt-core/issues/7625))
- accept publications in dbt.invoke ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Fixes
- Honor `--skip-profile-setup` parameter when inside an existing project ([#7594](https://github.com/dbt-labs/dbt-core/issues/7594))
- Fix: Relative project paths weren't working with deps ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491))
- Exclude password fields from Jinja rendering. ([#7629](https://github.com/dbt-labs/dbt-core/issues/7629))
- Add --target-path to more CLI subcommands ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- Stringify flag paths for Jinja context ([#7495](https://github.com/dbt-labs/dbt-core/issues/7495))
- write run_results.json for run operation ([#7502](https://github.com/dbt-labs/dbt-core/issues/7502))
### Under the Hood
- Add ability to instantiate Flags class from dict ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607))
- Add other relation to reffable nodes ([#7550](https://github.com/dbt-labs/dbt-core/issues/7550))
- Move node patch method to schema parser patch_node_properties and refactor schema parsing ([#7430](https://github.com/dbt-labs/dbt-core/issues/7430))
- Remove legacy file logger code ([#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
- Break up integration tests as a short term fix for Windows CI runs ([#7668](https://github.com/dbt-labs/dbt-core/issues/7668))
- Include null checks in utils test base ([#7670](https://github.com/dbt-labs/dbt-core/issues/7670))
- Write pub artifact to log ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Dependencies
- Bump ubuntu from 23.04 to 23.10 ([#7675](https://github.com/dbt-labs/dbt-core/pull/7675))
- ([#7681](https://github.com/dbt-labs/dbt-core/pull/7681))
### Contributors
- [@dradnan89@hotmail.com](https://github.com/dradnan89@hotmail.com) ([#7681](https://github.com/dbt-labs/dbt-core/pull/7681))
- [@dwreeves](https://github.com/dwreeves) ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- [@iknox-fa](https://github.com/iknox-fa) ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491), [#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
- [@sdebruyn](https://github.com/sdebruyn) ([#7670](https://github.com/dbt-labs/dbt-core/issues/7670))
- [@stu-k](https://github.com/stu-k) ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607), [#7550](https://github.com/dbt-labs/dbt-core/issues/7550))

46
.changes/1.6.0-b3.md Normal file
View File

@@ -0,0 +1,46 @@
## dbt-core 1.6.0-b3 - June 08, 2023
### Breaking Changes
- Dropped support for Python 3.7 ([#7082](https://github.com/dbt-labs/dbt-core/issues/7082))
- Switch from dbt-metrics to dbt-semantic-interfaces for MetricNode definitions ([#7500](https://github.com/dbt-labs/dbt-core/issues/7500), [#7404](https://github.com/dbt-labs/dbt-core/issues/7404))
### Features
- Add support for materialized views ([#6911](https://github.com/dbt-labs/dbt-core/issues/6911))
- Optimize template rendering for common parse scenarios ([#7449](https://github.com/dbt-labs/dbt-core/issues/7449))
- nodes in packages respect custom generate_alias_name, generate_schema_name, generate_database_name macro overrides defined in packages ([#7444](https://github.com/dbt-labs/dbt-core/issues/7444))
- Enable state for deferral to be separate from state for selectors ([#7300](https://github.com/dbt-labs/dbt-core/issues/7300))
- add access selection syntax ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- add project_name to manifest metadata ([#7752](https://github.com/dbt-labs/dbt-core/issues/7752))
- dbt retry ([#7299](https://github.com/dbt-labs/dbt-core/issues/7299))
- Revamp debug, add --connection flag. Prepare for future refactors/interface changes. ([#7104](https://github.com/dbt-labs/dbt-core/issues/7104))
- Validate public models are not materialized as ephemeral ([#7226](https://github.com/dbt-labs/dbt-core/issues/7226))
- Added support for parsing and serializaing semantic models ([#7499](https://github.com/dbt-labs/dbt-core/issues/7499), [#7503](https://github.com/dbt-labs/dbt-core/issues/7503))
### Fixes
- Constraint rendering fixes: wrap check expression in parentheses, foreign key 'references', support expression in all constraint types ([#7417](https://github.com/dbt-labs/dbt-core/issues/7417), [#7480](https://github.com/dbt-labs/dbt-core/issues/7480), [#7416](https://github.com/dbt-labs/dbt-core/issues/7416))
- Fix warning messages for deprecated dbt_project.yml configs ([#7424](https://github.com/dbt-labs/dbt-core/issues/7424))
- Add `%` to adapter suite test cases for `persist_docs` ([#7698](https://github.com/dbt-labs/dbt-core/issues/7698))
- Improve warnings for constraints and materialization types ([#7335](https://github.com/dbt-labs/dbt-core/issues/7335))
- Incorrect paths used for "target" and "state" directories ([#7465](https://github.com/dbt-labs/dbt-core/issues/7465))
- fix StopIteration error when publication for project not found ([#7711](https://github.com/dbt-labs/dbt-core/issues/7711))
- Using version 0 works when resolving single model ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
- Fix empty --warn-error-options error message ([#7730](https://github.com/dbt-labs/dbt-core/issues/7730))
- send sql header on contract enforcement ([#7714](https://github.com/dbt-labs/dbt-core/issues/7714))
- Fixed doc link in selector.py ([#7533](https://github.com/dbt-labs/dbt-core/issues/7533))
- Fix null-safe equals comparison via `equals` ([#7778](https://github.com/dbt-labs/dbt-core/issues/7778))
- Log PublicationArtifactAvailable even when partially parsing unchanged public models ([#7782](https://github.com/dbt-labs/dbt-core/issues/7782))
- fix RuntimeError when removing project dependency from dependencies.yml ([#7743](https://github.com/dbt-labs/dbt-core/issues/7743))
### Under the Hood
- Fix flaky test for --fail-fast ([#7744](https://github.com/dbt-labs/dbt-core/issues/7744))
- Create `add_from_artifact` to populate `state_relation` field of nodes ([#7551](https://github.com/dbt-labs/dbt-core/issues/7551))
### Contributors
- [@dave-connors-3](https://github.com/dave-connors-3) ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- [@quazi-irfan](https://github.com/quazi-irfan) ([#7533](https://github.com/dbt-labs/dbt-core/issues/7533))
- [@sdebruyn](https://github.com/sdebruyn) ([#7082](https://github.com/dbt-labs/dbt-core/issues/7082))
- [@stu-k](https://github.com/stu-k) ([#7299](https://github.com/dbt-labs/dbt-core/issues/7299), [#7551](https://github.com/dbt-labs/dbt-core/issues/7551))

21
.changes/1.6.0-b4.md Normal file
View File

@@ -0,0 +1,21 @@
## dbt-core 1.6.0-b4 - June 13, 2023
### Fixes
- Respect column 'quote' config in model contracts ([#7370](https://github.com/dbt-labs/dbt-core/issues/7370))
- Fix path selector when using project-dir ([#7819](https://github.com/dbt-labs/dbt-core/issues/7819))
- Allow project dependencies to use miscellaneous keys ([#7497](https://github.com/dbt-labs/dbt-core/issues/7497))
- Updating this error message to point to the correct URL ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))
- Update SemanticModel node to properly impelment the DSI 0.1.0dev3 SemanticModel protocol spec ([#7833](https://github.com/dbt-labs/dbt-core/issues/7833), [#7827](https://github.com/dbt-labs/dbt-core/issues/7827))
### Docs
- Fix for column tests not rendering on quoted columns ([dbt-docs/#201](https://github.com/dbt-labs/dbt-docs/issues/201))
### Under the Hood
- Rm space from NodeType strings ([#7841](https://github.com/dbt-labs/dbt-core/issues/7841))
### Contributors
- [@drewbanin](https://github.com/drewbanin) ([#201](https://github.com/dbt-labs/dbt-core/issues/201))
- [@mirnawong1](https://github.com/mirnawong1) ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))

28
.changes/1.6.0-b5.md Normal file
View File

@@ -0,0 +1,28 @@
## dbt-core 1.6.0-b5 - June 22, 2023
### Features
- Enable setting packages in dependencies.yml ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372), [#7736](https://github.com/dbt-labs/dbt-core/issues/7736))
- Add AdapterRegistered event log message ([#7038](https://github.com/dbt-labs/dbt-core/issues/7038))
- Further integrate semantic models into the DAG and partial parsing module ([#7800](https://github.com/dbt-labs/dbt-core/issues/7800))
### Fixes
- Fix regression in `run-operation` to not require the name of the package to run ([#7753](https://github.com/dbt-labs/dbt-core/issues/7753))
- Allow dbt show --inline preview of private models ([#7837](https://github.com/dbt-labs/dbt-core/issues/7837))
- Skip jinja parsing of metric filters ([#7864](https://github.com/dbt-labs/dbt-core/issues/7864))
- Fix a bad implicit string conversion regression in debug --config-dir code. ([#7774](https://github.com/dbt-labs/dbt-core/issues/7774))
- Fix UninstalledPackagesFoundError error message to use correct packages specified path ([#7921](https://github.com/dbt-labs/dbt-core/issues/7921))
- Fix: safe remove of external nodes from nodes.depends_on ([#7924](https://github.com/dbt-labs/dbt-core/issues/7924))
### Under the Hood
- Upgrade to dbt-semantic-interfaces v0.1.0dev5 ([#7853](https://github.com/dbt-labs/dbt-core/issues/7853))
- Refactoring: consolidating public_nodes and nodes ([#7890](https://github.com/dbt-labs/dbt-core/issues/7890))
- Resolve SemanticModel ref is the same way as other refs ([#7822](https://github.com/dbt-labs/dbt-core/issues/7822))
- Move from dbt-semantic-intefaces 0.1.0dev5 to 0.1.0dev7 ([#7898](https://github.com/dbt-labs/dbt-core/issues/7898))
- Don't jinja render packages from dependencies.yml ([#7905](https://github.com/dbt-labs/dbt-core/issues/7905))
### Dependencies
- Bump mypy from 0.981 to 1.0.1 ([#7027](https://github.com/dbt-labs/dbt-core/pull/7027))

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Dropped support for Python 3.7
time: 2023-05-15T05:31:48.375649+02:00
custom:
Author: sdebruyn
Issue: "7082"

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Switch from dbt-metrics to dbt-semantic-interfaces for MetricNode definitions
time: 2023-06-07T19:03:09.680189-07:00
custom:
Author: QMalcolm
Issue: 7500 7404

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Updating this error message to point to the correct URL
time: 2023-06-12T16:11:59.181953+01:00
custom:
Author: mirnawong1
Issue: "7789"

View File

@@ -0,0 +1,6 @@
kind: "Dependencies"
body: "Bump mypy from 0.981 to 1.0.1"
time: 2023-02-22T16:28:07.00000Z
custom:
Author: dependabot[bot]
PR: 7027

View File

@@ -0,0 +1,6 @@
kind: "Dependencies"
body: "Bump ubuntu from 23.04 to 23.10"
time: 2023-05-22T00:59:48.00000Z
custom:
Author: dependabot[bot]
PR: 7675

View File

@@ -0,0 +1,5 @@
kind: Dependencies
time: 2023-05-22T21:22:01.336704+02:00
custom:
Author: dradnan89@hotmail.com
PR: "7681"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Fix for column tests not rendering on quoted columns
time: 2023-05-31T11:54:19.687363-04:00
custom:
Author: drewbanin
Issue: "201"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add support for materialized views
time: 2023-03-29T12:03:13.862041-04:00
custom:
Author: mikealfare McKnight-42
Issue: "6911"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Optimize template rendering for common parse scenarios
time: 2023-04-24T16:36:11.24088-04:00
custom:
Author: peterallenwebb
Issue: "7449"

View File

@@ -0,0 +1,7 @@
kind: Features
body: nodes in packages respect custom generate_alias_name, generate_schema_name,
generate_database_name macro overrides defined in packages
time: 2023-05-09T21:29:35.557761-04:00
custom:
Author: michelleark
Issue: "7444"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Added warnings for model and ref deprecations
time: 2023-05-09T23:33:29.679333-04:00
custom:
Author: peterallenwebb
Issue: "7433"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Update drop_relation macro to allow for configuration of drop statement separately
from object name
time: 2023-05-15T12:23:04.177141-04:00
custom:
Author: mikealfare
Issue: "7625"

View File

@@ -0,0 +1,6 @@
kind: Features
body: accept publications in dbt.invoke
time: 2023-05-17T18:56:27.585784-04:00
custom:
Author: MichelleArk
Issue: "7372"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Enable state for deferral to be separate from state for selectors
time: 2023-05-23T22:59:55.920975-07:00
custom:
Author: aranke
Issue: "7300"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add access selection syntax
time: 2023-05-30T16:48:47.740037-05:00
custom:
Author: dave-connors-3
Issue: "7738"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add project_name to manifest metadata
time: 2023-06-01T13:22:23.259448-04:00
custom:
Author: michelleark jtcohen6
Issue: "7752"

View File

@@ -0,0 +1,6 @@
kind: Features
body: dbt retry
time: 2023-06-02T08:33:02.410456-07:00
custom:
Author: stu-k aranke
Issue: "7299"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Revamp debug, add --connection flag. Prepare for future refactors/interface changes.
time: 2023-06-04T02:59:56.28283-07:00
custom:
Author: versusfacit
Issue: 7104

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix null-safe equals comparison via `equals`
time: 2023-06-04T08:00:52.537967-06:00
custom:
Author: dbeatty10
Issue: "7778"

View File

@@ -0,0 +1,6 @@
kind: Features
body: ' Validate public models are not materialized as ephemeral'
time: 2023-06-05T22:20:39.382019-04:00
custom:
Author: michelleark
Issue: "7226"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Added support for parsing and serializaing semantic models
time: 2023-06-06T16:53:51.117429-04:00
custom:
Author: peterallenwebb
Issue: 7499 7503

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add AdapterRegistered event log message
time: 2023-06-13T15:15:07.367371-07:00
custom:
Author: colin-rogers-dbt
Issue: "7038"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Enable setting packages in dependencies.yml
time: 2023-06-13T15:19:17.199297-04:00
custom:
Author: gshank
Issue: 7372 7736

View File

@@ -0,0 +1,6 @@
kind: Features
body: Further integrate semantic models into the DAG and partial parsing module
time: 2023-06-16T16:30:45.513314-04:00
custom:
Author: peterallenwebb
Issue: "7800"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: 'Constraint rendering fixes: wrap check expression in parentheses, foreign key
''references'', support expression in all constraint types'
time: 2023-05-04T14:06:42.545193-04:00
custom:
Author: MichelleArk
Issue: 7417 7480 7416

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix warning messages for deprecated dbt_project.yml configs
time: 2023-05-06T18:09:00.361961+02:00
custom:
Author: jtcohen6
Issue: "7424"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Respect column 'quote' config in model contracts
time: 2023-05-06T19:18:13.351819+02:00
custom:
Author: jtcohen6
Issue: "7370"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Honor `--skip-profile-setup` parameter when inside an existing project
time: 2023-05-11T14:04:41.382181-06:00
custom:
Author: dbeatty10
Issue: "7594"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Fix: Relative project paths weren''t working with deps'
time: 2023-05-15T12:36:54.807413-05:00
custom:
Author: iknox-fa
Issue: "7491"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Exclude password fields from Jinja rendering.
time: 2023-05-15T14:28:51.400321-04:00
custom:
Author: peterallenwebb
Issue: "7629"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add --target-path to more CLI subcommands
time: 2023-05-16T15:26:44.557072-04:00
custom:
Author: dwreeves
Issue: "7646"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: write run_results.json for run operation
time: 2023-05-22T13:29:24.182612-07:00
custom:
Author: aranke
Issue: "7502"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Stringify flag paths for Jinja context
time: 2023-05-22T13:50:07.897354-04:00
custom:
Author: gshank
Issue: "7495"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add `%` to adapter suite test cases for `persist_docs`
time: 2023-05-24T16:06:48.477708-06:00
custom:
Author: dbeatty10
Issue: "7698"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Improve warnings for constraints and materialization types
time: 2023-05-25T07:36:51.855641-05:00
custom:
Author: emmyoop
Issue: "7335"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Incorrect paths used for "target" and "state" directories
time: 2023-05-25T16:50:53.718564-04:00
custom:
Author: gshank
Issue: "7465"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix StopIteration error when publication for project not found
time: 2023-05-26T15:37:38.952939-04:00
custom:
Author: michelleark
Issue: "7711"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Using version 0 works when resolving single model
time: 2023-05-26T16:47:27.6065-04:00
custom:
Author: gshank
Issue: "7372"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix empty --warn-error-options error message
time: 2023-05-30T10:42:28.382804-04:00
custom:
Author: michelleark
Issue: "7730"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: send sql header on contract enforcement
time: 2023-05-31T13:19:19.801391-04:00
custom:
Author: michelleark
Issue: "7714"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fixed doc link in selector.py
time: 2023-06-01T13:05:49.952-05:00
custom:
Author: quazi-irfan
Issue: "7533"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Log PublicationArtifactAvailable even when partially parsing unchanged public
models
time: 2023-06-05T12:11:27.739183-04:00
custom:
Author: michelleark
Issue: "7782"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix RuntimeError when removing project dependency from dependencies.yml
time: 2023-06-05T12:44:25.978022-04:00
custom:
Author: michelleark
Issue: "7743"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Fix regression in `run-operation` to not require the name of the package to
run
time: 2023-06-06T14:52:17.38538-07:00
custom:
Author: aranke
Issue: "7753"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix path selector when using project-dir
time: 2023-06-08T13:59:52.95775-04:00
custom:
Author: gshank
Issue: "7819"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Allow project dependencies to use miscellaneous keys
time: 2023-06-09T12:19:30.469487-05:00
custom:
Author: emmyoop
Issue: "7497"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Allow dbt show --inline preview of private models
time: 2023-06-09T19:15:46.716379-04:00
custom:
Author: jtcohen6
Issue: "7837"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Update SemanticModel node to properly impelment the DSI 0.1.0dev3 SemanticModel
protocol spec
time: 2023-06-12T17:58:54.289704-07:00
custom:
Author: QMalcolm
Issue: 7833 7827

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Skip jinja parsing of metric filters
time: 2023-06-15T14:29:49.900201-07:00
custom:
Author: QMalcolm
Issue: "7864"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix a bad implicit string conversion regression in debug --config-dir code.
time: 2023-06-21T03:07:33.815966-07:00
custom:
Author: versusfacit
Issue: "7774"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Fix UninstalledPackagesFoundError error message to use correct packages specified
path
time: 2023-06-22T10:58:29.576179-04:00
custom:
Author: michelleark
Issue: "7921"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Fix: safe remove of external nodes from nodes.depends_on'
time: 2023-06-22T12:19:07.657855-04:00
custom:
Author: michelleark
Issue: "7924"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add ability to instantiate Flags class from dict
time: 2023-05-15T09:51:16.291352-05:00
custom:
Author: stu-k
Issue: "7607"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add other relation to reffable nodes
time: 2023-05-15T15:21:07.808892-05:00
custom:
Author: stu-k
Issue: "7550"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move node patch method to schema parser patch_node_properties and refactor schema
parsing
time: 2023-05-16T09:42:41.793503-04:00
custom:
Author: gshank
Issue: "7430"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove legacy file logger code
time: 2023-05-18T11:42:51.825799-05:00
custom:
Author: iknox-fa
Issue: NA

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Break up integration tests as a short term fix for Windows CI runs
time: 2023-05-19T15:30:59.138043-04:00
custom:
Author: mikealfare
Issue: "7668"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Include null checks in utils test base
time: 2023-05-21T12:57:20.659726+02:00
custom:
Author: sdebruyn
Issue: "7670"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Write pub artifact to log
time: 2023-05-23T12:21:37.235938-04:00
custom:
Author: gshank
Issue: "7372"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix flaky test for --fail-fast
time: 2023-06-02T14:57:43.266412-04:00
custom:
Author: michelleark
Issue: "7744"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Create `add_from_artifact` to populate `state_relation` field of nodes
time: 2023-06-05T23:47:06.581326-07:00
custom:
Author: stu-k aranke
Issue: "7551"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Rm space from NodeType strings
time: 2023-06-12T07:08:27.276551-04:00
custom:
Author: jtcohen6
Issue: "7841"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Upgrade to dbt-semantic-interfaces v0.1.0dev5
time: 2023-06-15T16:32:17.602189-07:00
custom:
Author: QMalcolm
Issue: "7853"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: 'Refactoring: consolidating public_nodes and nodes'
time: 2023-06-16T13:15:03.668124-04:00
custom:
Author: michelleark
Issue: "7890"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Resolve SemanticModel ref is the same way as other refs
time: 2023-06-16T15:57:49.245262-04:00
custom:
Author: gshank
Issue: "7822"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move from dbt-semantic-intefaces 0.1.0dev5 to 0.1.0dev7
time: 2023-06-19T13:52:01.947206-07:00
custom:
Author: QMalcolm
Issue: "7898"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Don't jinja render packages from dependencies.yml
time: 2023-06-20T18:08:52.848093-04:00
custom:
Author: gshank
Issue: "7905"

37
.github/CODEOWNERS vendored
View File

@@ -11,14 +11,13 @@
# As a default for areas with no assignment,
# the core team as a whole will be assigned
* @dbt-labs/core
* @dbt-labs/core-team
### OSS Tooling Guild
/.github/ @dbt-labs/guild-oss-tooling
.bumpversion.cfg @dbt-labs/guild-oss-tooling
/.changes/ @dbt-labs/guild-oss-tooling
.changie.yaml @dbt-labs/guild-oss-tooling
pre-commit-config.yaml @dbt-labs/guild-oss-tooling
@@ -31,40 +30,6 @@ dev_requirements.txt @dbt-labs/guild-oss-tooling
/core/setup.py @dbt-labs/guild-oss-tooling
/core/MANIFEST.in @dbt-labs/guild-oss-tooling
### LANGUAGE
# Language core modules
/core/dbt/config/ @dbt-labs/core-language
/core/dbt/context/ @dbt-labs/core-language
/core/dbt/contracts/ @dbt-labs/core-language
/core/dbt/deps/ @dbt-labs/core-language
/core/dbt/events/ @dbt-labs/core-language # structured logging
/core/dbt/parser/ @dbt-labs/core-language
# Language misc files
/core/dbt/dataclass_schema.py @dbt-labs/core-language
/core/dbt/hooks.py @dbt-labs/core-language
/core/dbt/node_types.py @dbt-labs/core-language
/core/dbt/semver.py @dbt-labs/core-language
### EXECUTION
# Execution core modules
/core/dbt/graph/ @dbt-labs/core-execution
/core/dbt/task/ @dbt-labs/core-execution
# Execution misc files
/core/dbt/compilation.py @dbt-labs/core-execution
/core/dbt/flags.py @dbt-labs/core-execution
/core/dbt/lib.py @dbt-labs/core-execution
/core/dbt/main.py @dbt-labs/core-execution
/core/dbt/profiler.py @dbt-labs/core-execution
/core/dbt/selected_resources.py @dbt-labs/core-execution
/core/dbt/tracking.py @dbt-labs/core-execution
/core/dbt/version.py @dbt-labs/core-execution
### ADAPTERS
# Adapter interface ("base" + "sql" adapter defaults, cache)

2
.github/_README.md vendored
View File

@@ -197,7 +197,7 @@ ___
```yaml
- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v1
uses: aws-actions/configure-aws-credentials@v2
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

View File

@@ -35,7 +35,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v3
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler

View File

@@ -13,7 +13,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v3
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler

View File

@@ -35,6 +35,6 @@ jobs:
github.event.pull_request.merged
&& contains(github.event.label.name, 'backport')
steps:
- uses: tibdex/backport@v2.0.2
- uses: tibdex/backport@v2.0.3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -50,7 +50,7 @@ jobs:
- name: Create and commit changelog on bot PR
if: ${{ contains(github.event.pull_request.labels.*.name, matrix.label) }}
id: bot_changelog
uses: emmyoop/changie_bot@v1.0.1
uses: emmyoop/changie_bot@v1.1.0
with:
GITHUB_TOKEN: ${{ secrets.FISHTOWN_BOT_PAT }}
commit_author_name: "Github Build Bot"

View File

@@ -18,8 +18,8 @@ permissions:
issues: write
jobs:
call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-creation.yml@main
call-creation-action:
uses: dbt-labs/actions/.github/workflows/jira-creation-actions.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -19,7 +19,7 @@ permissions:
jobs:
call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-label.yml@main
uses: dbt-labs/actions/.github/workflows/jira-label-actions.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -19,8 +19,8 @@ on:
permissions: read-all
jobs:
call-label-action:
uses: dbt-labs/jira-actions/.github/workflows/jira-transition.yml@main
call-transition-action:
uses: dbt-labs/actions/.github/workflows/jira-transition-actions.yml@main
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}

View File

@@ -45,7 +45,7 @@ jobs:
uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4.3.0
uses: actions/setup-python@v4
with:
python-version: '3.8'
@@ -69,7 +69,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
python-version: ["3.8", "3.9", "3.10", "3.11"]
env:
TOXENV: "unit"
@@ -80,7 +80,7 @@ jobs:
uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4.3.0
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
@@ -116,7 +116,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
python-version: ["3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-20.04]
include:
- python-version: 3.8
@@ -137,7 +137,7 @@ jobs:
uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4.3.0
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
@@ -192,7 +192,7 @@ jobs:
uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4.3.0
uses: actions/setup-python@v4
with:
python-version: '3.8'

View File

@@ -1,11 +1,7 @@
# **what?**
# The purpose of this workflow is to trigger CI to run for each
# release branch and main branch on a regular cadence. If the CI workflow
# fails for a branch, it will post to dev-core-alerts to raise awareness.
# The 'aurelien-baudet/workflow-dispatch' Action triggers the existing
# CI worklow file on the given branch to run so that even if we change the
# CI workflow file in the future, the one that is tailored for the given
# release branch will be used.
# fails for a branch, it will post to #dev-core-alerts to raise awareness.
# **why?**
# Ensures release branches and main are always shippable and not broken.
@@ -28,63 +24,8 @@ on:
permissions: read-all
jobs:
fetch-latest-branches:
runs-on: ubuntu-latest
outputs:
latest-branches: ${{ steps.get-latest-branches.outputs.repo-branches }}
steps:
- name: "Fetch dbt-core Latest Branches"
uses: dbt-labs/actions/fetch-repo-branches@v1.1.1
id: get-latest-branches
with:
repo_name: ${{ github.event.repository.name }}
organization: "dbt-labs"
pat: ${{ secrets.GITHUB_TOKEN }}
fetch_protected_branches_only: true
regex: "^1.[0-9]+.latest$"
perform_match_method: "match"
retries: 3
- name: "[ANNOTATION] ${{ github.event.repository.name }} - branches to test"
run: |
title="${{ github.event.repository.name }} - branches to test"
message="The workflow will run tests for the following branches of the ${{ github.event.repository.name }} repo: ${{ steps.get-latest-branches.outputs.repo-branches }}"
echo "::notice $title::$message"
kick-off-ci:
needs: [fetch-latest-branches]
name: Kick-off CI
runs-on: ubuntu-latest
strategy:
# must run CI 1 branch at a time b/c the workflow-dispatch Action polls for
# latest run for results and it gets confused when we kick off multiple runs
# at once. There is a race condition so we will just run in sequential order.
max-parallel: 1
fail-fast: false
matrix:
branch: ${{ fromJSON(needs.fetch-latest-branches.outputs.latest-branches) }}
include:
- branch: 'main'
steps:
- name: Call CI workflow for ${{ matrix.branch }} branch
id: trigger-step
uses: aurelien-baudet/workflow-dispatch@v2.1.1
with:
workflow: main.yml
ref: ${{ matrix.branch }}
token: ${{ secrets.FISHTOWN_BOT_PAT }}
- name: Post failure to Slack
uses: ravsamhq/notify-slack-action@v1
if: ${{ always() && !contains(steps.trigger-step.outputs.workflow-conclusion,'success') }}
with:
status: ${{ job.status }}
notification_title: 'dbt-core scheduled run of "${{ matrix.branch }}" branch not successful'
message_format: ':x: CI on branch "${{ matrix.branch }}" ${{ steps.trigger-step.outputs.workflow-conclusion }}'
footer: 'Linked failed CI run ${{ steps.trigger-step.outputs.workflow-url }}'
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}
run_tests:
uses: dbt-labs/actions/.github/workflows/release-branch-tests.yml@main
with:
workflows_to_run: '["main.yml"]'
secrets: inherit

View File

@@ -36,7 +36,7 @@ jobs:
latest: ${{ steps.latest.outputs.latest }}
minor_latest: ${{ steps.latest.outputs.minor_latest }}
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v3
- name: Split version
id: version
run: |
@@ -60,7 +60,7 @@ jobs:
needs: [get_version_meta]
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
uses: docker/setup-buildx-action@v2
build_and_push:
name: Build images and push to GHCR
@@ -76,14 +76,14 @@ jobs:
echo "build_arg_value=$BUILD_ARG_VALUE" >> $GITHUB_OUTPUT
- name: Log in to the GHCR
uses: docker/login-action@v1
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push MAJOR.MINOR.PATCH tag
uses: docker/build-push-action@v2
uses: docker/build-push-action@v4
with:
file: docker/Dockerfile
push: True
@@ -94,7 +94,7 @@ jobs:
ghcr.io/dbt-labs/${{ github.event.inputs.package }}:${{ github.event.inputs.version_number }}
- name: Build and push MINOR.latest tag
uses: docker/build-push-action@v2
uses: docker/build-push-action@v4
if: ${{ needs.get_version_meta.outputs.minor_latest == 'True' }}
with:
file: docker/Dockerfile
@@ -106,7 +106,7 @@ jobs:
ghcr.io/dbt-labs/${{ github.event.inputs.package }}:${{ needs.get_version_meta.outputs.major }}.${{ needs.get_version_meta.outputs.minor }}.latest
- name: Build and push latest tag
uses: docker/build-push-action@v2
uses: docker/build-push-action@v4
if: ${{ needs.get_version_meta.outputs.latest == 'True' }}
with:
file: docker/Dockerfile

View File

@@ -37,17 +37,17 @@ jobs:
steps:
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: 3.8
- name: Checkout dbt repo
uses: actions/checkout@v2.3.4
uses: actions/checkout@v3
with:
path: ${{ env.DBT_REPO_DIRECTORY }}
- name: Checkout schemas.getdbt.com repo
uses: actions/checkout@v2.3.4
uses: actions/checkout@v3
with:
repository: dbt-labs/schemas.getdbt.com
ref: 'main'
@@ -83,7 +83,7 @@ jobs:
fi
- name: Upload schema diff
uses: actions/upload-artifact@v2.2.4
uses: actions/upload-artifact@v3
if: ${{ failure() }}
with:
name: 'schema_schanges.txt'

View File

@@ -39,12 +39,12 @@ jobs:
steps:
- name: checkout dev
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
persist-credentials: false
- name: Setup Python
uses: actions/setup-python@v2.2.2
uses: actions/setup-python@v4
with:
python-version: "3.8"

View File

@@ -24,10 +24,8 @@ permissions:
jobs:
triage_label:
if: contains(github.event.issue.labels.*.name, 'awaiting_response')
runs-on: ubuntu-latest
steps:
- name: initial labeling
uses: andymckay/labeler@master
with:
add-labels: "triage"
remove-labels: "awaiting_response"
uses: dbt-labs/actions/.github/workflows/swap-labels.yml@main
with:
add_label: "triage"
remove_label: "awaiting_response"
secrets: inherit

View File

@@ -37,7 +37,7 @@ repos:
alias: flake8-check
stages: [manual]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.981
rev: v1.3.0
hooks:
- id: mypy
# N.B.: Mypy is... a bit fragile.

View File

@@ -5,6 +5,145 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.6.0-b5 - June 22, 2023
### Features
- Enable setting packages in dependencies.yml ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372), [#7736](https://github.com/dbt-labs/dbt-core/issues/7736))
- Add AdapterRegistered event log message ([#7038](https://github.com/dbt-labs/dbt-core/issues/7038))
- Further integrate semantic models into the DAG and partial parsing module ([#7800](https://github.com/dbt-labs/dbt-core/issues/7800))
### Fixes
- Fix regression in `run-operation` to not require the name of the package to run ([#7753](https://github.com/dbt-labs/dbt-core/issues/7753))
- Allow dbt show --inline preview of private models ([#7837](https://github.com/dbt-labs/dbt-core/issues/7837))
- Skip jinja parsing of metric filters ([#7864](https://github.com/dbt-labs/dbt-core/issues/7864))
- Fix a bad implicit string conversion regression in debug --config-dir code. ([#7774](https://github.com/dbt-labs/dbt-core/issues/7774))
- Fix UninstalledPackagesFoundError error message to use correct packages specified path ([#7921](https://github.com/dbt-labs/dbt-core/issues/7921))
- Fix: safe remove of external nodes from nodes.depends_on ([#7924](https://github.com/dbt-labs/dbt-core/issues/7924))
### Under the Hood
- Upgrade to dbt-semantic-interfaces v0.1.0dev5 ([#7853](https://github.com/dbt-labs/dbt-core/issues/7853))
- Refactoring: consolidating public_nodes and nodes ([#7890](https://github.com/dbt-labs/dbt-core/issues/7890))
- Resolve SemanticModel ref is the same way as other refs ([#7822](https://github.com/dbt-labs/dbt-core/issues/7822))
- Move from dbt-semantic-intefaces 0.1.0dev5 to 0.1.0dev7 ([#7898](https://github.com/dbt-labs/dbt-core/issues/7898))
- Don't jinja render packages from dependencies.yml ([#7905](https://github.com/dbt-labs/dbt-core/issues/7905))
### Dependencies
- Bump mypy from 0.981 to 1.0.1 ([#7027](https://github.com/dbt-labs/dbt-core/pull/7027))
## dbt-core 1.6.0-b4 - June 13, 2023
### Fixes
- Respect column 'quote' config in model contracts ([#7370](https://github.com/dbt-labs/dbt-core/issues/7370))
- Fix path selector when using project-dir ([#7819](https://github.com/dbt-labs/dbt-core/issues/7819))
- Allow project dependencies to use miscellaneous keys ([#7497](https://github.com/dbt-labs/dbt-core/issues/7497))
- Updating this error message to point to the correct URL ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))
- Update SemanticModel node to properly impelment the DSI 0.1.0dev3 SemanticModel protocol spec ([#7833](https://github.com/dbt-labs/dbt-core/issues/7833), [#7827](https://github.com/dbt-labs/dbt-core/issues/7827))
### Docs
- Fix for column tests not rendering on quoted columns ([dbt-docs/#201](https://github.com/dbt-labs/dbt-docs/issues/201))
### Under the Hood
- Rm space from NodeType strings ([#7841](https://github.com/dbt-labs/dbt-core/issues/7841))
### Contributors
- [@drewbanin](https://github.com/drewbanin) ([#201](https://github.com/dbt-labs/dbt-core/issues/201))
- [@mirnawong1](https://github.com/mirnawong1) ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))
## dbt-core 1.6.0-b3 - June 08, 2023
### Breaking Changes
- Dropped support for Python 3.7 ([#7082](https://github.com/dbt-labs/dbt-core/issues/7082))
- Switch from dbt-metrics to dbt-semantic-interfaces for MetricNode definitions ([#7500](https://github.com/dbt-labs/dbt-core/issues/7500), [#7404](https://github.com/dbt-labs/dbt-core/issues/7404))
### Features
- Add support for materialized views ([#6911](https://github.com/dbt-labs/dbt-core/issues/6911))
- Optimize template rendering for common parse scenarios ([#7449](https://github.com/dbt-labs/dbt-core/issues/7449))
- nodes in packages respect custom generate_alias_name, generate_schema_name, generate_database_name macro overrides defined in packages ([#7444](https://github.com/dbt-labs/dbt-core/issues/7444))
- Enable state for deferral to be separate from state for selectors ([#7300](https://github.com/dbt-labs/dbt-core/issues/7300))
- add access selection syntax ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- add project_name to manifest metadata ([#7752](https://github.com/dbt-labs/dbt-core/issues/7752))
- dbt retry ([#7299](https://github.com/dbt-labs/dbt-core/issues/7299))
- Revamp debug, add --connection flag. Prepare for future refactors/interface changes. ([#7104](https://github.com/dbt-labs/dbt-core/issues/7104))
- Validate public models are not materialized as ephemeral ([#7226](https://github.com/dbt-labs/dbt-core/issues/7226))
- Added support for parsing and serializaing semantic models ([#7499](https://github.com/dbt-labs/dbt-core/issues/7499), [#7503](https://github.com/dbt-labs/dbt-core/issues/7503))
### Fixes
- Constraint rendering fixes: wrap check expression in parentheses, foreign key 'references', support expression in all constraint types ([#7417](https://github.com/dbt-labs/dbt-core/issues/7417), [#7480](https://github.com/dbt-labs/dbt-core/issues/7480), [#7416](https://github.com/dbt-labs/dbt-core/issues/7416))
- Fix warning messages for deprecated dbt_project.yml configs ([#7424](https://github.com/dbt-labs/dbt-core/issues/7424))
- Add `%` to adapter suite test cases for `persist_docs` ([#7698](https://github.com/dbt-labs/dbt-core/issues/7698))
- Improve warnings for constraints and materialization types ([#7335](https://github.com/dbt-labs/dbt-core/issues/7335))
- Incorrect paths used for "target" and "state" directories ([#7465](https://github.com/dbt-labs/dbt-core/issues/7465))
- fix StopIteration error when publication for project not found ([#7711](https://github.com/dbt-labs/dbt-core/issues/7711))
- Using version 0 works when resolving single model ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
- Fix empty --warn-error-options error message ([#7730](https://github.com/dbt-labs/dbt-core/issues/7730))
- send sql header on contract enforcement ([#7714](https://github.com/dbt-labs/dbt-core/issues/7714))
- Fixed doc link in selector.py ([#7533](https://github.com/dbt-labs/dbt-core/issues/7533))
- Fix null-safe equals comparison via `equals` ([#7778](https://github.com/dbt-labs/dbt-core/issues/7778))
- Log PublicationArtifactAvailable even when partially parsing unchanged public models ([#7782](https://github.com/dbt-labs/dbt-core/issues/7782))
- fix RuntimeError when removing project dependency from dependencies.yml ([#7743](https://github.com/dbt-labs/dbt-core/issues/7743))
### Under the Hood
- Fix flaky test for --fail-fast ([#7744](https://github.com/dbt-labs/dbt-core/issues/7744))
- Create `add_from_artifact` to populate `state_relation` field of nodes ([#7551](https://github.com/dbt-labs/dbt-core/issues/7551))
### Contributors
- [@dave-connors-3](https://github.com/dave-connors-3) ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- [@quazi-irfan](https://github.com/quazi-irfan) ([#7533](https://github.com/dbt-labs/dbt-core/issues/7533))
- [@sdebruyn](https://github.com/sdebruyn) ([#7082](https://github.com/dbt-labs/dbt-core/issues/7082))
- [@stu-k](https://github.com/stu-k) ([#7299](https://github.com/dbt-labs/dbt-core/issues/7299), [#7551](https://github.com/dbt-labs/dbt-core/issues/7551))
## dbt-core 1.6.0-b2 - May 25, 2023
### Features
- Added warnings for model and ref deprecations ([#7433](https://github.com/dbt-labs/dbt-core/issues/7433))
- Update drop_relation macro to allow for configuration of drop statement separately from object name ([#7625](https://github.com/dbt-labs/dbt-core/issues/7625))
- accept publications in dbt.invoke ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Fixes
- Honor `--skip-profile-setup` parameter when inside an existing project ([#7594](https://github.com/dbt-labs/dbt-core/issues/7594))
- Fix: Relative project paths weren't working with deps ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491))
- Exclude password fields from Jinja rendering. ([#7629](https://github.com/dbt-labs/dbt-core/issues/7629))
- Add --target-path to more CLI subcommands ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- Stringify flag paths for Jinja context ([#7495](https://github.com/dbt-labs/dbt-core/issues/7495))
- write run_results.json for run operation ([#7502](https://github.com/dbt-labs/dbt-core/issues/7502))
### Under the Hood
- Add ability to instantiate Flags class from dict ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607))
- Add other relation to reffable nodes ([#7550](https://github.com/dbt-labs/dbt-core/issues/7550))
- Move node patch method to schema parser patch_node_properties and refactor schema parsing ([#7430](https://github.com/dbt-labs/dbt-core/issues/7430))
- Remove legacy file logger code ([#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
- Break up integration tests as a short term fix for Windows CI runs ([#7668](https://github.com/dbt-labs/dbt-core/issues/7668))
- Include null checks in utils test base ([#7670](https://github.com/dbt-labs/dbt-core/issues/7670))
- Write pub artifact to log ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Dependencies
- Bump ubuntu from 23.04 to 23.10 ([#7675](https://github.com/dbt-labs/dbt-core/pull/7675))
- ([#7681](https://github.com/dbt-labs/dbt-core/pull/7681))
### Contributors
- [@dradnan89@hotmail.com](https://github.com/dradnan89@hotmail.com) ([#7681](https://github.com/dbt-labs/dbt-core/pull/7681))
- [@dwreeves](https://github.com/dwreeves) ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- [@iknox-fa](https://github.com/iknox-fa) ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491), [#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
- [@sdebruyn](https://github.com/sdebruyn) ([#7670](https://github.com/dbt-labs/dbt-core/issues/7670))
- [@stu-k](https://github.com/stu-k) ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607), [#7550](https://github.com/dbt-labs/dbt-core/issues/7550))
## dbt-core 1.6.0-b1 - May 12, 2023
### Features
@@ -56,7 +195,6 @@
- [@dwreeves](https://github.com/dwreeves) ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- [@thomasgjerdekog](https://github.com/thomasgjerdekog) ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
## dbt-core 1.6.0-a1 - April 17, 2023
## Previous Releases
@@ -64,6 +202,7 @@
For information on prior major and minor releases, see their changelogs:
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)

View File

@@ -56,7 +56,7 @@ There are some tools that will be helpful to you in developing locally. While th
These are the tools used in `dbt-core` development and testing:
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.7, 3.8, 3.9, 3.10 and 3.11
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.8, 3.9, 3.10 and 3.11
- [`pytest`](https://docs.pytest.org/en/latest/) to define, discover, and run tests
- [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting
- [`black`](https://github.com/psf/black) for code formatting
@@ -163,7 +163,7 @@ suites.
#### `tox`
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.7, Python 3.8, Python 3.9, Python 3.10 and Python 3.11 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration for these tests in located in `tox.ini`.
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.8, Python 3.9, Python 3.10 and Python 3.11 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py38`. The configuration for these tests in located in `tox.ini`.
#### `pytest`
@@ -171,9 +171,9 @@ Finally, you can also run a specific test or group of tests using [`pytest`](htt
```sh
# run all unit tests in a file
python3 -m pytest test/unit/test_graph.py
python3 -m pytest tests/unit/test_graph.py
# run a specific unit test
python3 -m pytest test/unit/test_graph.py::GraphTest::test__dependency_list
python3 -m pytest tests/unit/test_graph.py::GraphTest::test__dependency_list
# run specific Postgres integration tests (old way)
python3 -m pytest -m profile_postgres test/integration/074_postgres_unlogged_table_tests
# run specific Postgres integration tests (new way)
@@ -185,7 +185,7 @@ python3 -m pytest tests/functional/sources
### Unit, Integration, Functional?
Here are some general rules for adding tests:
* unit tests (`test/unit` & `tests/unit`) dont need to access a database; "pure Python" tests should be written as unit tests
* unit tests (`tests/unit`) dont need to access a database; "pure Python" tests should be written as unit tests
* functional tests (`test/integration` & `tests/functional`) cover anything that interacts with a database, namely adapter
* *everything in* `test/*` *is being steadily migrated to* `tests/*`

View File

@@ -3,7 +3,7 @@
# See `/docker` for a generic and production-ready docker file
##
FROM ubuntu:23.04
FROM ubuntu:23.10
ENV DEBIAN_FRONTEND noninteractive
@@ -33,13 +33,6 @@ RUN apt-get update \
python \
python-dev \
python3-pip \
python3.6 \
python3.6-dev \
python3-pip \
python3.6-venv \
python3.7 \
python3.7-dev \
python3.7-venv \
python3.8 \
python3.8-dev \
python3.8-venv \

View File

@@ -384,7 +384,7 @@ class BaseAdapter(metaclass=AdapterMeta):
return {
self.Relation.create_from(self.config, node).without_identifier()
for node in manifest.nodes.values()
if (node.is_relational and not node.is_ephemeral_model)
if (node.is_relational and not node.is_ephemeral_model and not node.is_external_node)
}
def _get_catalog_schemas(self, manifest: Manifest) -> SchemaSearchMap:
@@ -415,7 +415,7 @@ class BaseAdapter(metaclass=AdapterMeta):
return info_schema_name_map
def _relations_cache_for_schemas(
self, manifest: Manifest, cache_schemas: Set[BaseRelation] = None
self, manifest: Manifest, cache_schemas: Optional[Set[BaseRelation]] = None
) -> None:
"""Populate the relations cache for the given schemas. Returns an
iterable of the schemas populated, as strings.
@@ -451,7 +451,7 @@ class BaseAdapter(metaclass=AdapterMeta):
self,
manifest: Manifest,
clear: bool = False,
required_schemas: Set[BaseRelation] = None,
required_schemas: Optional[Set[BaseRelation]] = None,
) -> None:
"""Run a query that gets a populated cache of the relations in the
database and set the cache on this adapter.
@@ -986,7 +986,7 @@ class BaseAdapter(metaclass=AdapterMeta):
manifest: Optional[Manifest] = None,
project: Optional[str] = None,
context_override: Optional[Dict[str, Any]] = None,
kwargs: Dict[str, Any] = None,
kwargs: Optional[Dict[str, Any]] = None,
text_only_columns: Optional[Iterable[str]] = None,
) -> AttrDict:
"""Look macro_name up in the manifest and execute its results.
@@ -1319,20 +1319,26 @@ class BaseAdapter(metaclass=AdapterMeta):
def render_column_constraint(cls, constraint: ColumnLevelConstraint) -> Optional[str]:
"""Render the given constraint as DDL text. Should be overriden by adapters which need custom constraint
rendering."""
if constraint.type == ConstraintType.check and constraint.expression:
return f"check {constraint.expression}"
constraint_expression = constraint.expression or ""
rendered_column_constraint = None
if constraint.type == ConstraintType.check and constraint_expression:
rendered_column_constraint = f"check ({constraint_expression})"
elif constraint.type == ConstraintType.not_null:
return "not null"
rendered_column_constraint = f"not null {constraint_expression}"
elif constraint.type == ConstraintType.unique:
return "unique"
rendered_column_constraint = f"unique {constraint_expression}"
elif constraint.type == ConstraintType.primary_key:
return "primary key"
elif constraint.type == ConstraintType.foreign_key:
return "foreign key"
elif constraint.type == ConstraintType.custom and constraint.expression:
return constraint.expression
else:
return None
rendered_column_constraint = f"primary key {constraint_expression}"
elif constraint.type == ConstraintType.foreign_key and constraint_expression:
rendered_column_constraint = f"references {constraint_expression}"
elif constraint.type == ConstraintType.custom and constraint_expression:
rendered_column_constraint = constraint_expression
if rendered_column_constraint:
rendered_column_constraint = rendered_column_constraint.strip()
return rendered_column_constraint
@available
@classmethod
@@ -1340,7 +1346,8 @@ class BaseAdapter(metaclass=AdapterMeta):
rendered_column_constraints = []
for v in raw_columns.values():
rendered_column_constraint = [f"{v['name']} {v['data_type']}"]
col_name = cls.quote(v["name"]) if v.get("quote") else v["name"]
rendered_column_constraint = [f"{col_name} {v['data_type']}"]
for con in v.get("constraints", None):
constraint = cls._parse_column_constraint(con)
c = cls.process_parsed_constraint(constraint, cls.render_column_constraint)
@@ -1399,13 +1406,15 @@ class BaseAdapter(metaclass=AdapterMeta):
constraint_prefix = f"constraint {constraint.name} " if constraint.name else ""
column_list = ", ".join(constraint.columns)
if constraint.type == ConstraintType.check and constraint.expression:
return f"{constraint_prefix}check {constraint.expression}"
return f"{constraint_prefix}check ({constraint.expression})"
elif constraint.type == ConstraintType.unique:
return f"{constraint_prefix}unique ({column_list})"
constraint_expression = f" {constraint.expression}" if constraint.expression else ""
return f"{constraint_prefix}unique{constraint_expression} ({column_list})"
elif constraint.type == ConstraintType.primary_key:
return f"{constraint_prefix}primary key ({column_list})"
elif constraint.type == ConstraintType.foreign_key:
return f"{constraint_prefix}foreign key ({column_list})"
constraint_expression = f" {constraint.expression}" if constraint.expression else ""
return f"{constraint_prefix}primary key{constraint_expression} ({column_list})"
elif constraint.type == ConstraintType.foreign_key and constraint.expression:
return f"{constraint_prefix}foreign key ({column_list}) references {constraint.expression}"
elif constraint.type == ConstraintType.custom and constraint.expression:
return f"{constraint_prefix}{constraint.expression}"
else:

View File

@@ -328,6 +328,10 @@ class BaseRelation(FakeAPIObject, Hashable):
def is_view(self) -> bool:
return self.type == RelationType.View
@property
def is_materialized_view(self) -> bool:
return self.type == RelationType.MaterializedView
@classproperty
def Table(cls) -> str:
return str(RelationType.Table)
@@ -344,6 +348,10 @@ class BaseRelation(FakeAPIObject, Hashable):
def External(cls) -> str:
return str(RelationType.External)
@classproperty
def MaterializedView(cls) -> str:
return str(RelationType.MaterializedView)
@classproperty
def get_relation_type(cls) -> Type[RelationType]:
return RelationType

View File

@@ -9,10 +9,11 @@ from dbt.adapters.base.plugin import AdapterPlugin
from dbt.adapters.protocol import AdapterConfig, AdapterProtocol, RelationProtocol
from dbt.contracts.connection import AdapterRequiredConfig, Credentials
from dbt.events.functions import fire_event
from dbt.events.types import AdapterImportError, PluginLoadError
from dbt.events.types import AdapterImportError, PluginLoadError, AdapterRegistered
from dbt.exceptions import DbtInternalError, DbtRuntimeError
from dbt.include.global_project import PACKAGE_PATH as GLOBAL_PROJECT_PATH
from dbt.include.global_project import PROJECT_NAME as GLOBAL_PROJECT_NAME
from dbt.semver import VersionSpecifier
Adapter = AdapterProtocol
@@ -89,7 +90,13 @@ class AdapterContainer:
def register_adapter(self, config: AdapterRequiredConfig) -> None:
adapter_name = config.credentials.type
adapter_type = self.get_adapter_class_by_name(adapter_name)
adapter_version = import_module(f".{adapter_name}.__version__", "dbt.adapters").version
adapter_version_specifier = VersionSpecifier.from_version_string(
adapter_version
).to_version_string()
fire_event(
AdapterRegistered(adapter_name=adapter_name, adapter_version=adapter_version_specifier)
)
with self.lock:
if adapter_name in self.adapters:
# this shouldn't really happen...

View File

@@ -0,0 +1,25 @@
# RelationConfig
This package serves as an initial abstraction for managing the inspection of existing relations and determining
changes on those relations. It arose from the materialized view work and is currently only supporting
materialized views for Postgres and Redshift as well as dynamic tables for Snowflake. There are three main
classes in this package.
## RelationConfigBase
This is a very small class that only has a `from_dict()` method and a default `NotImplementedError()`. At some
point this could be replaced by a more robust framework, like `mashumaro` or `pydantic`.
## RelationConfigChange
This class inherits from `RelationConfigBase` ; however, this can be thought of as a separate class. The subclassing
merely points to the idea that both classes would likely inherit from the same class in a `mashumaro` or
`pydantic` implementation. This class is much more restricted in attribution. It should really only
ever need an `action` and a `context`. This can be though of as being analogous to a web request. You need to
know what you're doing (`action`: 'create' = GET, 'drop' = DELETE, etc.) and the information (`context`) needed
to make the change. In our scenarios, the context tends to be an instance of `RelationConfigBase` corresponding
to the new state.
## RelationConfigValidationMixin
This mixin provides optional validation mechanics that can be applied to either `RelationConfigBase` or
`RelationConfigChange` subclasses. A validation rule is a combination of a `validation_check`, something
that should evaluate to `True`, and an optional `validation_error`, an instance of `DbtRuntimeError`
that should be raised in the event the `validation_check` fails. While optional, it's recommended that
the `validation_error` be provided for clearer transparency to the end user.

View File

@@ -0,0 +1,12 @@
from dbt.adapters.relation_configs.config_base import ( # noqa: F401
RelationConfigBase,
RelationResults,
)
from dbt.adapters.relation_configs.config_change import ( # noqa: F401
RelationConfigChangeAction,
RelationConfigChange,
)
from dbt.adapters.relation_configs.config_validation import ( # noqa: F401
RelationConfigValidationMixin,
RelationConfigValidationRule,
)

View File

@@ -0,0 +1,44 @@
from dataclasses import dataclass
from typing import Union, Dict
import agate
from dbt.utils import filter_null_values
"""
This is what relation metadata from the database looks like. It's a dictionary because there will be
multiple grains of data for a single object. For example, a materialized view in Postgres has base level information,
like name. But it also can have multiple indexes, which needs to be a separate query. It might look like this:
{
"base": agate.Row({"table_name": "table_abc", "query": "select * from table_def"})
"indexes": agate.Table("rows": [
agate.Row({"name": "index_a", "columns": ["column_a"], "type": "hash", "unique": False}),
agate.Row({"name": "index_b", "columns": ["time_dim_a"], "type": "btree", "unique": False}),
])
}
"""
RelationResults = Dict[str, Union[agate.Row, agate.Table]]
@dataclass(frozen=True)
class RelationConfigBase:
@classmethod
def from_dict(cls, kwargs_dict) -> "RelationConfigBase":
"""
This assumes the subclass of `RelationConfigBase` is flat, in the sense that no attribute is
itself another subclass of `RelationConfigBase`. If that's not the case, this should be overriden
to manually manage that complexity.
Args:
kwargs_dict: the dict representation of this instance
Returns: the `RelationConfigBase` representation associated with the provided dict
"""
return cls(**filter_null_values(kwargs_dict)) # type: ignore
@classmethod
def _not_implemented_error(cls) -> NotImplementedError:
return NotImplementedError(
"This relation type has not been fully configured for this adapter."
)

View File

@@ -0,0 +1,23 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Hashable
from dbt.adapters.relation_configs.config_base import RelationConfigBase
from dbt.dataclass_schema import StrEnum
class RelationConfigChangeAction(StrEnum):
alter = "alter"
create = "create"
drop = "drop"
@dataclass(frozen=True, eq=True, unsafe_hash=True)
class RelationConfigChange(RelationConfigBase, ABC):
action: RelationConfigChangeAction
context: Hashable # this is usually a RelationConfig, e.g. IndexConfig, but shouldn't be limited
@property
@abstractmethod
def requires_full_refresh(self) -> bool:
raise self._not_implemented_error()

View File

@@ -0,0 +1,57 @@
from dataclasses import dataclass
from typing import Set, Optional
from dbt.exceptions import DbtRuntimeError
@dataclass(frozen=True, eq=True, unsafe_hash=True)
class RelationConfigValidationRule:
validation_check: bool
validation_error: Optional[DbtRuntimeError]
@property
def default_error(self):
return DbtRuntimeError(
"There was a validation error in preparing this relation config."
"No additional context was provided by this adapter."
)
@dataclass(frozen=True)
class RelationConfigValidationMixin:
def __post_init__(self):
self.run_validation_rules()
@property
def validation_rules(self) -> Set[RelationConfigValidationRule]:
"""
A set of validation rules to run against the object upon creation.
A validation rule is a combination of a validation check (bool) and an optional error message.
This defaults to no validation rules if not implemented. It's recommended to override this with values,
but that may not always be necessary.
Returns: a set of validation rules
"""
return set()
def run_validation_rules(self):
for validation_rule in self.validation_rules:
try:
assert validation_rule.validation_check
except AssertionError:
if validation_rule.validation_error:
raise validation_rule.validation_error
else:
raise validation_rule.default_error
self.run_child_validation_rules()
def run_child_validation_rules(self):
for attr_value in vars(self).values():
if hasattr(attr_value, "validation_rules"):
attr_value.run_validation_rules()
if isinstance(attr_value, set):
for member in attr_value:
if hasattr(member, "validation_rules"):
member.run_validation_rules()

Some files were not shown because too many files have changed in this diff Show More