Compare commits

...

470 Commits

Author SHA1 Message Date
Gerda Shank
e983ee4a75 Update sqslparse to 0.4.4 2023-06-22 17:37:23 -04:00
FishtownBuildBot
533988233e [Automated] Merged prep-release/1.6.0b5_5349512955 into target main during release process 2023-06-22 15:07:48 -05:00
Peter Webb
8bc0e77a1d CT-2719: Rename the semantic_nodes collection on the manifest to semantic_models (#7927) 2023-06-22 15:59:54 -04:00
Github Build Bot
1c93c9bb58 Bumping version to 1.6.0b5 and generate changelog 2023-06-22 19:27:29 +00:00
Michelle Ark
6d7b32977c Fix: safe remove of external nodes from nodes.depends_on (#7923) 2023-06-22 13:42:06 -04:00
Michelle Ark
bf15466bec UninstalledPackagesFoundError references correct packages specified path (#7886) 2023-06-22 11:40:59 -04:00
Gerda Shank
fb1ebe48f0 Resolve SemanticModel refs in the same way as other refs (#7895) 2023-06-22 11:26:08 -04:00
Peter Webb
de65697ff9 Further Integrate Semantic Models (#7917)
* CT-2651: Add Semantic Models to the manifest and various pieces of graph linking code

* CT-2651: Finish integrating semantic models into the partial parsing system

* CT-2651: More semantic model details for partial parsing

* CT-2651: Remove merged references to project_dependencies

* CT-2651: Revise changelog entry

* CT-2651: Disable unit test until partial parsing of semantic models is complete.

* CT-2651: Temporarily disable an apparently-flaky test.
2023-06-21 18:31:54 -04:00
Michelle Ark
ecf90d689e Refactor/unify public and model nodes (#7891) 2023-06-21 17:12:15 -04:00
Mila Page
4cdeff11cd Remove --config-dir instead of fixing #7774 (#7793)
* Stringfy the dir always to solve the bug.

* Add changelog

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-06-21 11:36:47 -07:00
Gerda Shank
9ff2f6e430 Do not jinja render packages from dependencies yml (#7910)
* Add some comments to methods constructing Project/RuntimeConfig

* Save flag that packages dict came from dependencies.yml

* Test for not rendering packages_dict

* Changie

* Ensure packages_yml_dict and dependencies_yml_dict are dictionaries

* Ensure "packages" passed to render_packages is a dict
2023-06-21 14:33:23 -04:00
Jeremy Cohen
73a0dc6d14 Reorganize, annotate, and revise dependency pins (#7368)
* Reorganize + annotate dependencies

* Loosen pins on other dbt-labs packages

* Minor pins, no patch pins

* Rm support for py37

* Fix protobuf pin

* Bump networkx upper bound #6551
2023-06-21 07:08:43 +02:00
Kshitij Aranke
0a1c73e0fd Fixes #7753: Fix regression in run-operation to not require the name of the package to run (#7811) 2023-06-20 11:19:16 -05:00
Quigley Malcolm
8653ffc5a4 Upgrade core to support dbt-semantic-interfaces 0.1.0dev7 (#7903)
* Bump DSI dependency version to 0.1.0dev7

* Cleaner DSI type enum importing

Previoulsy we had to use individual import paths for each type enum
that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces
has been updated to allow for importing all the type enums from a
singular path.

* Cleaner DSI protocol importing

Previoulsy we had to use individual import paths for each protocol
that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces
has been updated to allow for importing all the protocols from a
singular path.

* Add semantic protocol satisifcation test for metric type params

* Replace `metric.type_params.measures` with `metric.type_params.input_measures`

In DSI 0.1.0dev7 `measures` on metric type params became `input_measures`.
Additionally `input_measures` should not be user specified but something
we compile at parse time, thus we've removed it from `UnparsedMetricTypeParams`.
Finally, actually populating `input_measures` is somewhat complicated due
to the existance of derived metrics, thus that work is being pushed
off to CT-2707.

* Update metric numerator/denominator to be `MetricInput`s

In DSI 0.1.0dev7 `metric.type_params.numerator` and `metric.type_params.denominator`
switched from being `MetricInputMeasure`s to `MetricInput`s. This
commit reflects that change. Additionally, some helper functions on
metric type params were removed related to the numerator and denominator.
Thus we've removed them respectively in this commit.

* Add protocol satisfaction tests for `MetricInput` and `MetricInputMeasure`

* Add `post_aggregation_reference` to `MetricInput` and fix typo in `MetricInputMeasure`

DSI 0.1.0dev7 added `post_aggregation_reference` to the `MetricInput` protocol,
thus we've added it to our implementation in core. Additionally, we had a typo
in a method name in our implementation of `MetricInputMeasure`, ironically
a similar function to the one we've added for `MetricInput`

* Changie doc for upgraded to DSI 0.1.0dev7

* Fix parsing of metric numerator and denominator in schema_yaml_readers

Previously numerator and denominator of a metric were `MetricInputMeasure`s,
now they're `MetricInput`s. Changing the typing isn't enough though.
We have parsing functions in `schema_yaml_readers` which were specifically
parsing the numerator and denominator as if they were `MetricInputMeasure`s.
Thus we had to updating the schema_yaml_readers to parse them as `MetricInput`s.
During this we had some logic in a parsing function `_get_metric_inputs` which
could be abstracted to newly added functions.
2023-06-20 08:08:57 -07:00
Quigley Malcolm
86583a350f Ct-2690 Support dbt-semantic-interfaces 0.1.0dev5 (#7888)
* Upgrade to dbt-semantic-interfaces v0.1.0dev5

This is a fairly simple upgrade. Literally it's just pointing at the
the new versions. The v3 schemas are directly compatible with v5 because
there were no protocol level changes from v3 to v5. All the changers were
updates to tools MetricFlow uses from DSI, not tools that we ourselves
are using in core (yet).

* Add changie doc for DSI version bump
2023-06-16 12:47:55 -07:00
Quigley Malcolm
fafab5d557 [CT-2696] Skip jinia parsing of metric filters (#7885)
* Update metric filters in testing fixtures

I incorrectly wrote the tests such that they didn't include curly
braces, `{{..}}`, around things like `dimension(..)` for filters.
This updates the tests fixtures to have proper filter specifications

* Skip jinja rendering of `filter` key of metrics

Note that `filter` can show up in multiple places: as a root key
on a metric (`metric.filter`), on a metric input (`metric.type_params.metrics[x].filter`),
denominator (`metric.type_params.denominator.filter`), numerator
(`metric.type_params.numerator.filter`), and a metric input measure
(`metric.type_params.measure.filter` and `metric.type_params.measures[x].filter`).
In this commit we skip all of them :)

* Add changie doc for skipping jinja parsing for metric filters

* Update yaml renderer test for metrics
2023-06-15 15:41:31 -07:00
Gerda Shank
39e0c22353 Allow setting packages in dependencies.yml and move dependencies to runtime config (#7857) 2023-06-15 14:41:57 -04:00
colin-rogers-dbt
f767943fb2 Add AdapterRegistered event log message (#7862)
* Add AdapterRegistered event log message

* Add AdapterRegistered to unit test

* make versioning and logging consistent

* make versioning and logging consistent

* add to_version_string

* remove extra equals

* format fire_event
2023-06-14 14:56:47 -07:00
dependabot[bot]
ae97831ebf Bump mypy from 0.981 to 1.0.1 (#7027)
* Bump mypy from 0.981 to 1.0.1

Bumps [mypy](https://github.com/python/mypy) from 0.981 to 1.0.1.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.981...v1.0.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

* upgrade mypy and fix all errors

* fixing some duplicate imports from conflict resolution

* fix mypy errors from merging in main

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-06-14 14:31:57 -05:00
Jeremy Cohen
f16bae0ab9 Fix: dbt show --inline with private models (#7838)
* Add functional test

* Check resource_type before DbtReferenceError

* Changelog entry
2023-06-14 12:23:20 -04:00
FishtownBuildBot
b947b2bc7e [Automated] Merged prep-release/1.6.0b4_5260249343 into target main during release process 2023-06-13 16:40:37 -05:00
Github Build Bot
7068688181 Bumping version to 1.6.0b4 and generate changelog 2023-06-13 20:54:08 +00:00
Quigley Malcolm
38c0600982 Update SemanticModel node to match DSI 0.1.0dev3 protocols (#7848)
* Add tests to ensure our semantic layer nodes satisfy the DSI protocols

These tests create runtime checkable versions of the protocols defined in
DSI. Thus we can instantiate instances of our semantic layer nodes and
use `isinstance` to check that they satisfy the protocol. These `runtime_checkable`
versions of the protocols should only exist in testing and should never
be used in the actual package code.

* Update the `Dimension` object of `SemanticModel` node to match DSI protocol

* Make `UnparsedDimension` more strict and update schema readers accordingly

* Update the `Entity` object of `SemanticModel` node to match DSI protocol

* Make `UnparsedEntity` more strict and update schema readers accordingly

* Update the `Measure` object of `SemanticModel` node to match DSI protocol

* Make `UnparsedMeasure` more strict and update schema readers accordingly

* Update the `SemanticModel` node to match DSI protocol

A lot of the additions are helper functions which we don't actually
use in core. This is a known issue. We're in the process of removing
a fair number of them from the DSI protocol spec. However, in the meantime
we need to implement them to satisfy the protocol unfortunately.

* Make `UnparsedSemanticModel` more strict and update schema readers accordingly

* Changie entry for updating SemanticModel node
2023-06-13 13:25:35 -07:00
Jeremy Cohen
83d163add5 Respect column quote config in model contracts (#7537) 2023-06-13 15:32:56 -04:00
Emily Rockman
d46e8855ef Allow ProjectDependency to have extra fields (#7834)
* Allow ProjectDependency to have extra fields

* changelog
2023-06-13 09:47:50 -05:00
mirnawong1
60524c0f8e update adapters url (#7779)
* update adapters url

in response to [docs.getedbt.com pr 3465](https://github.com/dbt-labs/docs.getdbt.com/issues/3465), updating this error message to point to the correct URL, which was recently changed. 

old URL: https://docs.getdbt.com/docs/supported-data-platforms#adapter-installation
new URL: https://docs.getdbt.com/docs/connect-adapters#install-using-the-cli

thank you @dbeatty10 for your 🦅 👀 !

* adding changie entry

* Update .changes/unreleased/Breaking Changes-20230612-161159.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-06-13 08:51:21 -05:00
Gerda Shank
ca73a2aa15 Use project directory in path selector instead of cwd (#7829)
* Use contextvar to store and get project_root for path selector method

* Changie

* Modify test to check Path selector with project-dir

* Don't set cv_project_root in base task if no config
2023-06-13 09:08:09 -04:00
Emily Rockman
4a833a4272 Move remaining unit tests to /tests directory (#7843)
* moved remaining unit tests to  directory

* fix path

* Delete profiles.yml

* rename remaining test refs
2023-06-13 07:08:28 -05:00
FishtownBuildBot
f9abeca231 Add new index.html and changelog yaml files from dbt-docs (#7836) 2023-06-12 11:55:42 -04:00
Jeremy Cohen
5f9e527768 Rm spaces from NodeType strings (#7842)
* Rm space from NodeType strings

* Add changelog entry
2023-06-12 11:55:07 -04:00
FishtownBuildBot
6f51de4cb5 [Automated] Merged prep-release/1.6.0b3_5215324721 into target main during release process 2023-06-08 15:52:09 -05:00
Github Build Bot
cb64682d33 Bumping version to 1.6.0b3 and generate changelog 2023-06-08 20:15:20 +00:00
Quigley Malcolm
98d1a94b60 Update spec MetricNode for dbt x MetricFlow integration and begin outputting semantic manifest artifact (#7812)
* Refactor MetricNode definition to satisfy DSI Metric protocol

* Fix tests involving metrics to have updated properties

* Update UnparsedMetricNode to match new metric yaml spec

* Update MetricParser for new unparsed and parsed MetricNodes

* Remove `rename_metric_attr`

We're intentionally breaking the spec. There will be a separate tool provided
for migrating from dbt-metrics to dbt x metricflow. This bit of code was renaming
things like `type` to `calculation_method`. This is problematic because `type` is
on the new spec, while `calculation_method` is not. Additionally, since we're
intentionally breaking the spec, this function, `rename_metric_attr`, shouldn't be
used for any property renaming.

* Fix tests for Metrics (1.6) changes

* Regenerated v10 manifest schema and associated functional test artifact state

* Remove no longer needed tests

* Skip / comment out tests for metrics functionality that we'll be implementing later

* Begin outputting semantic manifest artifact on every run

* Drop metrics during upgrade_manifest_json if manifest is v9 or before

* Update properties of `minimal_parsed_metric_dict` to match new metric spec

* Add changie entry for metric node breaking changes

* Add semantic model nodes to semantic manifest
2023-06-08 10:23:36 -07:00
Peter Webb
a89da7ca88 Add SemanticModel Node Type (#7769)
* Add dbt-semantic-interfaces as a dependency

With the integration with MetricFlow we're taking a dependency on
`dbt-semantic-interfaces` which acts as the source of truth for
protocols which MetricFlow and dbt-core need to agree on. Additionally
we're hard pinning to 0.1.0.dev3 for now. We plan on having a less
restrictive specification when dbt-core 1.6 hits GA.

* Add implementations of DSI Metadata protocol to nodes.py

* CT-2521: Initial work on adding new SemanticModel node

* CT-2521: Second rough draft of SemanticModels

* CT-2521: Update schema v10

* CT-2521: Update unit tests for new SemanticModel collection in manifest

* CT-2521: Add changelog entry

* CT-2521: Final touches on initial implementation of SemanticModel parsing

* Change name of Metadata class to reduce potential for confusion

* Remove "Replaceable" inheritance, per review

* CT-2521: Rename internal variables from semantic_models to semantic_nodes

* CT-2521: Update manifest schema to reflect change

---------

Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
2023-06-08 09:39:04 -04:00
Mike Alfare
2d237828ae ADAP-2: Materialized Views (#7239)
* changie

* ADAP-387: Stub materialized view as a materialization (#7211)

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* updating to feature main

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>

* ADAP-387: reverting db_api implementation (#7322)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* updating to feature main

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* added tests

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* typo

* moved materialized view tests into adapter test framework

* add enum to relation for comparison in jinja

---------

Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>

* ADAP-391: Add configuration change option (#7272)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* move up pre-existing drop of backup

* change relation type to view to be consistent

* add base test case

* fix jinja exception message expression, basic test passing

* align names between postgres and default to same convention

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integration tests

* add on_configuration_change to expected_manifest to pass functional tests

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* moved materialized view tests into adapter test framework

* add alter materialized view macro

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* updated default option for on_configuration_change on NodeConfig

* removed explicit calls to enum values

* add test setup for testing fail config option

* updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name

* setup configuration change framework

* skipped tests that are expected to fail without adapter implementation

* cleaned up log checks

---------

Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>

* ADAP-388: Stub materialized view as a materialization - postgres (#7244)

* move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>

* ADAP-402: Add configuration change option - postgres (#7334)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* added materialized view stubs and test

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* updated postgres to use the new macros structure

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* init copy of pr 387 to begin 391 implementation

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* remove postgres-specific materialization in favor of core default materialization

* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies

* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integretion tests

* add on_configuration_change to expected_manifest to pass functuional tests

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* revert to current approach for materializations

* added tests

* move materialized view logic into the `/materializations` directory in line with `dbt-core`

* moved default macros in `dbt-core` into `dbt-postgres`

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* moved materialized view tests into adapter test framework

* updated materialized view tests to use adapter test framework

* add alter materialized view macro

* add alter materialized view macro

* change class name, and config setup

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* added in `on_configuration_change` setting

* updated default option for on_configuration_change on NodeConfig

* updated default option for on_configuration_change on NodeConfig

* fixed list being passed as string bug

* removed explicit calls to enum values

* removed unneeded test class

* fixed on_configuration_change to be picked up appropriately

* add test setup for testing fail config option

* remove breakpoint, uncomment tests

* update skip scenario to use empty strings

* update skip scenario to avoid using sql at all, remove extra whitespace in some templates

* push up initial addition of indexes for mv macro

* push slight change up

* reverting alt macro and moving the do create_index call to be more in line with other materializations

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402

# Conflicts:
#	core/dbt/contracts/graph/model_config.py
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/base.py
#	tests/functional/materializations/test_materialized_view.py

* merge feature branch into story branch

* merge feature branch into story branch

* added indexes into the workflow

* fix error in jinja that caused print error

* working on test messaging and skipping tests that might not fit quite into current system

* add drop and show macros for indexes

* add drop and show macros for indexes

* add logic to determine the indexes to create or drop

* pulled index updates through the workflow properly

* convert configuration changes to fixtures, implement index changes into tests

* created Model dataclass for readability, added column to swap index columns for testing

* fixed typo

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>

* ADAP-395: Implement native materialized view DDL (#7336)

* changie

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* added materialized view stubs and test

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* updated postgres to use the new macros structure

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* init copy of pr 387 to begin 391 implementation

* updating to feature main

* updating to feature main

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* remove postgres-specific materialization in favor of core default materialization

* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies

* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integretion tests

* add on_configuration_change to expected_manifest to pass functuional tests

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* revert to current approach for materializations

* added tests

* move materialized view logic into the `/materializations` directory in line with `dbt-core`

* moved default macros in `dbt-core` into `dbt-postgres`

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* typo

* moved materialized view tests into adapter test framework

* updated materialized view tests to use adapter test framework

* add alter materialized view macro

* add alter materialized view macro

* added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update

* change class name, and config setup

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* added in `on_configuration_change` setting

* updated default option for on_configuration_change on NodeConfig

* updated default option for on_configuration_change on NodeConfig

* fixed list being passed as string bug

* fixed list being passed as string bug

* removed explicit calls to enum values

* removed explicit calls to enum values

* removed unneeded test class

* fixed on_configuration_change to be picked up appropriately

* add test setup for testing fail config option

* remove breakpoint, uncomment tests

* update skip scenario to use empty strings

* update skip scenario to avoid using sql at all, remove extra whitespace in some templates

* push up initial addition of indexes for mv macro

* push slight change up

* reverting alt macro and moving the do create_index call to be more in line with other materializations

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402

# Conflicts:
#	core/dbt/contracts/graph/model_config.py
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/base.py
#	tests/functional/materializations/test_materialized_view.py

* merge feature branch into story branch

* merge feature branch into story branch

* added indexes into the workflow

* fix error in jinja that caused print error

* working on test messaging and skipping tests that might not fit quite into current system

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395

# Conflicts:
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	plugins/postgres/dbt/include/postgres/macros/adapters.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py
#	tests/functional/materializations/test_materialized_view.py

* moved postgres implemention into plugin directory

* update index methods to align with the configuration update macro

* added native ddl to postgres macros

* removed extra docstring

* updated references to View, now references MaterializedView

* decomposed materialization into macros

* refactor index create statement parser, add exceptions for unexpected formats

* swapped conditional to check for positive state

* removed skipped test now that materialized view is being used

* return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test

* add check for indexes in particular for apply on configuration scenario

* removed extra argument

* add materialized views to get_relations / list_relations

* typos in index change logic

* moved full refresh check inside the build sql step

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>

* removing returns from tests to stop logs from printing

* moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures

* fixed overwrite issue, simplified assertion method

* updated import order to standard

* fixed test import paths

* updated naming convention for proper test collection with the test runner

* still trying to make the test runner happy

* rewrite index updates to use a better source in Postgres

* break out a large test suite as a separate run

* update `skip` and `fail` scenarios with more descriptive results

* typo

* removed call to skip status

* reverting `exceptions_jinja.py`

* added FailFastError back, the right way

* removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports

* removed assumed models in method calls, removed odd insert records and replaced with get row count

* fixed index issue, removed some indirection in testing

* made test more readable

* remove the "apply" from the tests and put it on the base as the default

* generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility

* fixed type to be more generic to accommodate adapters with their own relation types

* fixed all the broken index stuff

* updated on_configuration_change to use existing patterns

* updated on_configuration_change to use existing patterns

* reflected update in tests and materialization logic

* reflected update in tests and materialization logic

* reverted the change to create a config object from the option object, using just the option object now

* reverted the change to create a config object from the option object, using just the option object now

* modelled database objects to support monitoring all configuration changes

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* reverted centralized framework, retained a few reusable base classes

* updated names to be more consistent

* readability updates

* added readme specifying that `relation_configs` only supports materialized views for now

---------

Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
2023-06-07 19:19:09 -04:00
Michelle Ark
f4253da72a fix: removing dependency from dependencies.yml (#7743) 2023-06-07 13:48:39 -04:00
Mila Page
919822e583 Adap 496/add test connection mode to debug (#7741)
* --connection-flag

* Standardize the plugin functions used by DebugTask

* Cleanup redundant code and help logic along.

* Add more output tests to add logic coverage and formatting.

* Code review

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-06-07 09:56:35 -07:00
Michelle Ark
444c787729 fix error message for empty/None: --warn-error-options handling (#7735) 2023-06-07 12:23:40 -04:00
Michelle Ark
3b63dd9f11 Validate public models are not materialized as ephemeral (#7794) 2023-06-07 12:22:57 -04:00
Kshitij Aranke
84166bf457 Fixes #7551: Create add_from_artifact to populate state_relation field of nodes (#7796) 2023-06-06 15:25:12 -07:00
Michelle Ark
dd445e1fde generalize BaseModelConstraintsRuntimeEnforcement (#7805) 2023-06-06 16:30:50 -04:00
Michelle Ark
6a22ec1b2e Package-namespaced generate_x_name macro resolution (#7509) 2023-06-06 15:05:44 -04:00
Emily Rockman
587bbcbf0d Improve warnings for constraints and materialization types (#7696)
* first pass

* debugging

* regen proto types

* refactor to use warn_supported flag

* PR feedback
2023-06-06 12:50:58 -05:00
Doug Beatty
8e1c4ec116 Fix not equals comparison to be null-safe for adapters/utils tests (#7776)
* Fix names within functional test

* Changelog entry

* Test for implementation of null-safe equals comparison

* Remove duplicated where filter

* Fix null-safe equals comparison

* Fix tests for `concat` and `hash` by using empty strings () instead of `null`

* Remove macro namespace interpolation
2023-06-06 06:11:48 -06:00
Kshitij Aranke
dc35f56baa Fixes #7299: dbt retry (#7763) 2023-06-05 15:51:00 -07:00
Michelle Ark
60d116b5b5 log PublicationArtifactAvailable even when partially parsing & public models unchanged (#7783) 2023-06-05 14:59:38 -04:00
Emily Rockman
4dbc4a41c4 remove entire changes folder (#7766)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-06-05 09:51:30 -05:00
Michelle Ark
89541faec9 force dependency between test models (#7767) 2023-06-05 10:45:59 -04:00
Gerda Shank
79bd98560b Version 0 for model works for latest_version (#7712) 2023-06-05 10:21:39 -04:00
Michelle Ark
7917bd5033 add project_name to manifest metadata (#7754) 2023-06-02 16:16:14 -04:00
Michelle Ark
05b0ebb184 Fix constraint rendering for expressions and foreign key constraint types (#7512) 2023-06-02 15:05:09 -04:00
Quazi Irfan
e1d7a53325 Fix doc link in selector.py (#7755)
* Fix doc link in selector.py

* Ran changie to modify changelog entry
2023-06-02 09:00:58 -05:00
Michelle Ark
7a06d354aa pass optional sql_header to empty subquery sql rendering (#7734) 2023-06-01 14:08:02 -04:00
dave-connors-3
9dd5ab90bf add ability to select models by access (#7739)
* add ability to select models by access

* changie

* Update core/dbt/graph/selector_methods.py
2023-06-01 09:25:59 -04:00
Michelle Ark
45d614533f fix StopIteration error when publication not found (#7710) 2023-05-30 16:50:36 -04:00
Peter Webb
00a531d9d6 Template rendering optimization (#7451)
* CT-2478: Template rendering optimization

* CT-2478: Fix type annotation, and accomodate non-string unit test cases.
2023-05-30 12:48:47 -04:00
Sam Debruyn
fd301a38db Dropped support for Python 3.7 (#7623) 2023-05-30 12:12:57 -04:00
Jeremy Cohen
9c7e01dbca Readd exp_path for config deprecation warnings (#7536) 2023-05-30 12:04:49 -04:00
github-actions[bot]
1ac6df0996 Adding performance modeling for 1.2.0 to refs/heads/main (#7560)
* adding performance baseline for 1.2.0

* Adding newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-27 22:10:38 -04:00
Gerda Shank
38ca4fce25 Target path should be relative to project dir, rather than current working directory (#7706) 2023-05-26 18:50:38 -04:00
Kshitij Aranke
7e3a6eec96 fix #7300: Enable state for deferral to be separate from state for selectors (#7690) 2023-05-26 13:00:16 -07:00
Emily Rockman
ac16a55c64 Update to reusable workflow for branch testing (#7676)
* fix overlooked node12 case with abandonded marketplace action

* update slack notification

* remove spaces per formatting

* replace with cli dispatch

* move conditional

* add explicit token, temp comment out slack

* add checkout

* checkout teh right branch

* switch to PAT

* add back repo checkout

* manually check workflow status

* fix notification

* swap to reusable workflow

* fix path

* swap permissions

* remove trigger

* fix secrets

* point to main
2023-05-26 14:51:00 -05:00
Doug Beatty
620ca40b85 Add % to adapter suite test cases for persist_docs (#7699)
* Test table/view/column-level comments with `%` symbol

* Test docs block with `%` symbol

* Changelog entry
2023-05-26 12:48:08 -04:00
leahwicz
aa11cf2956 Adding link to 1.5 release notes (#7707) 2023-05-26 08:38:13 -04:00
FishtownBuildBot
feb06e2107 [Automated] Merged prep-release/1.6.0b2_5081502847 into target main during release process 2023-05-25 10:41:03 -05:00
Github Build Bot
a3d40e0abf Bumping version to 1.6.0b2 and generate changelog 2023-05-25 15:06:24 +00:00
Gerda Shank
7c1bd91d0a CT 2590 write pub artifact to log (#7686) 2023-05-24 13:54:58 -04:00
leahwicz
70a132d059 Updating CODEOWNERS to consolidate (#7693)
Consolidating to remove Language and Execution and instead default to the `core-team`
2023-05-24 09:12:25 -04:00
Anis Nasir
1fdebc660b Relaxed the pyyaml dependency to >=5.3. (#7681) 2023-05-24 08:41:38 -04:00
Gerda Shank
0516192d69 CT 2516 ensure that paths in Jinja context flags object are strings (#7678) 2023-05-24 08:24:36 -04:00
github-actions[bot]
f99be58217 Adding performance modeling for 1.4.6 to refs/heads/main (#7532)
* adding performance baseline for 1.4.6

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-24 08:17:19 -04:00
github-actions[bot]
3b6222e516 Adding performance modeling for 1.3.0 to refs/heads/main (#7530)
* adding performance baseline for 1.3.0

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-24 08:16:30 -04:00
github-actions[bot]
b88e60f8dd Adding performance modeling for 1.4.1 to refs/heads/main (#7527)
* adding performance baseline for 1.4.1

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-24 08:16:02 -04:00
github-actions[bot]
9373c4d1e4 Adding performance modeling for 1.3.4 to refs/heads/main (#7525)
* adding performance baseline for 1.3.4

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-24 08:15:26 -04:00
dependabot[bot]
0fe3ee8eca Bump ubuntu from 23.04 to 23.10 (#7675)
* Bump ubuntu from 23.04 to 23.10

Bumps ubuntu from 23.04 to 23.10.

---
updated-dependencies:
- dependency-name: ubuntu
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-05-24 08:01:10 -04:00
Sam Debruyn
0d71a32aa2 Include null checks in utils test base (#7672)
* Include null checks in utils test base

* Add tests for the schema test

* Add tests for this macro

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-05-23 19:49:48 -07:00
Doug Beatty
0f223663bb Honor --skip-profile-setup parameter when inside an existing project (#7609)
* Honor `--skip-profile-setup` parameter when inside an existing project

* Use project name as the profile name

* Use separate file connections for reading and writing

* Raise a custom exception when no adapters are installed

* Test skipping interactive profile setup when inside a dbt project

* Replace `assert_not_called()` since it does not work

* Verbose CLI argument for skipping profile setup

* Use separate file connections for reading and writing

* Check empty list in a Pythonic manner
2023-05-23 17:56:54 -06:00
Kshitij Aranke
c25d0c9f9c fix #7502: write run_results.json for run operation (#7655) 2023-05-23 14:56:23 -07:00
Peter Webb
4a4b7beeb9 Model Deprecation (#7562)
* CT-2461: Work toward model deprecation

* CT-2461: Remove unneeded conversions

* CT-2461: Fix up unit tests for new fields, correct a couple oversights

* CT-2461: Remaining implementation and tests for model/ref deprecation warnings

* CT-2461: Changelog entry for deprecation warnings

* CT-2461: Refine datetime handling and tests

* CT-2461: Fix up unit test data

* CT-2461: Fix some more unit test data.

* CT-2461: Fix merge issues

* CT-2461: Code review items.

* CT-2461: Improve version -> str conversion
2023-05-23 09:30:32 -04:00
Ian Knox
265e09dc93 Remove DelayedFileHandler (#7661)
* remove DelayedFileHandler

* Changelog

* set_path to no-op

* more no-ops for rpc

* Clearer comments
2023-05-22 16:42:48 -04:00
Mike Alfare
87ea28fe84 break out a large test suite as a separate execution to avoid memory issues on windows CI runs (#7669) 2023-05-19 17:04:35 -04:00
Michelle Ark
af0f786f2e Accept PublicationArtifacts in dbtRunner.invoke (#7656) 2023-05-18 16:42:50 -04:00
David Bloss
50528a009d update used gh actions ahead of node12 deprecation (#7651)
* update used gh actions ahead of node12 deprecation

* replace with valid tag

---------

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-05-17 16:20:26 -05:00
Stu Kilgore
f6e5582370 Add "other" relation to reffable node classes (#7645) 2023-05-17 12:14:16 -05:00
Peter Webb
dea3181d96 Exclude some profile fields from Jinja rendering when they are not valid Jinja. (#7630)
* CT-2583: Exclude some profile fields from Jinja rendering.

* CT-2583: Add functional test.

* CT-2583: Change approach to password jinja detection

* CT-2583: Extract string constant and add additional checks

* CT-2583: Improve unit test coverage
2023-05-17 11:38:48 -04:00
Gerda Shank
5f7ae2fd4c Move node patch method to schema parser patch_node_properties and refactor schema parsing (#7640) 2023-05-16 21:08:04 -04:00
Daniel Reeves
4f249b8652 Add target_path to more cli commands that use it (#7647) 2023-05-16 16:22:59 -05:00
Ian Knox
df23f68dd4 Missed PR fedback (#7642) 2023-05-16 13:50:28 -05:00
Stu Kilgore
4b091cee9e Instantiate Flags class from dict and command name (#7624) 2023-05-16 13:31:23 -05:00
Ian Knox
dcb5acdf29 bugfix: Deps hangs when using relative paths via --project-dir (#7628) 2023-05-16 10:00:23 -05:00
Emily Rockman
7fbeced315 updates for github deprecations (#7614)
* updates for github deprecations

* fix jira file name

* swap out abandonded action

* add quotes to env var

* revert main.yml
2023-05-15 15:28:52 -05:00
Mike Alfare
47e7b1cc80 Feature/drop relation/ct 2581 (#7626)
* changie
* move drop_relation macros into their own file, add scenarios for table, view, and materialized view
2023-05-15 15:51:19 -04:00
FishtownBuildBot
8f998c218e [Automated] Merged prep-release/1.6.0b1_4961250999 into target main during release process 2023-05-12 12:37:41 -05:00
Github Build Bot
41c0797d7a Bumping version to 1.6.0b1 and generate changelog 2023-05-12 17:04:26 +00:00
Michelle Ark
3f2cba0dec add --artifact flag to scripts/collect-artifact-schema (#7599) 2023-05-11 14:31:00 -04:00
Michelle Ark
b60c67d107 add publication artifact to schemas (#7590) 2023-05-11 13:18:35 -04:00
Doug Beatty
630cd3aba0 Allow missing profiles.yml for dbt deps and dbt init (#7546)
* Allow missing `profiles.yml` for `dbt deps` and `dbt init`

* Some commands allow the `--profiles-dir` to not exist

* Remove fix to verify that CI tests work

* Allow missing `profiles.yml` for `dbt deps` and `dbt init`

* CI is not finding any installed adapters

* Remove functional test for `dbt init`
2023-05-10 19:24:04 -06:00
Emily Rockman
05595f5920 Detect breaking changes to constraints in state:modifed (#7476)
* added test that fails

* added new exception

* add partial error checking - needs more specifics

* move contract check under modelnode

* try adding only enforced constraints

* add checks for enforced constraints

* changelog

* add materialization logic

* clean up tests, tweak materializations

* PR feedback

* more PR feedback

* change to tuple
2023-05-10 15:39:26 -05:00
Gerda Shank
29f2cfc48d CT 2510 Throw error for duplicate versioned and non versioned model names (#7577)
* Check for versioned/unversioned duplicates

* Add new exception DuplicateVersionedUnversionedError

* Changie

* Handle packages when finding versioned and unversioned duplicates
2023-05-10 16:16:38 -04:00
Gerda Shank
43d949c5cc CT 2494 check for project level dependency cycles (#7558)
* Raise error if dependent project depends on current project

* Test for project dependency cycle

* Changie
2023-05-10 10:20:03 -04:00
Mike Alfare
58312f1816 CT-2556: pin urllib3 to 1.x (#7574)
* pin urllib3 to 1.x

* changie
2023-05-09 18:36:20 -04:00
Ian Knox
dffbb6a659 Always write run_results.json (#7539) 2023-05-09 13:46:21 -05:00
Kshitij Aranke
272beb21a9 fix #7413: inject sql header in query for show (#7568) 2023-05-09 11:29:12 -07:00
Gerda Shank
d34c511fa5 CT 2552 pin protobuf to >=4.0.0 (#7566)
* Pin protobuf to >=4.0.0

* Changie
2023-05-09 13:10:09 -04:00
github-actions[bot]
2945619eb8 Adding performance modeling for 1.4.0 to refs/heads/main (#7523)
* adding performance baseline for 1.4.0

* Fix formatting

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-05-09 09:14:53 -04:00
Kshitij Aranke
078a83679a fix #7390: push down limit filtering to adapter (#7545) 2023-05-08 22:22:58 -07:00
Kshitij Aranke
881437e890 fix #7273: enable dbt show for seeds (#7544) 2023-05-08 15:08:24 -07:00
Kshitij Aranke
40aca4bc17 fix #7407: print model version in dbt show if specified (#7543) 2023-05-08 14:30:07 -07:00
Michelle Ark
0de046dfbe Allow duplicate refable node names across packages (#7374) 2023-05-08 16:41:37 -04:00
Jeremy Cohen
5a7b73be26 Do not rewrite manifest.json during 'docs serve' command (#7554) 2023-05-08 15:36:07 -04:00
Jeremy Cohen
35f8ceb7f1 Back compat for previous return type of collect_freshness (#7535)
* Back compat for previous retrurn type of 'collect_freshness'

* Test fixups

* PR feedback
2023-05-08 10:11:42 -04:00
Doug Beatty
19d6dab973 Fix inverted --print/--no-print flag (#7524) 2023-05-08 07:20:29 -06:00
leahwicz
810ef7f556 Adding perf testing GHA (#5851)
* Adding perf testing GHA

* Fixing tigger syntax

* Fixing PR creation issue

* Updating testing var

* Remove unneeded branch names

* Fixing branch naming convention

* Standardizing branch name to var

* Consolidating PR jobs

* Updating inputs and making more readable

* Splitting steps up

* Making some updates here to simplify and update

* Remove tab

* Cleaned up testing TODOs before committing

* Fixing spacing

* Fixing spacing issue
2023-05-05 09:41:41 -04:00
Gerda Shank
fd7306643f Initial implementation of cross-project ref (#7276)
* Create publication.py, various Publication classes, Dependency class

* Load dependencies.yml and the corresponding publication file

* Add "public_nodes" and populate ref_lookup

* resolve_ref working

* Add public nodes to parent and child maps

* Bump manifest version and fix tests, use ModelDependsOn

* Split out PublicationArtifact and PublicationConfig, store public_models
separately

* Store dependencies in publication artifact

* change detection of PublicModel for >= python3.10

* Handle removing references for re-processing if publication has changed

* Handle only changed publication artifacts

* Add some logging events

* Remove duplicate nodes from manifest

* refactor relation_from_relation_name

* Remove duplicate writing of manifest.json

* Add public_nodes to flat_graph

* Move some file name constants to core/dbt/constants.py

* Remove "environment" from ProjectDependency. Add
database/schema/identifier to PublicModel. Update TargetNotFound
exception.

* Include external publication dependencies in publication artifact dependencies

* Remove create_from_relation_name, call create_from_node instead

* Change PublicationArtifactChanged message to debug level

* Make write_publication_artifact a function in parser/manifest.py

* Create fixture to create minimal alternate project (just models)

* develop multi project test case
2023-05-03 10:56:40 -04:00
Brice Luu
f1dddaa6e9 Ignore parent tests added edges for build selection (#7431) 2023-04-28 17:01:06 -05:00
Michelle Ark
a7eb89d645 active project > local project in ConfiguredVar (#7441) 2023-04-28 16:17:28 -04:00
Peter Webb
c56a9b2b7f CT-2414: Add graph summaries to target directory output (#7358)
* CT-2414: Add graph summaries to target directory output

* CT-2414: Make graph representation more compact

* CT-2414: Add changelog entry

* CT-2414: Remove temporary diagnostic code.

* CT-2414: Combine graphs into a single file

* CT-2414: Simplify graph summary format.

* CT-2414: Add invocation id to summary, add unit test
2023-04-28 16:00:31 -04:00
Emily Rockman
17a8f462dd Update CODEOWNERS to include the OSS Tooling Guild (#7472)
* Update CODEOWNERS to include the OSS Tooling Guild

* add a few more files
2023-04-28 08:17:53 -05:00
Jeremy Cohen
e3498bdaa5 Remove noisy parse events (#7388)
* Rm noisy parse events

* PR feedback
2023-04-27 18:17:53 +02:00
Gerda Shank
d2f963e20e CT 2483 duplicate depends on nodes (#7455)
* Remove unnecessary "_update_into" methods
2023-04-25 14:10:59 -04:00
Jeremy Cohen
d53bb37186 UX improvements to model versions (#7435)
* Latest version should use un-suffixed alias

* Latest version can be in un-suffixed file

* FYI when unpinned ref to model with prerelease version

* [WIP] Nicer error if versioned ref to unversioned model

* Revert "Latest version should use un-suffixed alias"

This reverts commit 3616c52c1eed7588b9e210e1c957dfda598be550.

* Revert "[WIP] Nicer error if versioned ref to unversioned model"

This reverts commit c9ae4af1cfbd6b7bfc5dcbb445556233eb4bd2c0.

* Define real event for UnpinnedRefNewVersionAvailable

* Update pp test for implicit unsuffixed defined_in

* Add changelog entry

* Fix unit test

* marky feedback

* Add test case for UnpinnedRefNewVersionAvailable event
2023-04-25 19:55:58 +02:00
Michelle Ark
9874f9e004 Fix groupable node partial parsing, raise DbtReferenceError in RuntimeRefResolver (#7438) 2023-04-25 11:09:35 -04:00
Michelle Ark
2739d5f4c4 fix partial parsing of versioned models - schedule child nodes if latest version has been modified (#7439) 2023-04-25 10:30:03 -04:00
Ian Knox
d07603b288 Clear cached statement results when retrieved from ProviderContext and subclasses (#7371) 2023-04-24 12:36:50 -05:00
Jeremy Cohen
723ac9493d Fix .gitignore to take heed of tests/functional/build (#7436) 2023-04-24 19:23:09 +02:00
Stu Kilgore
de75777ede Persist timing info for failed nodes (#7353) 2023-04-24 11:13:41 -05:00
Daniel Reeves
75703c10ee add --target-path to snapshot command (#7419) 2023-04-21 08:29:15 -07:00
Michelle Ark
1722079a43 fix v0 ref resolution and latest_version configuration(#7415) 2023-04-20 12:06:14 -04:00
Michelle Ark
f5aea191d1 add cloud-artifacts as codeowners of /schemas/dbt (#7406) 2023-04-20 12:00:59 -04:00
Chenyu Li
b2418b0634 fix target dir behavior with sources.json (#7412) 2023-04-20 08:02:49 -07:00
Andrey Bystrov
aac034d9ba Empty catalog dbt docs generate (#7202) 2023-04-20 09:57:42 -05:00
Jeremy Cohen
ada8860e48 Update --help text for cache-related parameters (#7389) 2023-04-19 07:31:52 +02:00
Michelle Ark
a87275a4ca [Fix] safe version attribute access in _check_resource_uniqueness (#7376)
safe version attribute access in _check_resource_uniqueness
2023-04-18 14:22:03 -04:00
Michelle Ark
0891aef8d7 [Fix] Typo in statically parsed ref unpacking (#7365)
test statically parsed two-argument ref
2023-04-18 13:06:07 -04:00
Kshitij Aranke
add924221a Add dbt 1.5 to changelog on main (#7403) 2023-04-18 09:17:40 -07:00
github-actions[bot]
ba40d07ea3 [create-pull-request] automated change (#7377)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
2023-04-18 11:15:45 -04:00
Michelle Ark
57e9096816 sqlparse <0.4.4 (#7394) 2023-04-18 11:01:38 -04:00
Jeremy Cohen
6fedfe0ece Update docs link in ContractBreakingChangeError message (#7367) 2023-04-18 14:26:45 +02:00
Mike Alfare
121fa5793f dbt.include import issue for ci (#7383)
* separating the installation of the two requirements files
2023-04-18 01:08:40 -04:00
FishtownBuildBot
a88f640395 Cleanup main after cutting new 1.5.latest branch (#7354)
* Bumping version to 1.5.0rc1

* Clean up changelog on main

* fixing linter issues

---------

Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
2023-04-13 23:18:12 -04:00
Michelle Ark
74419b0e86 [Fix] Fix versioned model selection in subdirectories (#7351)
* fix versioned model selection in subdirectories
2023-04-13 10:58:52 -04:00
Jeremy Cohen
2ddf296a8e Adding a new column is not a breaking contract change (#7333)
* Adding a new column is not a breaking contract change

* Add changelog entry

* More structured exception

* same_contract: False if non-breaking changes

* PR feedback: rm build_contract_checksum, more comments
2023-04-13 00:19:46 +02:00
Chenyu Li
6b42a712a8 remove providing profile/project to dbtRunner (#7337) 2023-04-12 12:23:48 -07:00
Benoît Allard
c3230d3374 Allow greater mashumaro versions (#7294)
* Allow greater mashumaro versions

Current 3.5 is working fine.

* Pin mashumaro to 3.6 (released 4 days ago)

* Add changie
2023-04-12 14:40:54 -04:00
Chenyu Li
602535fe71 compile inline query doesn't add node (#7326)
* remove inline node after compile

* add test

* unset inline node
2023-04-12 09:03:55 -07:00
FishtownBuildBot
f9b28bcaed Add new index.html and changelog yaml files from dbt-docs (#7328)
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2023-04-12 11:19:49 -04:00
Gerda Shank
922c75344b Add access, has_group, contract_enforced, versioned to run_model (#7309) 2023-04-12 10:56:10 -04:00
Michelle Ark
2caf87c247 Add version selector method (#7330)
Add version selector method: "latest", "prerelease", "old", and "none" selectors supported
2023-04-12 10:48:17 -04:00
Michelle Ark
f2a3535c3f add schema_string_type fixture to test_constraints (#7229) 2023-04-12 10:27:00 -04:00
Peter Webb
a500e60b7f Reset invocation_id Between Commands (#7317)
* CT-2317: Reset invocation id in preflight for each dbt command.

* CT-2317: Add unit test for invocation_id behavior.

* CT-2317: Add changelog entry.

* CT-2317: Modify freshness test to ignore invocation_id

* CT-2317: Assign invocation_id before tracking initialization.

* CT-2317: Fix unit test failures and a bunch of other stuff

* CT-2317: Remove checks which make outdated assumptions about invocation_id being stable between runs

* CT-2317: Review tweak, more unit test fixes.
2023-04-12 09:54:17 -04:00
Michelle Ark
c7ebc8935f model versions (#7287)
model versioning and versioned ref resolution
2023-04-12 09:50:39 -04:00
Neelesh Salian
56f8f8a329 Fix for simple snapshot for seed and test assert (#7296)
* Fix for simple snapshot for seed and test assert

* Bigquery varchar type fix
2023-04-11 19:20:41 -07:00
Doug Beatty
828d723512 Update help text for commands + params in CLI and top-level API (#7176)
* Removed options for `dbt parse`

* Fix misspellings

* Capitalize JSON when appropriate

* Update help text for --write-json/--no-write-json

* Update help text for --config-dir

* Update help text for --resource-types

* Removed decorators for removed dbt parse options

* Remove `--write-manifest` flag from `parse`

* Remove `--parse-only` flag from `compile`

* Update help text for `dbt list --output`

* Standardize on one line per argument

* Factor 3 from 12 Factor CLI Apps

* Update help text for `dbt --version`

* Standardize capitalization of resource types for `dbt build`

* `debug --config-dir` is a boolean flag

* Update help text for `--version-check`

* Specify `-q` as a conventional alias for `--quiet`

* Update help text for `debug --config-dir`

* Update help text for `debug`

* Treat more dense text blobs as binary for `git grep`

* Update help text for `--version-check`

* Update help text for `--defer`

* Update help text for `--indirect-selection`

* Co-locate log colorization with other log settings

* Update help text for `--log-format*`, `--log-level*`, and `--use-colors*`

* Temporarily re-add option for CI tests

* Remove `--parse-only` flag from `show`

* Remove `--write-manifest` flag from `parse` (again)
2023-04-11 17:09:00 -06:00
Robbert Hofman
b450a5754e Snapshot strategies: newline for subquery (#6780)
* Snapshot strategies: newline for subquery

* add changie output

* add test for snapshot ending in comment

* remove import to be flake8 compliant

* add seed import

* add newlines for flake8 compliance

* typo fix

* Fixing up a test, adding a comment or two

* removed un-needed test fixtures

* removed even more un-needed fixtures, collapsed test to single class

* removed errant breakpoint()

* Fix a little typo

---------

Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com>
2023-04-11 14:49:36 -07:00
Emily Rockman
2971b9a027 Generalize constraint warnings (#7250)
* CT-1922: Rough in functionality for parsing model level constraints

* CT-1922: (Almost) complete support for model level constraints

* CT-1922: Fix typo affecting correct model constraint parsing.

* CT-1922: Rework base class for model tests for greater simplicity

* CT-1922: Rough in functionality for parsing model level constraints

* CT-1922: Revise unit tests for new model-level constraints property

* CT-1922: (Almost) complete support for model level constraints

* first pass

* implement in core

* add proto

* WIP

* resolve errors in columns_spec_ddl

* changelog

* update comment

* move logic over to python

* rename and use enum

* update default constraint_support dict

* generate new proto definition after conflicts

* reorganize code and break warnings into each constraint

* fix postgres constraint support

* remove breakpoint

* convert constraint support to constant

* update postgres

* add to export

* add to export

* regen proto types file

* standardize names

* put back mypy error

* more naming + add back comma

* add constraint support to model level constraints

* update event message and method signature

* rename method

* CT-1922: Rough in functionality for parsing model level constraints

* CT-1922: Revise unit tests for new model-level constraints property

* CT-1922: (Almost) complete support for model level constraints

* CT-1922: Fix typo affecting correct model constraint parsing.

* CT-1922: Improve whitespace handling

* CT-1922: Render raw constraints to constraint list directly

* make method return consistent

* regenerate proto defn

* update evvent test

* add some code cleanup

---------

Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
2023-04-11 14:25:55 -05:00
Chenyu Li
3c54959829 remove functions and added time to remove the rest (#7270)
* remove functions and added time to remove the rest

* completely remove lib.py
2023-04-11 11:58:09 -07:00
Stu Kilgore
87e25e8692 Add option to skip relation cache population (#7307) 2023-04-11 13:15:23 -05:00
Peter Webb
6ac5c90a0b CT-1922: Parsing and postgres implementations for model level constraints (#7230)
* CT-1922: Rough in functionality for parsing model level constraints

* CT-1922: Revise unit tests for new model-level constraints property

* CT-1922: (Almost) complete support for model level constraints

* CT-1922: Fix typo affecting correct model constraint parsing.

* CT-1922: Minor code review refinements

* CT-1922: Improve whitespace handling

* CT-1922: Render raw constraints to constraint list directly

* CT-1922: Rework base class for model tests for greater simplicity

* CT-1922: Remove debugging properties. Oops.

* CT-1922: Fix type annotation
2023-04-11 13:59:03 -04:00
Ian Knox
a58fb24e2b Handle duplicated flags better (#7303) 2023-04-11 11:49:15 -05:00
dave-connors-3
9ce593c47f allow resources to be selected by patch path (#7321)
* allow resources to be selected by patch path

* don't ever changie

* use hasattr

* Ignore mypy complaints about patch_path

---------

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-04-11 12:30:46 -04:00
Jeremy Cohen
c9d4051136 Rebase #7223: improve contracts error message (#7319)
* improved first line of error

* added basic printing of yaml and sql cols as columns

* added changie log

* used listed dictionary as input to match columns

* swapped order of col headers for printing

* used listed dictionary as input to match columns

* removed merge conflict text from file

* Touch-ups

* Update log introspection in functional tests

* Update format_column macro. Case insensitive test

* PR feedback: just data_type, not formatted

---------

Co-authored-by: Kyle Kent <kyle.kent321@gmail.com>
2023-04-11 18:27:23 +02:00
Jeremy Cohen
26f3518cea Update CLI/API docs example, header (#7320)
* Rm CLI/API example

* Update autogen docs header
2023-04-11 18:27:05 +02:00
Ian Knox
49eed67ab0 Parse command should return manifest when invoked via dbtRunner (#7314) 2023-04-11 10:37:56 -05:00
Matteo Ferrando
7a4d3bd2dc fix: avoid dbtRunner default callbacks being shared across instances (#7279)
* fix: avoid dbtRunner default callbacks being shared across instances

* changie
2023-04-11 10:43:17 +02:00
Stu Kilgore
2afb4ccd68 Handle internal exceptions in postflight (#7242) 2023-04-10 17:25:51 -05:00
Gerda Shank
f38d5ad8e2 Add unique_id to CompiledNode and ShowNode logging events (#7308) 2023-04-10 17:09:01 -04:00
Jeremy Cohen
7e1f04c667 Prettify ListRelations message (#7311) 2023-04-10 23:02:01 +02:00
Jeremy Cohen
ef2ba39dcf Add relation info to node_info in logging (#7304)
* Add db, schema, alias, rel_name to node_info

* Add changelog entry

* Default to None

* Nested node_relation message
2023-04-10 22:16:54 +02:00
Tobie Tusing
7045e11aa0 Remove trivial nodes before building subdag (#7194)
* remove trial nodes before building subdag

* add changie

* Update graph.py

remove comment

* further optimize by sorting node search by degree

* change degree to product of in and out degree
2023-04-07 12:09:58 -04:00
Neelesh Salian
a9016c37f5 Fix the base classes of simple snapshot for better import (#7293) 2023-04-07 08:26:38 -07:00
Neelesh Salian
fe62ab8ec5 Moving simple_snapshot test from individual adapters to core adapter zone (#7275)
* Simple snapshot adapter zone

* Class naming

* Class name to Base

* delete extra test file
2023-04-06 18:18:47 -07:00
Gerda Shank
893daedc42 Ensure that same_contract is called for state:modified (#7283) 2023-04-06 15:45:25 -04:00
Doug Beatty
44be13b006 Avoid revoking grants for views when copy_grants=true (#7285)
* Replace hard-coded parameter with intended variable instead

* Changelog entry
2023-04-06 11:37:20 -06:00
Quigley Malcolm
a5131ecc7d CT-2264, CT-2259, CT-1783: Improved event serialization failure handling (#7249)
* Add tests for logging jinja2.Undefined objects

[CT-2259](https://github.com/dbt-labs/dbt-core/issues/7108) identifies
an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined
object is attempted to be logged. This generally happened in the form
of `{{ log(undefined_variable, info=True) }}`. This commit adding this
test exists for two reasons
1. Ensure we don't have a regression in this going forward
2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3

* Add tests for checking `DBT_ENV_SECRET_`s don't break logging

[CT-1783](https://github.com/dbt-labs/dbt-core/issues/6568) describes
a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all
`{{ log("logging stuff", info=True) }}` invocations break. This commit
adds a test for this for two reasons:
1. Ensure we don't regress to this behavior going forward
2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3

* Add tests ensuring failed event serialization is handled correctly

[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113) states
that failed serialization should result in an exception handling path
which will fire another event instead of raising an exception. This is
hard to test perfectly because the exception handling path for
serialization depending on whether pytest is present. If pytest isn't
present, a new event documentation the failed serialization is fired.
If pytest is present, the failed serialization gets raised as an exception.
Thus this added test ensures that the expected exception is raised and
assumes that the correct event will be fired normally.

* Log warning when event serialization fails in `msg_to_dict`

This commit updates the `msg_to_dict` exception handling path to
fire a warning level event instead of raising an exception.
Truthfully, we're not sure if this exception handling path is even
possible to hit. That's because we recently switched from betterproto
to google's protobuf. However, exception path is the subject of
[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113). Though we
don't think it's actually possible to hit it anymore, we still want
to handle the case if it is.

* Update serialization failure note to be a warn level event in `BaseEvent`

[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113) wants
logging messages about event serialization failure to be `WARNING`
level events. This does that.

* Add changie info for changes

* Add test to check exception handling of `msg_to_dict`
2023-04-05 19:11:03 -07:00
Kshitij Aranke
ce5d02569f Add kwargs support to dbtRunner (#7274) 2023-04-05 09:40:12 -07:00
Jeremy Cohen
4fc7456000 Small logging fix: stringify datetimes in messages (#7257)
* Stringify datetimes for logging messages

* Add changelog
2023-04-05 15:33:45 +02:00
Kshitij Aranke
28e3412556 New command: dbt show (#7208) 2023-04-04 16:28:22 -07:00
Gerda Shank
86fe510bcf CT 2346 fix read and check versions (#7253) 2023-04-04 18:13:14 -04:00
dependabot[bot]
eaedbd3187 Bump python from 3.11.1-slim-bullseye to 3.11.2-slim-bullseye in /docker (#7196)
* Bump python from 3.11.1-slim-bullseye to 3.11.2-slim-bullseye in /docker

Bumps python from 3.11.1-slim-bullseye to 3.11.2-slim-bullseye.

---
updated-dependencies:
- dependency-name: python
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-04-04 09:31:43 -04:00
Ryan Hill
b31fcc4edf Add double to float types for columns (#6897)
* add double to float types for columns

* add changie item
2023-04-03 14:51:39 -07:00
Emily Rockman
edb5634b9a remove deprecated exception functions (#7251)
* remove deprecated exception functions

* fully remove deprecated exception functions
2023-04-03 09:43:24 -05:00
Gerda Shank
ad21458e10 Fix DepsNotifyUpdatesAvailable logging event (#7262) 2023-04-03 10:11:53 -04:00
dependabot[bot]
622bc43ced Bump black from 22.12.0 to 23.3.0 (#7243)
* Bump black from 22.12.0 to 23.3.0

Bumps [black](https://github.com/psf/black) from 22.12.0 to 23.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.12.0...23.3.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-03-31 09:55:56 -04:00
FishtownBuildBot
e5d99da0bc [Automated] Merged prep-release/1.5.0b5_4566650829 into target main during release process 2023-03-30 11:53:45 -05:00
Github Build Bot
618499b379 Bumping version to 1.5.0b5 and generate changelog 2023-03-30 16:21:36 +00:00
Gerda Shank
bca361acf9 Remove unnecessary "version" keys in partial parsing pp_dict (#7238) 2023-03-29 16:15:57 -04:00
Ian Knox
567e2ca2be [Missed Commit] Replace tracking events: project_id, adapter_info (#7240) 2023-03-29 14:18:53 -05:00
Ian Knox
474143466f Replace tracking events: project_id, adapter_info (#7231) 2023-03-29 09:41:22 -05:00
Gerda Shank
050161c78f Detect breaking changes to column names and data types in state:modified check (#7216) 2023-03-28 19:54:55 -04:00
Emily Rockman
ab496af1f0 convert contract to dict (#7222)
* first pass at changes before modifying tests

* test updates

* add default

* update manifest

* fix tests

* changelog

* fix unit tests

* rename strict -> enforced

* convert to object

* fix tests

* Update Under the Hood-20230217-105223.yaml

* Update Under the Hood-20230217-105223.yaml

* remove stray breakpoints

* move Contract definition to model_config

* fix contract reference
2023-03-28 17:21:03 -05:00
Doug Beatty
c3c2b27e97 Allow --select and --exclude multiple times (#7169)
* One argument per line

* Tests for multiple `--select` or `--exclude`

* Allow `--select` and `--exclude` multiple times

* Changelog entry

* MultiOption options must be specified with type=tuple or type=ChoiceTuple

* Testing for `--output-keys` and `--resource-type`

* Validate that any new param with `MultiOption` should also have `type=tuple` (or `ChoiceTuple`) and `multiple=True`
2023-03-28 10:50:55 -06:00
Emily Rockman
5789d717ba Enforce model contracts for incremental materializations (#7167)
* first pass

* adding tests

* changelog

* split up tests due to order importance

* update test

* add back comment

* rename base test classes

* move sql

* fix test name

* move sql

* test changes to match main

* organize and cleanup fixtures

* more cleanup of tests
2023-03-28 11:25:24 -05:00
Vyacheslav
14e2c3ec21 CT-2129 Bug: added timeout for dbt version (#7009)
* CT-2129: added timeout for dbt version

* CT-2129: updated CHANGELOG.md

* delete the files added under core/dbt/.changes
2023-03-27 17:13:28 -07:00
Gerda Shank
b718c537a7 Raise upper pin for hologram to 0.0.16 (#7221) 2023-03-27 13:40:25 -04:00
Stu Kilgore
6992151081 Add postflight to handle click exc and exit codes (#7212) 2023-03-27 11:49:37 -05:00
Matthieu Blais
bf5ed39db3 Filter out already existing edges in the subgraph created while building the job queue (#7192) 2023-03-24 13:08:18 -05:00
Gerda Shank
f573870232 Use function for datetime in CommandCompleted and throw error when testing (#7217) 2023-03-24 10:52:01 -04:00
Quigley Malcolm
da4a90aa11 CT-1928: dbtRunner to EventManager callback support (#7214)
* add utility function to EventManager for explicitly adding callbacks

Technically these aren't necessary in their current state. We could instead
have people do `<InstantiatedEventManager>.callbacks.extend(...)` directly.
However, it's not hard to imagine a world wherein extra things need to take
place when a callback is added. Thus abstracting to a utility method
now means that as the implementation of how callbacks are actually added
changes, the invocation to do so can stay the same.

* update `setup_event_logger` to optionally take in callbacks add them to the EventManager

* update preflight decorator to check for and pass along callbacks for event logger setup

* Add `callbacks` to `dbtRunner`

On instantiation of `dbtRunner` one can now provide `callbacks`. These
callbacks are for the `EventLogger`. When `invoke` is called on a `dbtRunner`,
the `callbacks` are added to the cli context object. In the preflight
decorator these callbacks are extracted from the cli context and then
passed to the `setup_event_logger`, finally `setup_event_logger` ensures
the callbacks are added to the global `EVENT_MANAGER`.

* add test to check dbtRunner callbacks get properly set

I believe technically this tests qualifies as more of an integration
test, but no other tests like it currently exist (that I could find
via a cursory search). The `tests/unit/test_dbt_runner.py` seemed like
the most intuitive spot. However, if somewhere else makes sense, I'd be
happy to move it.

* add changie documentation for CT-1928
2023-03-22 17:32:01 -07:00
Mila Page
2cfc386773 Convert simple copy. (#7205)
* Convert simple copy.

* Adjust class names for import.

* adjust test namespacing

* Resolve test error.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-03-22 15:53:07 -07:00
Gerda Shank
ae485f996a CT 1998 use google protobuf to enable more flexible dictionaries (#7190) 2023-03-22 15:59:50 -04:00
Peter Webb
73ff497200 ct-2198: Unify constraints and check_constraints fields (#7130)
* ct-2198: clean up some type names and uses

* CT-2198: Unify constraints and constraints_check properties on columns

* Make mypy version consistently 0.981 (#7134)

* CT 1808 diff based partial parsing (#6873)

* model contracts on models materialized as views (#7120)

* first pass

* rename tests

* fix failing test

* changelog

* fix functional test

* Update core/dbt/parser/base.py

* Update core/dbt/parser/schemas.py

* Create method for env var deprecation (#7086)

* update to allow adapters to change model name resolution in py models (#7115)

* update to allow adapters to change model name resolution in py models

* add changie

* fix newline adds

* move quoting into macro

* use single quotes

* add env DBT_PROJECT_DIR support #6078 (#6659)

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

* Add new index.html and changelog yaml files from dbt-docs (#7141)

* Make version configs optional (#7060)

* [CT-1584] New top level commands: interactive compile (#7008)

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>

* CT-2198: Add changelog entry

* CT-2198: Fix tests which broke after merge

* CT-2198: Add explicit validation of constraint types w/ unit test

* CT-2198: Move access property, per code review

* CT-2198: Remove a redundant macro

* CT-1298: Rework constraints to be adapter-generated in Python code

* CT-2198: Clarify function name per review

---------

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Stu Kilgore <stu.kilgore@dbtlabs.com>
Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
Co-authored-by: Leo Schick <67712864+leo-schick@users.noreply.github.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: FishtownBuildBot <77737458+FishtownBuildBot@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-03-22 13:08:06 -04:00
Quigley Malcolm
9a7305d43f CT-2049: Add CommandCompleted event (#7180)
* add protobuf message/class for new CommandCompleted event

For [CT-2049](https://github.com/dbt-labs/dbt-core/issues/6878) we
concluded that we wanted a new event type, [CommandCompleted](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1419718606)
with [four (4) values](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1426118283):
which command was run, whether the command succeeded, the timestamp
that the command finished, and how long the command took. This commit
adds the new event proto defition, the auto generated proto_types, and
the instantiatable even type.

* begin emitting CommandCompleted event in the preflight decorator

The [preflight decorator](4186f99b74/core/dbt/cli/requires.py (L19))
runs at the start of every CLI invocation. Thus is a perfect candidate
for emitting the CommandCompleted event. This is noted in the [dicussion
on CT-2049](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1428643539).

* add CommandCompleted event to event unit tests

* Add: changelog entry

* fire CommandCompleted event reguardless of upstream exceptions

Previously, if `--fail-fast` was specified and an issue was run into
or an unhandled issue became an exception, the CommandCompleted event
would not get fired because at this point in the stack we'd be in
exception thrown handling mode. If an exception does reach this point,
we want to still fire the event and also continue to propogate the
exception. Hence the bare `raise` exists to reraise the caught exception

* Update CommandCompleted event to be a `Debug` level event

We don't actually "always" need this event to be logged. Thus we've
updated it to `Debug` level. [Discussion Context](https://github.com/dbt-labs/dbt-core/pull/7180#discussion_r1139281963)
2023-03-22 08:45:11 -07:00
Emily Rockman
ca23148908 Stop ignoring test directory for precommit (#7201)
* reformat test directory to pass formatting checks

* remove test comment
2023-03-22 08:04:13 -05:00
Jeremy Cohen
8225a009b5 Add deprecation warnings for log-path, target-path in dbt_project.yml (#7185)
* Add deprecation warnings for log-path, target-path in dbt_project.yml

* Fix tests/unit/test_events

* Fix failing tests

* PR feedback
2023-03-21 22:31:15 +01:00
Emily Rockman
9605b76178 update workflow to install dev requirements and remove action deprecations (#7203) 2023-03-21 10:41:33 -05:00
Stu Kilgore
137dd9aa1b Deprecate more env vars (#7175) 2023-03-20 11:51:32 -05:00
Gerda Shank
a203fe866a CT 2196, CT2121 constraints column order (#7161) 2023-03-19 19:24:07 -04:00
FishtownBuildBot
4186f99b74 [Automated] Merged prep-release/1.5.0b4_4438341695 into target main during release process 2023-03-16 10:30:09 -05:00
Github Build Bot
6db899eddd Bumping version to 1.5.0b4 and generate changelog 2023-03-16 14:52:41 +00:00
dependabot[bot]
8ea20b4ba2 Update pathspec requirement from <0.11,>=0.9 to >=0.9,<0.12 in /core (#6737)
* Update pathspec requirement from <0.11,>=0.9 to >=0.9,<0.12 in /core

Updates the requirements on [pathspec](https://github.com/cpburnz/python-pathspec) to permit the latest version.
- [Release notes](https://github.com/cpburnz/python-pathspec/releases)
- [Changelog](https://github.com/cpburnz/python-pathspec/blob/master/CHANGES.rst)
- [Commits](https://github.com/cpburnz/python-pathspec/compare/v0.9.0...v0.11.0)

---
updated-dependencies:
- dependency-name: pathspec
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2023-03-16 10:24:04 -04:00
FishtownBuildBot
3f76f82c88 Add new index.html and changelog yaml files from dbt-docs (#7174) 2023-03-15 12:21:01 -05:00
dependabot[bot]
6cbf66db58 Bump python from 3.10.7-slim-bullseye to 3.11.1-slim-bullseye in /docker (#6424)
* Bump python from 3.10.7-slim-bullseye to 3.11.1-slim-bullseye in /docker

Bumps python from 3.10.7-slim-bullseye to 3.11.1-slim-bullseye.

---
updated-dependencies:
- dependency-name: python
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-03-14 23:05:25 -04:00
dependabot[bot]
8cd11b380f Bump black from 22.10.0 to 22.12.0 (#6425)
* Bump black from 22.10.0 to 22.12.0

Bumps [black](https://github.com/psf/black) from 22.10.0 to 22.12.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.10.0...22.12.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-03-14 23:04:11 -04:00
Emily Rockman
814eb65d59 Support contract config outside model yaml (#7148)
* first pass

* next pass

* works on local project

* fix tests

* changelog

* remove comment

* update error message

* format message

* update tests
2023-03-14 10:19:14 -05:00
Emily Rockman
f24452a3ab use timezone with no DST in test (#7159) 2023-03-13 12:34:14 -05:00
Kshitij Aranke
30503697f2 [CT-1584] New top level commands: interactive compile (#7008)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-03-10 17:58:33 -08:00
dave-connors-3
90902689c3 Make version configs optional (#7060) 2023-03-10 10:32:29 -05:00
FishtownBuildBot
5a0e776cff Add new index.html and changelog yaml files from dbt-docs (#7141) 2023-03-09 17:23:38 -08:00
Leo Schick
9368e7a6a1 add env DBT_PROJECT_DIR support #6078 (#6659)
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-03-09 10:41:52 -08:00
colin-rogers-dbt
c02ddf8c0e update to allow adapters to change model name resolution in py models (#7115)
* update to allow adapters to change model name resolution in py models

* add changie

* fix newline adds

* move quoting into macro

* use single quotes
2023-03-08 15:33:32 -08:00
Stu Kilgore
64b8a12a42 Create method for env var deprecation (#7086) 2023-03-08 11:07:44 -06:00
Emily Rockman
e895fe9e4b model contracts on models materialized as views (#7120)
* first pass

* rename tests

* fix failing test

* changelog

* fix functional test

* Update core/dbt/parser/base.py

* Update core/dbt/parser/schemas.py
2023-03-07 16:27:25 -06:00
Gerda Shank
8d987521dd CT 1808 diff based partial parsing (#6873) 2023-03-07 16:37:38 -05:00
Gerda Shank
4aafc5ef4a Make mypy version consistently 0.981 (#7134) 2023-03-07 15:12:01 -05:00
Alexander Butler
24ca76ea58 [Feature] Add unix-style fqn wildcard selector method (#6599)
resolves https://github.com/dbt-labs/dbt-core/issues/6598
2023-03-05 06:36:17 -08:00
Michelle Ark
b681908ee2 get_column_schema_from_query macro (#6986)
Add adapter.get_column_schema_from_query
2023-03-03 14:21:22 -05:00
FishtownBuildBot
72076b3fe5 [Automated] Merged prep-release/1.5.0b3_4316612471 into target main during release process 2023-03-02 12:02:33 -06:00
Github Build Bot
0683c59dcd Bumping version to 1.5.0b3 and generate changelog 2023-03-02 17:31:25 +00:00
Stu Kilgore
8019498f09 Remove cli doc generation workflow (#7089) 2023-03-01 10:59:39 -06:00
FishtownBuildBot
6234aec7d2 [Automated] Merged prep-release/1.5.0b2_4298598835 into target main during release process 2023-02-28 18:43:05 -06:00
Github Build Bot
edd8059eb3 Bumping version to 1.5.0b2 and generate changelog 2023-03-01 00:14:39 +00:00
Jeremy Cohen
e3be347768 Roadmap update (Feb 2023) (#7091)
* Init roadmap

* Rework the top paragraph

* Clean-up the whole thing

* Typos and stuff

* Add a missing word

* Fix typo

* Update "when" note

* Next draft

* Propose rename

* Resolve TODOs, still needs a reread

* Being cute

* Another read through

* Fix sentence fragment

---------

Co-authored-by: Florian Eiden <florian.eiden@dbtlabs.com>
2023-02-28 18:48:29 -05:00
FishtownBuildBot
597acf1fa1 Add new index.html and changelog yaml files from dbt-docs (#7092) 2023-02-28 16:11:37 -06:00
Gerda Shank
effa1a0813 Move check of invalid groups earlier in parsing (#7090) 2023-02-28 17:05:20 -05:00
Stu Kilgore
726800be57 Make output_keys param MultiOption (#7068) 2023-02-28 09:29:58 -06:00
Sam Debruyn
8b79747908 fix: add pytz dependency (#7077) 2023-02-28 08:05:28 -06:00
Gerda Shank
ec5d31de0e CT 1993 handle invalid access to private models (#7069) 2023-02-27 17:33:47 -05:00
Emily Rockman
5d61ebbfdb add group for selection logic (#7047)
* add group for selection logic

* changelog

* add more tests

* Update .changes/unreleased/Features-20230224-134533.yaml

* fix broken tests
2023-02-27 15:27:12 -06:00
Gerda Shank
0ef9931d19 CT 1990 add access property to parsed nodes (#7007)
Update manifest schema to version 9
2023-02-24 17:13:09 -05:00
Peter Webb
a2213abbc0 Paw/ct 1844 log params take 2 (#6994)
* CT-1844: Add flags and flag-interaciton tests for more granular log control.

* CT-1844: Apply parameter settings to log configuration

* CT-1844: Fixes per testing

* CT-1844: Type annotation fix.

* Add generated CLI API docs

* CT-1844: Add changelog entry.

* Add generated CLI API docs

* CT-1844: Add new logging parameters to UserConfig

* Add generated CLI API docs

* CT-1844: Code review fixes

* Add generated CLI API docs

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-24 09:18:14 -05:00
Jeremy Cohen
915585c36e Fix regression in semver comparison logic (#7040) 2023-02-23 15:47:22 -08:00
Michelle Ark
5ddd40885e parse group config on groupable nodes (#6965)
parse group config on groupable nodes
2023-02-23 14:17:32 -05:00
Gerda Shank
58d1bccd26 Rename "constraints_enabled" to "contract" (#7002)
* Rename "constraints_enabled" to "contract"

* Changie
2023-02-23 12:16:06 -05:00
Gerda Shank
70c26f5c74 CT 2057 Fix compilation logic for ephemeral nodes (#7023)
* Don't overwrite sql in extra_ctes when compiling (rendering) nodes
2023-02-22 17:24:10 -05:00
dependabot[bot]
ac962a4a31 Bump mypy from 0.971 to 0.981 (#5937)
* Bump mypy from 0.971 to 0.981

Bumps [mypy](https://github.com/python/mypy) from 0.971 to 0.981.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.971...v0.981)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

* add type to self

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-02-22 10:26:55 -06:00
Chenyu Li
bb2d062cc5 remove old pass through method (#7019) 2023-02-21 13:44:11 -08:00
José Duarte
7667784985 Remove trailing slashes from source paths (#6102) (#6179)
* Remove trailing slashes from source paths (#6102)

* Run changie

* Handle mypy complaints

* Revert format
2023-02-21 10:50:00 -06:00
Emily Rockman
05ecfbcc3a rename workflow to match other files (#7000) 2023-02-21 09:40:27 -06:00
Emily Rockman
e06ae97068 fix regex for completeness (#7014) 2023-02-21 08:32:13 -06:00
FishtownBuildBot
ed50877c4f Add new index.html and changelog yaml files from dbt-docs (#7015) 2023-02-20 16:25:14 -06:00
Brice Loustau
6b5e38ee28 Made project version optional (default None) (#6718)
* Made project version optional (default None)

* Changie

* Removed version from fixture dbt_project_yml

* Linting
2023-02-20 08:40:08 -06:00
Jeremy Cohen
63a1bf9adb Update comment for build_manifest_state_check (#7004) 2023-02-19 17:45:59 +01:00
FishtownBuildBot
2c7238fbb4 [Automated] Merged prep-release/1.5.0b1_4206349891 into target main during release process 2023-02-17 12:39:27 -06:00
colin-rogers-dbt
b1d597109f move hook tests to adapter zone (#6952)
* move hook tests to adapter zone

* test update to sql

* fix fixtures

* Add generated CLI API docs

* remove port

* fix column name

* fix missing "host" in profile

* fixture fix

* fixture fix

* fixture/test fix

* Add generated CLI API docs

* fix fixtures

* remove host assumption from test_run_hooks.py

* undo bumpversion.cfg change

* clean up string interpolation and get method

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-17 10:35:21 -08:00
Github Build Bot
7617eece3a Bumping version to 1.5.0b1 and generate changelog 2023-02-17 18:08:59 +00:00
Gerda Shank
8ce92b56d7 CT 2026 test proto serialization of logging events (#6995)
* Fix unit test, functional tests, update structured logging schema check

* Changie

* Remove unneeded cast_to_str
2023-02-17 13:02:51 -05:00
FishtownBuildBot
21fae1c4a4 Add new index.html and changelog yaml files from dbt-docs (#7001) 2023-02-17 09:03:53 -07:00
Chenyu Li
c952d44ec5 fix small issues with flags and logger to support dbt-rpc (#6990) 2023-02-16 16:58:12 -08:00
Doug Beatty
971b38c26b Use IEC standard abbreviations (GiB, TiB, etc) (#6742)
* Use IEC standard abbreviations (GiB, TiB, etc)

* Fix failing test cases

* Remove  accidentally left behind BigQuery-specific methods

* Changelog entry
2023-02-16 14:14:19 -07:00
Michelle Ark
b7884facbf parse group resource (#6921)
parse group resource
2023-02-15 16:49:47 -05:00
Callum McCann
57ce461067 Fixing exposure error (#6934)
* fixing exposure error message

* adding changie
2023-02-15 14:00:20 -06:00
Kshitij Aranke
b1b830643e Revert "Hoist dbt.cli.main into dbt.main namespace (#6939)" (#6987) 2023-02-15 10:11:09 -08:00
Niels Pardon
3cee9d16fa Make use of hashlib.md5() FIPS compliant (#6982)
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
2023-02-15 11:22:25 -05:00
Sung Won Chung
c647706ac2 dbt Constraints / model contracts (#6271)
* start off the blueprints

* test commit

* working snowflake env

* update manifest expectation

* add error handling

* clean up language

* constraints validator

* cleaner example

* better terminal output

* add python error handling

* add to manifest schema

* add to schema config

* clean up comments

* backwards compatible nodeconfig

* remove comments

* clean up more comments

* add changelog

* clarify error message

* constraints list type

* fix grammar

* add global macros

* clearer compile error

* remove comments

* fix tests in this file

* conditional compile errors

* add conditional check in ddl

* add macro to dispatch

* fix regressions in parsed

* fix regressions in manifest tests

* fix manifest test regressions

* fix test_list regressions

* concise data_type terminal error

* remove placeholder function

* fix failed regressions finally

* Revert "Merge branch 'main' of https://github.com/dbt-labs/dbt into dbt-constraints"

This reverts commit 92d2ea7ff5, reversing
changes made to 5d59cc17bc.

* Revert "Revert "Merge branch 'main' of https://github.com/dbt-labs/dbt into dbt-constraints""

This reverts commit 4f747b0bc0.

* remove tmp.csv

* template test plans

* postgres columns spec macro

* schema does not exist error handling

* update postgres adapter

* remove comments

* first passing test

* fix postgres macro

* add more passing tests

* Add generated CLI API docs

* add disabled config test

* column configs match

* test python error handling

* adjust macro with rollback

* start postgres tests

* remove begin commit

* remove begin commit comments

* passing expected compiled sql test

* passing rollback test

* update changelog

* fix artifacts regression

* modularize validator

* PR feedback

* verify database error occurs

* focus on generic outcomes

* fix global macro

* rename to constraints_check

* missed a check rename

* validate at parse time

* raise error for modelparser only

* better spacing in terminal output

* fix test regressions

* fix manifest test regressions

* these are parsing errors now

* fix tests

* test passes in json log format

* add column compile error handling

* update global macros for column handling

* remove TODO

* uppercase columns for consistency

* more specific error handling

* migrate tests

* clean up core tests

* Update core/dbt/include/global_project/macros/materializations/models/table/columns_spec_ddl.sql

For whitespace control in the dbt.log file!

Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>

* Revert "Update core/dbt/include/global_project/macros/materializations/models/table/columns_spec_ddl.sql
"

This reverts commit 59b02980dc.

* update for pre-commit hooks

* update for black formatter

* update for black formatter on all files

* Refactor functional tests

* Fixup formatting

* Dave feedback

* another one - dave

* the hits keep coming

* adjust whitespace

* Light touchup

* Add more flexibility for spark

* Nearly there for spark

* Try regenerating docs

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Dave Connors <dave.connors@fishtownanalytics.com>
2023-02-15 16:50:39 +01:00
Emily Rockman
7b33ffb1bd point at main (#6973) 2023-02-14 14:17:28 -06:00
Chenyu Li
f38cbc4feb add helper function for load project/profile (#6964) 2023-02-14 08:47:43 -08:00
Gerda Shank
480e0e55c5 Set relation_name in tests at compile time (#6949) 2023-02-14 11:32:27 -05:00
Michelle Ark
e5c468bb93 dbt_project.yml log-path respects project directory (#6908)
dbt_project.yml log-path respects project directory
2023-02-14 11:30:11 -05:00
Emily Rockman
605c72e86e Automate cutting new .latest branches (#6854)
* first pass

* WIP

* add notes/stubs on more pieces

* more work

* more cleanup

* cleanup

* add more cleanup and generalization

* update to use reusable workflow

* add TODO
2023-02-14 08:09:16 -06:00
Emily Rockman
aad46ac5a8 Changelog Cleanup from feature branch merge (#6935)
* remove changelogs from 1.4

* remove yaml files for already released changes
2023-02-14 08:08:50 -06:00
Emily Rockman
d85618ef26 remove +nightly from version parsing (#6966)
* remove  from version parsing

* update comment

* undo partial version bump
2023-02-14 08:04:26 -06:00
Jeremy Cohen
1250f23c44 Fix petits logging regressions from feature/click-cli merge (#6940)
* Add back initialization events

* Fix log_cache_events. Default stdout logger knows less than it used to

* Add back exception handling events

* Revert "Add back exception handling events"

This reverts commit 26f22d91b660f51f0df6a59f9e5cae16b0ee6fe5.

* Add changelog entry

* Fix test by stringifying dict values

* Add generated CLI API docs

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-14 10:48:56 +01:00
Gerda Shank
daea7d59a7 CT 1921 add missing nodes to previous state test (#6961)
* Update WritableManifest.disabled to include metrics and exposures in disabled dictionary
2023-02-13 21:02:20 -05:00
Emily Rockman
4575757c2a remove '+nightly' from release test version (#6907)
* remove '+nightly' from release test version

* fix release branch variable name

* remove comment
2023-02-13 19:41:34 -06:00
Emily Rockman
d7a2f77705 Infer core team membership in changie (#6420)
* part 1 of env var for core team

* add logic to use env vars to generate changelog

* modify version bump to add members via env var

* pull in main and tweak

* add token

* changes for testing

* split step

* remove leading slash

* add version check

* more debugging

* try curl

* try more things

* try more things

* chnage auth

* put back token

* update permissions

* add back fishtown pat

* use new pat

* fix typo

* swap token

* comment out list teams

* change url

* debug path

* add continue

* change core case

* more tweaks

* send output to file

* add file view

* make array

* tweak

* remove []

* add quotes

* add tojson

* add quotes to set

* tweak

* fix id

* tweaks

* more

* more

* remove new lines

* more tweaks

* update to generate changelog

* remove debugging bits

* use central version-bump

* use correct author list

* testing with changelog team automation

* add new token to input

* move secret

* remove testing aspects from workflow

* clean up team logic

* explicitly send secret

* move bumpversion comment

* move comments

* point workflow back tp main

* point to branch for testing

* point back to main

* inherit secrets
2023-02-13 14:08:49 -06:00
Kshitij Aranke
4a4b89606b Hoist dbt.cli.main into dbt.main namespace (#6939) 2023-02-13 12:06:28 -08:00
Michelle Ark
1ebe2e7118 ExposureOwner: Rename to Owner, accept additional fields, require one of 'name' or 'email' (#6912)
update ExposureOwner to require one of email/name, allow additional fields
2023-02-13 10:13:15 -05:00
Gerda Shank
f1087e57bf Fix types.proto and test_events.py (#6947) 2023-02-10 16:31:34 -05:00
Michelle Ark
250537ba58 handle_and_check in main.py (#6941)
handle_and_check in cli/main.py
2023-02-10 15:39:31 -05:00
Peter Webb
ccc7222868 CT-2079: Re-apply accidentally reverted logging fix (#6937) 2023-02-10 14:40:57 -05:00
Chenyu Li
311a57a21e Merge pull request #6931 from dbt-labs/merge_feature_click_cli_main
Merge feature/click-cli into main

Co-authored-by: Ian Knox <ian.knox@dbtlabs.com >
Co-authored-by: Stu Kilgore <stu.kilgore@dbtlabs.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com >
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com >
Co-authored-by: Jeremy Cohen <jeremey@dbtlabs.com >
2023-02-09 18:03:04 -08:00
Chenyu Li
b7c45de6b1 remove old test 2023-02-09 15:55:24 -08:00
Chenyu Li
c53c3cf181 remove old main 2023-02-09 15:53:37 -08:00
Chenyu Li
a77d325c8a fix all tests 2023-02-09 15:51:42 -08:00
Kshitij Aranke
dd41384d82 revert core.dbt.contracts back to code from main 2023-02-09 15:32:34 -08:00
Github Build Bot
aa55fb2d30 Add generated CLI API docs 2023-02-09 22:59:54 +00:00
Kshitij Aranke
864f4efb8b Merge feature/click-cli into main 2023-02-09 14:57:10 -08:00
dependabot[bot]
83c5a8c24b Bump ubuntu from 22.04 to 23.04 (#6865)
* Bump ubuntu from 22.04 to 23.04

Bumps ubuntu from 22.04 to 23.04.

---
updated-dependencies:
- dependency-name: ubuntu
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-09 16:59:25 -05:00
Kshitij Aranke
57aef33fb3 Click cli main merge (#6926) 2023-02-09 12:21:39 -08:00
FishtownBuildBot
6d78e5e640 Add most recent dbt-docs changes (#6923)
* Add new index.html and changelog yaml files from dbt-docs

* Update .changes/unreleased/Docs-20230209-082901.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-02-09 13:23:12 -06:00
Ryan Harris
f54a876f65 Update link to dbt install docs (#6883)
* update dbt install link

* add changelog entry
2023-02-09 09:35:33 -08:00
Kshitij Aranke
8bbae7926b Fix Project Env Var Tests (#6916)
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2023-02-09 08:53:39 -08:00
Jeremy Cohen
db2b12021e Fix test_builtin_invocation_args_dict_function (#6898)
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2023-02-09 09:53:42 -06:00
Peter Webb
8b2c9bf39d Ensure flush() after logging write() (#6909)
* ct-2063: Ensure flush after logging, by using Python's logging subsystem directly

* ct-2063: Add changelog entry
2023-02-09 09:37:37 -05:00
Jeremy Cohen
298bf8a1d4 Add back depends_on for seeds - only macros, never nodes (#6851)
* Extend functional tests for seeds w hooks

* Add MacroDependsOn to seeds, raise exception for other deps

* Add changelog entry

* Fix unit tests

* Update upgrade_seed_content

* Cleanup

* Regen manifest v8 schema. Fix tests

* Be less magical

* PR feedback
2023-02-09 10:56:12 +01:00
Michelle Ark
77748571b4 profiles dir exists=False for dbt debug (#6910)
profiles dir exists=False for dbt debug
2023-02-08 18:32:06 -05:00
Stu Kilgore
8ce4c289c5 Docs generate doesn't write manifest (#6905) 2023-02-08 16:16:31 -06:00
Emily Rockman
abbece8876 1.4 regression: Check if status has node attribute (#6899)
* check for node

* add changelog

* add test for regression
2023-02-08 13:49:24 -06:00
dave-connors-3
3ad40372e6 add base class for merge exclude tests (#6700)
* add base class for merge exclude tests

* changie <33

* remove comments

* add comments to sql, remove and clarify contents of resultholder
2023-02-08 10:20:13 -08:00
Kshitij Aranke
c6d0e7c926 Fix Click CLI test DB Name (#6895) 2023-02-08 09:58:16 -08:00
Michelle Ark
bc015843d4 use UnsetProfile in deps and clean commands (#6890)
use unset_profile in deps and clean commands
2023-02-08 12:26:58 -05:00
Emily Rockman
df64511feb Dynamically list all .latest branches for scheduled testing (#6682)
* first pass at automating latest branches

* checkout repo first

* fetch all history

* reorg

* debugging

* update test id

* swap lines

* incorporate new branch aciton

* tweak vars
2023-02-08 08:02:21 -06:00
Michelle Ark
db0981afe7 NoneConfig for DebugTask (#6893)
NoneConfig for DebugTask
2023-02-07 21:33:15 -05:00
Michelle Ark
dcf6544f93 flags.THREADS defaults to None (#6887)
flags.THREADS defaults to None
2023-02-07 17:15:26 -05:00
Ian Knox
c2c8959fee Merge main to feature (#6817)
merge main to feature/click-cli
2023-02-07 16:32:31 -05:00
Peter Webb
ccb4fa26cd CT-1917: Fix a regression in the behavior of the -q/--quiet cli parameter (#6886) 2023-02-07 16:15:42 -05:00
Chenyu Li
d0b5d752df consolidate flags (#6788)
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-02-07 08:35:35 -08:00
Neelesh Salian
4c63b630de [CT-1959]: moving simple_seed tests to adapter zone (#6859)
* Formatting

* Changelog entry

* Rename to BaseSimpleSeedColumnOverride

* Better error handling

* Update test to include the BOM test

* Cleanup and formating

* Unused import remove

* nit line

* Pr comments
2023-02-06 19:51:32 -08:00
Kshitij Aranke
9c0b62b4f5 Fix CLI vars test to check for object instead of string (#6850) 2023-02-06 15:13:40 -08:00
Ian Knox
e08eede5e2 Remove unused cli_runner (#6877) 2023-02-06 15:24:09 -06:00
Ian Knox
05e53d4143 Test fix: TestProfileEnvVars::test_profile_env_vars (#6856) 2023-02-06 10:49:40 -06:00
colin-rogers-dbt
b2ea2b8b25 move test_store_test_failures.py to adapter zone (#6816) 2023-02-02 10:55:09 -08:00
Emily Rockman
2245d8d710 update regex to match all iterations (#6839)
* update regex to match all iterations

* convert to num to match all adapters

* add comments, remove extra .

* clarify with more comments

* Update .bumpversion.cfg

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>

---------

Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
2023-02-02 12:16:06 -06:00
Gerda Shank
d9424cc710 CT 2000 fix semver prerelease comparisons (#6838)
* Modify semver.py to not use packaging.version.parse

* Changie
2023-02-02 12:22:48 -05:00
Stu Kilgore
0503c141b7 Lazily call --version (#6813)
* Lazily call --version

* Add generated CLI API docs

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-31 17:20:18 -08:00
Mila Page
1a6e4a00c7 Add clearer directions for custom test suite vars in Makefile. (#6764)
* Add clearer directions for custom test suite vars in Makefile.

* Fix up PR for review

* Fix erroneous whitespace.

* Fix a spelling error.

* Add documentation to discourage makefile edits but provide override tooling.

* Fix quotation marks. Very strange behavior

* Compact code and verify quotations happy inside bash and python.

* Fold comments into Makefile.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-31 13:53:55 -08:00
Mila Page
42b7caae19 Ct 1827/064 column comments tests conversion (#6766)
* Convert test and make it a bit more pytest-onic

* Ax old integration test.

* Run black on test conversion

* I didn't like how pytest was running the fixture so wrapped it into a closure.

* Merge converted test into persist docs.

* Move persist docs tests to the adapter zone. Prep for adapter tests.

* Fix up test names

* Fix name to be less confusing.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-31 12:58:19 -08:00
Emily Rockman
622e5fd71d fix contributor list generation (#6799) 2023-01-31 13:05:13 -06:00
Stu Kilgore
59d773ea7e Implement --version in click (#6802) 2023-01-31 12:40:01 -06:00
Kshitij Aranke
84bf5b4620 [CT-1947] Alias --models to --select for all commands except dbt ls (#6787)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-31 10:18:10 -08:00
Ian Knox
726c4d6c58 Enable the new Click Cli (#6785) 2023-01-30 19:30:01 -06:00
Michelle Ark
acc88d47a3 mutually exclusive handling for warn_error_options and warn_error params in Click CLI (#6771)
warn_error_options, warn_error mutual exclusivity with click
2023-01-30 18:38:36 -05:00
Chenyu Li
0a74594d09 move favor state arg to click (#6774) 2023-01-30 11:55:31 -08:00
Kshitij Aranke
d2f3cdd6de [CT-1841] Convert custom target test to Pytest (#6765) 2023-01-30 07:55:28 -08:00
Alexander Smolyakov
92d1ef8482 Update release workflow (#6778)
- Update AWS secrets
- Rework condition for Slack notification
2023-01-30 09:19:10 -06:00
Neelesh Salian
a8abc49632 [CT-1940] Stand-alone Python module for PostgresColumn (#6773) 2023-01-27 17:00:39 -08:00
Ian Knox
d6ac340df0 Merge main into feature/click-cli (#6761) 2023-01-27 15:07:30 -06:00
Neelesh Salian
c653330911 Adding nssalian to committers list (#6769) 2023-01-27 11:16:59 -08:00
Alexander Smolyakov
82d9b2fa87 [CI/CD] Update release workflow and introduce workflow for nightly releases (#6602)
* Add release workflows

* Update nightly-release.yml

* Set default `test_run` value to `true`

* Update .bumpversion.cfg

* Resolve review comment

- Update workflow docs
- Change workflow name
- Set `test_run` default value to `true`

* Update Slack secret

* PyPI
2023-01-27 09:04:31 -06:00
Mila Page
3f96fad4f9 Ct 1629/052 column quoting tests conversion (#6652)
* Test converted and reformatted for pytest.

* Ax old versions of 052 test

* Nix the 'os' import and black format

* Change names of models to be more PEP like

* cleanup code

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-26 12:23:02 -08:00
Peter Webb
c2c4757a2b Graph Analysis Optimization for Large Dags (#6720)
* Optimization to remove graph analysis bottleneck in large dags.

* Add changelog entry.
2023-01-26 14:27:42 -05:00
Kshitij Aranke
08b2d94ccd [CT-920][CT-1900] Create Click CLI runner and use it to fix dbt docs … (#6723)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-26 10:42:49 -08:00
Michelle Ark
7fa61f0816 dbt init works with click (#6698)
dbt init works with click
2023-01-26 12:45:15 -05:00
Mila Page
c65ba11ae6 Ct 1827/064 column comments tests conversion (#6654)
* Convert test and make it a bit more pytest-onic

* Ax old integration test.

* Run black on test conversion

* I didn't like how pytest was running the fixture so wrapped it into a closure.

* Merge converted test into persist docs.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-26 00:54:00 -08:00
Matthew Beall
b0651b13b5 change exposure_content to source_content (#6739)
* change `exposure_content` to `source_content`

* Adding changelog

Co-authored-by: Leah Antkiewicz <leah.antkiewicz@fishtownanalytics.com>
2023-01-25 19:51:34 -05:00
Gerda Shank
a34521ec07 CT 1894 log partial parsing var changes and sort cli vars before hashing (#6713)
* Log information about vars_hash, normalize cli_vars before hashing

* Changie

* Add to test_events.py

* Update core/dbt/events/types.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-01-25 17:47:45 -05:00
Matthew McKnight
da47b90503 [CT-1630] Convert Column_types tests (#6690)
* init commit for column_types test conversion

* init start of test_column_types.py

* pass tes macros into both tests

* remove alt tests, remove old tests, push up working conversion

* rename base class, move to adapter zone so adapters can use

* typo fix
2023-01-25 14:57:16 -06:00
Stu Kilgore
d27016a4e7 Migrate debug task to click (#6728) 2023-01-25 14:52:57 -06:00
Peter Webb
db99e2f68d Event Clean-Up (#6716)
* CT-1857: Event cleanup

* Add changelog entry.
2023-01-25 13:51:52 -05:00
Michelle Ark
cbb9117ab9 test_init conversion (#6610)
* convert 044_init_tests
2023-01-24 16:22:26 -05:00
Gerda Shank
e2ccf011d9 CT 1886 include adapter_response in NodeFinished log message (#6709)
* Include adapter_response in run_result in NodeFinished log event

* Changie
2023-01-24 14:25:32 -05:00
Aezo
17014bfad3 add adapter_response for test (#6645)
resolves https://github.com/dbt-labs/dbt-core/issues/2964
2023-01-24 09:58:08 -08:00
Stu Kilgore
92b7166c10 Abstract manifest generation from tasks (#6565) 2023-01-24 11:05:57 -06:00
Peter Webb
7b464b8a49 CT-1718: Add Note and Formatting event types (#6691)
* CT-1718: Add Note and Formatting event types

* CT-1718: Add changelog entry
2023-01-23 16:39:29 -05:00
Sean McIntyre
5c765bf3e2 Cheeky performance improvement on big DAGs (#6694)
* Short-circuit set operations for nice speed boost

* Add changelog

* Fix issue

* Update .changes/unreleased/Under the Hood-20230122-215235.yaml

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-01-23 09:09:09 -07:00
Mila Page
93619a9a37 Ct 738/dbt debug log fix (#6541)
* Code cleanup and adding stderr to capture dbt

* Debug with --log-format json now prints structured logs.

* Add changelog.

* Move logs into miscellaneous and add values to test.

* nix whitespace and fix log levels

* List will now do structured logging when log format set to json.

* Add a quick None check.

* Add a get guard to class check.

* Better null checking

* The boolean doesn't reflect the original logic but a try-catch does.

* Address some code review comments and get us working again.

* Simplify logic now that we have a namespace object for self.config.args.

* Simplify logic for json log format checking.

* Simplify code for allowing our GraphTest cases to pass while also hiding compile stats from dbt ls/list .

* Simplify structured logging types.

* Fix up boolean logic and simplify via De'Morgan.

* Nix unneeded fixture.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-20 16:37:54 -08:00
Doug Beatty
a181cee6ae Improve error message for packages missing dbt_project.yml (#6685)
* Improve error message for packages missing `dbt_project.yml`

* Use black formatting

* Update capitalization of expected error message
2023-01-20 13:46:36 -07:00
Michelle Ark
a0ade13f5a dbt docs generate works with click (#6681) 2023-01-20 15:26:20 -05:00
Michelle Ark
9823a56e1d dbt build works in click (#6680)
* build working with click
2023-01-20 14:27:39 -05:00
Michelle Ark
3aeab73740 convert 069_build_tests (#6678) 2023-01-20 14:27:02 -05:00
Jeremy Cohen
9801eebc58 Consolidate changie entries from #6620 (#6684) 2023-01-20 19:58:40 +01:00
Jeremy Cohen
11c622230c Add run-operation to click CLI (#5552) (#6656)
* Add run-operation to click CLI

* Add changelog entry

* PR feedback

* Fix unit test
2023-01-20 01:58:47 +01:00
Chenyu Li
f0349488ed Seed and freshness works with click (#6651) 2023-01-19 16:14:26 -08:00
Chenyu Li
c85be323f5 fix MP_CONTEXT is not JSON serializable (#6650) 2023-01-19 08:16:58 -08:00
Peter Webb
6954c4df1b CT-1786: Port docs tests to pytest (#6608)
* CT-1786: Port docs tets to pytest

* Add generated CLI API docs

* CT-1786: Comply with the new style requirements

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-19 11:11:17 -05:00
Michelle Ark
30a1595f72 click working with list (#6641)
* list working with click
2023-01-19 09:28:32 -05:00
dave-connors-3
f841a7ca76 add backwards compatibility and default argument for incremental_predicates (#6628)
* add backwards compatibility and default argument

* changie <3

* Update .changes/unreleased/Fixes-20230117-101342.yaml

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-01-19 15:20:19 +01:00
Jeremy Cohen
07a004b301 convert 062_defer_state_tests (#6616)
* Fix --favor-state flag

* Convert 062_defer_state_tests

* Revert "Fix --favor-state flag"

This reverts commit ccbdcbad98b26822629364e6fdbd2780db0c20d3.

* Reformat

* Revert "Revert "Fix --favor-state flag""

This reverts commit fa9d2a09d6.
2023-01-19 11:00:09 +01:00
Jeremy Cohen
b05582de39 mv on_schema_change tests -> "adapter zone" (#6618)
* Mv incremental on_schema_change tests to 'adapter zone'

* Use type_string()

* Cleanup
2023-01-19 10:12:59 +01:00
Jeremy Cohen
fa7c4d19f0 Respect quoting config in dbt-py models (#6620)
* Respect quoting for 'this' in dbt-py models #6619

* Respect quoting for ref/source in dbt-py models #6103

* Add changelog entries
2023-01-19 09:34:08 +01:00
Chenyu Li
1913eac5ed Click snapshot as click command (#5972) (#6640) 2023-01-18 16:01:58 -08:00
Jeremy Cohen
066346faa2 convert 038_caching_tests (#6612)
* convert 038_caching_tests

* Adapt for dbt-snowflake

* PR feedback

* Reformat
2023-01-18 22:37:50 +01:00
Emily Rockman
0a03355ceb update test matrix (#6604) 2023-01-18 14:16:34 -06:00
David Bloss
43e24c5ae6 update gh action set-output variables (#6635)
* update gh action set-output variables

* add changie file
2023-01-18 11:23:13 -06:00
Gerda Shank
89d111a5f6 CT 1440 Fix code to emit ConnectionReused event (#6605)
* Refactor "set_connection_name" to properly handle reused connection

* Update test

* Changie

* Limit test of ConnectionUsed events to non-Windows
2023-01-17 13:18:07 -05:00
Gerda Shank
e1b5e68904 Convert 068_partial_parsing_tests (#6614)
* Convert partial parsing tests

* reformat
2023-01-17 12:22:31 -05:00
Jeremy Cohen
065ab2ebc2 Reformat tests/ (#6622)
* Run black + flake8 on tests dir

* Run pre-commit
2023-01-16 16:39:54 +01:00
Michelle Ark
20c95a4993 convert 057_run_query_tests (#6607) 2023-01-13 15:02:17 -05:00
Michelle Ark
c40b488cb4 convert 037_external_references (#6589) 2023-01-13 14:15:03 -05:00
Chenyu Li
585e7c59e8 migrate ref override (#6600) 2023-01-13 09:15:04 -08:00
Michelle Ark
7077c47551 converting 022_test_timezones (#6588) 2023-01-13 10:52:21 -05:00
Chenyu Li
f789b2535a convert adapter table materialization test (#6595) 2023-01-12 16:05:51 -08:00
Emily Rockman
2bfc6917e2 finish message rename in types.proto (#6594)
* finish message rename in types.proto

* add new parameter
2023-01-12 14:33:56 -06:00
github-actions[bot]
d74ae19523 Bumping version to 1.5.0a1 and generate changelog (#6593)
* Bumping version to 1.5.0a1 and generate CHANGELOG

* make relevvant changelog updates

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-01-12 12:59:28 -06:00
Michelle Ark
1c7c23ac73 convert 043_custom_alias_tests (#6590) 2023-01-12 11:07:26 -05:00
Jeremy Cohen
86e8722cd8 Call update_event_status earlier + rename an event (#6572)
* Rename HookFinished -> FinishedRunningStats

* Move update_event_status earlier when node finishes

* Add changelog entry

* Add update_event_status for skip

* Update changelog entry
2023-01-12 09:34:36 +01:00
Kshitij Aranke
53127daad8 [CT-921] dbt compile works in click (#5545) (#6586)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
resolves https://github.com/dbt-labs/dbt-core/issues/5545
2023-01-11 15:01:50 -08:00
Kshitij Aranke
91b20b7482 dbt test works with Click (#5556)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
resolves https://github.com/dbt-labs/dbt-core/issues/5556
2023-01-11 12:19:43 -08:00
Michelle Ark
7a61602738 Add --warn-error-options (#6520)
* adding --warn-error-options

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-10 19:43:52 -05:00
Doug Beatty
dd4b47d8b1 Check length of escaped strings in the adapter test (#6567)
* Check length of escaped strings in the adapter test

* One column per line to improve readability
2023-01-10 17:26:35 -07:00
Emily Rockman
eb200b4687 Rename exceptions (#6539)
* rename InternalException

* rename RuntimeException

* rename DatabaseException

* rename CompilationException

* cleanup renames in tests and postgres

* rename ValidationException

* rename IncompatibleSchemaException

* more renaming

* more renaming

* rename InternalException again

* convert ParsingException

* replace JSONValidationException and SemverException

* replace VersionsNotCompatibleException

* replace NotImplementedException

* replace FailedToConnectException

* replace InvalidConnectionException

* replace InvalidSelectorException

* replace DuplicateYamlKeyException

* replace ConnectionException

* minor cleanup

* update comment

* more cleanup

* add class decorator

* rename more exceptions

* more renamed, add changelog

* rename exception

* rework class deprecations

* removing testing line

* fix failing test

* rename newer exceptions

* fix failing test

* commit unsaved faile

* convert back an rpc exception

* remove class deprecations
2023-01-10 17:43:27 -06:00
mivanicova
0fc080d222 add defer_to_manifest in before_run to fix faulty deferred docs generate (#6488)
* add defer_to_manifest in before_run to fix faulty deferred docs generate

* add a changelog

* add declaration of defer_to_manifest to FreshnessTask and GraphRunnableTask

* fix: add defer_to_manifest method  to ListTask
2023-01-10 15:18:20 -08:00
AGPapa
5da63602b3 Adds buildable selection mode (#6366) 2023-01-10 15:15:48 -08:00
nshuman1
457ff3ef48 fixed minor typos (#6499)
* fixed minor typos

* adding changelog file

* linter

* Update Docs-20230102-170216.yaml
2023-01-10 13:36:21 -08:00
Gerda Shank
0dbdecef10 Ct 1716 cleanup logging events (#6561)
* Combine DbtProfileError log events

* Combine DbtProjectErrorException with DbtProjectError

* Combine cache logging events

* Changie

* fix ticket number

* Ooops. Add another file.

* fix serialization of profile names
2023-01-10 15:56:52 -05:00
Doug Beatty
b13b0e9492 Re-factor list of YAML keys for hooks to late-render (#6435)
* Re-factor list of YAML keys for hooks to late-render

* Add `pre_` and `post_hook` to list of late-rendered hooks

* Check for non-empty set intersection

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>

* Test functional synonymy of `*_hook` with `*-hook`

Test that `pre_hook`/`post_hook` are functionally synonymous with `pre-hook`/`post-hook` for model project config

* Undo bugfix to validate the new test fails

* Revert "Undo bugfix to validate the new test fails"

This reverts commit e83a2be2eb.

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-01-10 12:35:06 -07:00
Tim Mastny
b9fdfd9e36 Adds the meta field to node_info in logs (#6493)
* add meta attribute to nodeinfo for events

* also add meta to dataclass

* add to unit test to ensure meta is added

* adding functional test to check that meta is passed to nodeinfo during logging

* changelog

* remove used imported

* add tests with non-string keys

* renaming test dict keys

* add non-string value

* resolve failing test

* test additional non-string values

* fix flake8

* Stringify meta dict in node_info

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-01-10 13:42:51 -05:00
Peter Webb
4d6352db14 CT-1645: Bump mashumaro version to receive regression fix, and add unit test to confirm fix. (#6564) 2023-01-10 13:14:24 -05:00
Doug Beatty
9eb82c6497 Remove duplicated changelog entry for 5521 (#6563) 2023-01-10 11:13:50 -07:00
Jeremy Cohen
89cc89dfdf Fix changelog entry for 5989/6432 (#6560) 2023-01-10 06:57:39 -07:00
Kshitij Aranke
2b0f6597a4 [CT-1694] Deprecate event tracking tests (#6538) 2023-01-09 15:23:02 -08:00
Peter Webb
294def205f Fix event level regression (#6556)
* CT-6501: Make certain disabled test events debug-level again to match historical behavior.

* CT-1740: Add changelog entry.
2023-01-09 18:15:59 -05:00
Emily Rockman
34fa703466 fix log msg format (#6557) 2023-01-09 15:56:42 -06:00
Mila Page
ab3f8dcbfd Ct 1517/convert 060 persist docs (#6409)
* convert the test and fix an error due to a dead code seed

* Get rid of old test

* Remove unfortunately added files. Don't use that *

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-01-09 11:07:34 -08:00
Emily Rockman
02c20477b9 add deprecation decorator (#6540)
* add derecation decorator

* fix tests

* updated changelog and deprecation reason
2023-01-09 12:09:12 -06:00
Gerda Shank
d9a4ee126a CT 1549 reorg logging events to have two top level keys (#6553)
* Update types.proto

* pre-commit passes

* Cleanup tests and tweak EventLevels

* Put node_info back on SQLCommit. Add "level" to fire_event function.

* use event.message() in warn_or_error

* Fix logging test

* Changie

* Fix a couple of unit tests

* import Protocol from typing_extensions for 3.7
2023-01-09 11:33:44 -05:00
Michelle Ark
5b31cc4266 Remove UnsetProfileConfig (#6504)
remove UnsetProfileConfig
2023-01-06 20:12:40 -05:00
Chenyu Li
9bb1250869 merge main to feature/click-cli (#6483)
* merge main to feature/click-cli

* fix var hash

* move back changes in feature branch for deps

* fix logging issues
2023-01-06 15:11:23 -08:00
Kshitij Aranke
94d6d19fb4 [CT-1693] Port severity test to Pytest (#6537) 2023-01-06 11:59:53 -08:00
Callum McCann
d43c070007 Making Metric Timestamps Optional (#6402)
* changing to optional

* adding tests

* tests and changie

* pre-commit cleaning

* formatting fixes

* pre-commit update
2023-01-05 15:49:55 -05:00
Callum McCann
9ef236601b Updating metric attributes renamed event (#6518)
* updating message

* adding changie
2023-01-05 15:49:44 -05:00
justbldwn
9d6f961d2b 🎨 add missing fstrings, convert format to fstring (#6243)
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2023-01-05 15:28:27 -05:00
Niall Woodward
5453840950 DBT_FAVOR_STATE_STATE -> DBT_FAVOR_STATE (#6392)
* DBT_FAVOR_STATE_STATE -> DBT_FAVOR_STATE

* add changelog
2023-01-04 15:53:35 -08:00
Gerda Shank
d453964546 CT 1551 partial parsing logging events (#6474) 2023-01-04 14:28:26 -05:00
dependabot[bot]
748a932811 Update agate requirement from <1.6.4,>=1.6 to >=1.6,<1.7.1 in /core (#6506)
* Update agate requirement from <1.6.4,>=1.6 to >=1.6,<1.7.1 in /core

Updates the requirements on [agate](https://github.com/wireservice/agate) to permit the latest version.
- [Release notes](https://github.com/wireservice/agate/releases)
- [Changelog](https://github.com/wireservice/agate/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/wireservice/agate/compare/1.6.0...1.7.0)

---
updated-dependencies:
- dependency-name: agate
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-01-04 09:02:37 -08:00
Jeremy Cohen
8217ad4722 Fix issue link for 5859 changelog entry (#6496) 2023-01-04 10:46:52 +01:00
Michelle Ark
cc5a38ec5a Example click API usage (#6307)
* Example python lib w click, written docs

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-01-03 18:46:34 -05:00
Gerda Shank
6ef3fbbf76 Partial parsing bug with empty schema file - ensure None is not passed to load_yaml_text (#6494) 2023-01-03 17:14:56 -05:00
justbldwn
76fd12c7cd adding pre-commit install to make dev (#6417)
*  adding pre-commit install to make dev

* 🎨 updating format of Makefile and CONTRIBUTING.md

* 📝 adding changelog via changie new

*  adding dev_req to Makefile + docs

* 🎨 remove dev_req from docs, dry makefile

* Align names of `.PHONY` targets with their associated rules

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
2023-01-03 11:18:55 -07:00
Doug Beatty
9ecb6e50e4 Treat dense text blobs as binary for git grep (#6462) 2022-12-20 17:10:01 -07:00
Kshitij Aranke
ce9d0afb8a [CT-1591] Don't parse empty Python files (#6433) 2022-12-20 10:56:36 -08:00
Emily Rockman
c39ea807e8 add back validator_error_message (#6465) 2022-12-20 10:56:08 -06:00
Ikko Ashimine
1e35339389 Fix typo in util.py (#6037)
* Fix typo in util.py

identifer -> identifier

* Add change log

* Update .changes/unreleased/Fixes-20221117-220320.yaml

Co-authored-by: Emily Rockman <ebuschang@gmail.com>
2022-12-19 20:01:19 -06:00
Emily Rockman
304797b099 exception cleanup (#6347)
* starting to move jinja exceptions

* convert some exceptions

* add back old functions for backward compatibility

* organize

* more conversions

* more conversions

* add changelog

* split out CacheInconsistency

* more conversions

* convert even more

* convert parsingexceptions

* fix tests

* more conversions

* more conversions

* finish converting exception functions

* convert more tests

* standardize to msg

* remove some TODOs

* fix test param and check the rest

* add comment, move exceptions

* add types

* fix type errors

* fix type for adapter_response

* remove 0.13 version from message
2022-12-19 19:36:24 -06:00
Emily Rockman
b9bdb775ab Fix dependency changelog kind (#6452) 2022-12-19 13:13:38 -06:00
Michelle Ark
b0909b8f5d Functional test framework working with Click, dbtRunner (#6387)
dbtRunner, dbt.cli.requires - preflight, profile, project
2022-12-16 19:54:42 -05:00
Stu Kilgore
5d278dacf1 Make clean task work with click (#6369) 2022-12-15 11:19:16 -06:00
leahwicz
df93858b4b Updating 1.4.0b1 changelog (#6450) 2022-12-15 09:55:23 -05:00
dave-connors-3
e8da84fb9e Feature/support incremental predicates (#5702)
* pass predicated to merge strategy

* postgres delete and insert

* merge with predicates

* update to use arbitrary list of predicates, not dictionaries, merge and delete

* changie

* add functional test to adapter zone

* comma in test config

* add test for incremental predicates delete and insert postgres

* update test structure for inheritance

* handle predicates config for backwards compatibility

* test for predicates keyword

* Add generated CLI API docs

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-12-14 14:07:41 -08:00
Peter Webb
7e90e067af Paw/ct 1652 restore default logging (#6447)
* CT-1652: Restore stdout logging before logger is fully configured

* CT-1652: Add changelog entry

* CT-1652: formatting fix for black
2022-12-14 17:04:50 -05:00
Peter Webb
5e4e917de5 CT-1685: Restore certain aspects of legacy logging behavior important… (#6443)
* CT-1685: Restore certain aspects of legacy logging behavior important to dbt-rpc

* CT-1658: And changelog entry
2022-12-14 11:13:34 -05:00
github-actions[bot]
05dc0212e7 Bumping version to 1.4.0b1 and generate changelog (#6440)
* Bumping version to 1.4.0b1 and generate CHANGELOG

* Updating date

* Updating date

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
2022-12-13 20:18:11 -05:00
Gerda Shank
c00052cbfb Add Optional back on "database" field of HasRelationMetadata (#6439) 2022-12-13 18:15:25 -05:00
Kshitij Aranke
3d54a83822 [CT-1284] Change Python model default materialization to table (#6432) 2022-12-13 15:07:56 -08:00
Gerda Shank
fafd5edbda CT 1644 node cleanup (#6427)
* Remove unneeded SQL compilation attributes from SeedNode

* Fix various places that referenced removed attributes

* Cleanup a few Unions

* More formatting in nodes.py

* Mypy passing. Untested.

* Unit tests working

* use "doc" in documentation unique_ids

* update some doc_ids

* Fix some artifact tests. Still need previous version.

* Update manifest/v8.json

* Move relation_names to parsing

* Fix a couple of tests

* Update some artifacts. snapshot_seed has wrong schema.

* Changie

* Tweak NodeType.Documentation

* Put store_failures property in the right place

* Fix setting relation_name
2022-12-13 12:39:35 -05:00
Josh Devlin
8478262580 Update docker README (#6423) 2022-12-13 11:12:34 -05:00
Kshitij Aranke
83b1fee062 Add aranke to core committers (#6431) 2022-12-12 15:13:03 -08:00
Emily Rockman
0fbbc896b2 Remove PR from most changelog kinds (#6374)
* update changie to require issue or pr, and allow multiple

* remove extraneous data from changelog files.

* allow for multiple PR/issues to be entered

* update contributing guide

* remove issue number from bot changelogs

* update format of PR

* fix dependency changelogs

* remove extra line

* remove extra lines, tweak contributor wording

* Update CONTRIBUTING.md

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2022-12-12 13:18:15 -06:00
Stu Kilgore
ce1aaec31d Adjust tox passenv to be multiline (#6405) (#6430) 2022-12-12 11:13:04 -06:00
Ian Knox
0544b08543 Add support for Python 3.11 (#6326)
* Get running with Python 3.11

* More tests passing, mypy still unhappy

* Upgrade to 3.11, and bump mashumaro

* patch importlib.import_module last

* lambda: Policy() default_factory on include and quote policy

* Add changelog entry

* Put a lambda on it

* Fix text formatting for log file

* Handle variant type return from e.log_level()

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Josh Taylor <joshuataylorx@gmail.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2022-12-08 18:34:03 +01:00
Ian Knox
1809852a0d dbt run works with Click (#6396) 2022-12-08 09:47:08 -06:00
Ian Knox
88d2ee4813 Profile works w Click (#6336) 2022-12-08 09:38:15 -06:00
bruno messias
bef6edb942 Fix dbt.config.get default values (python-model) (#6317)
* feat: add a list of default values to the ctx manager

* tests: dbt.get.config default values

* feat: validate the num of args in config.get

* feat: jinja template for dbt.config.get default values

* docs: changie yaml

* fix:typo on error message

Co-authored-by: Chenyu Li <chenyulee777@gmail.com>

Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
2022-12-07 15:52:38 -08:00
timle
99f27de934 Feature/dbt deps tarball (#4689)
* v0 - new dbt deps type: tarball url

in support of
https://github.com/dbt-labs/dbt-core/issues/4205

* flake8 fixes

* adding max size tarball condition

* clean up imports

* typing

* adding sha1 and subdirectory options; improve logging feedback

sha1: allow user to specify sha1 in packages.yaml, will only install if package matches
subdirectory: allow user to specify subdirectory of package in tarfile, if the package is a non standard structure (like with git subdirectory option)

* simple tests added

* flake fixes

* changes to support tests; adding exceptions; fire_event logging

* new logging events

* tarball exceptions added

* build out tests

* removing in memory tarball test

* update type codes to M - Misc

* adding new events to test_events

* fix spacing for flake

* add retry download code - as used in registry calls

* clean

* remove saving tar in memory inside tarfile object

will hit url multiple times instead

* remove duplicative code after refactor

* black updates

* black formatting

* black formatting

* refactor - no more in-memory tarfile - all as file operations now

- remove tarfile passing, always use tempfile instead
- reorganize system.* functions, removing duplicative code
- more notes on current flow and structure - esp need for pattern of 1) unpack 2) scan for package dir 3) copy to destination.
- cleaning

* cleaning and sync to new tarball code

* cleaning and sync to new tarball code

* requested changes from PR

https://github.com/dbt-labs/dbt-core/pull/4689#discussion_r812970847

* reversions from revision 2

removing sha1 check to simplify/mirror hub install pattern

* simplify/mirror hub install pattern

to simplify/mirror hub install pattern
- removing sha1 check
- supply name/version to act as our 'metadata' source

* simplify/mirror hub install pattern

simplify with goal of mirroring hub install pattern
- supporting subfolders like git packages, and sha1 checks are removed
- existing code from RegistryPinnedPackage (install() and download_and_untar()) performs the operations
- RegistryPinnedPackage install() and download_and_untar() are not currently set up as functions that can be used across classes - this should be moved to dbt.deps.base, or to a dbt.deps.common file - need dbt labs feedback on how to proceed (or leave as is)

* remove revisions, no longer doing package check

* slim down to basic tests

more complex features have been removed (sha1, subfolder) so testing is much simpler!

* fix naming to match hubs behavior

remove version from package folder name

* refactor install and download to upstream PinnedPackage class

i'm on the fence if this is right approach, but seems like most sensible after some thought

* Create Features-20221107-105018.yaml

* fix flake, black, mypy errors

* additional flake/black fixes

* Update .changes/unreleased/Features-20221107-105018.yaml

fix username on changelog

Co-authored-by: Emily Rockman <ebuschang@gmail.com>

* change to fstring

Co-authored-by: Emily Rockman <ebuschang@gmail.com>

* cleaning - remove comment

* remove comment/question for dbt team

* in support of issuecomment 1334055944

https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334055944

* in support of issuecomment 1334118433

https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334118433

* black fixes; remove debug bits

* remove `.format` & add 'tarball' as version

'tarball' as version so that the temp files format nicely:
[tempfile_location]/dbt_utils_2..tar.gz # old
vs
[tempfile_location]/dbt_utils_1.tarball.tar.gz # current

* port os.path refs in `PinnedPackage._install` to pathlib

* lowercase as per PR feedback

* update tests after removing version arg

goes along with 8787ba41af

Co-authored-by: Emily Rockman <ebuschang@gmail.com>
2022-12-07 15:48:17 -06:00
Jeremy Cohen
9c91f3a7bd Adjust tox passenv to be multiline (#6405) 2022-12-07 22:47:51 +01:00
Gerda Shank
1b6fed2ffd CT 1604 remove compiled classes (#6384)
* removed Compiled versions of nodes

* Remove compiled fields from dictionary if not compiled

* check compiled is False instead of attribute existence in env_var
processing

* Update artifacts test (CompiledSnapshotNode did not have SnapshotConfig)

* Changie

* more complicated 'compiling' check in env_var

* Update test_exit_codes.py
2022-12-07 15:21:05 -05:00
dependabot[bot]
0721f2c1b7 Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core (#6375)
* Bump mashumaro[msgpack] from 3.1.1 to 3.2 in /core

Bumps [mashumaro[msgpack]](https://github.com/Fatal1ty/mashumaro) from 3.1.1 to 3.2.
- [Release notes](https://github.com/Fatal1ty/mashumaro/releases)
- [Commits](https://github.com/Fatal1ty/mashumaro/compare/v3.1.1...v3.2)

---
updated-dependencies:
- dependency-name: mashumaro[msgpack]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-12-07 10:16:31 -05:00
Doug Beatty
b9a35da118 Fix intermittent database connection failure in Windows CI test (#6395)
* Fix intermittent database connection failure in Windows CI test

* Changelog entry
2022-12-06 18:34:39 -07:00
Peter Webb
60f80056b1 CT-1405: Refactor event logging code (#6291)
* CT-1405: Refactor event logging code

* CT-1405: Add changelog entry

* CT-1405: Add code to protect against using closed streams from past tests.

* CT-1405: Restore unit test which was only failing locally

* CT-1405: Document a hack with issue # to resolve it in the future

* CT-1405: Make black happy

* CT-1405: Get rid of confusing factory function and duplicated function

* CT-1405: Remove unused event from types.proto and auto-gen'd file
2022-12-06 15:51:52 -05:00
Stu Kilgore
540c3b79aa Prevent docs gen workflow on forks (#6390) 2022-12-06 11:14:20 -06:00
Stu Kilgore
77be2e4fdf Pin ubuntu at 20.04 for some workflows (#6380) 2022-12-05 10:55:36 -06:00
Gerda Shank
16f529e1d4 CT 1477 enrich logging events with data similar to legacy logger (#6325) 2022-12-02 19:29:25 -05:00
Doug Beatty
ebfcf2a9ef Update core/dbt/README.md to match current (#6371)
* Update core/dbt/README.md to match current

Add missing files/folders and alphabetize

* Changelog entry
2022-12-02 15:45:53 -07:00
Alexander Butler
67a8138b65 [fix] Fix the partial parse write path (#6081)
* Fix the partial parse path

Partial parse should use project root or it does not resolve to correct path. 
Eg. `target-path: ../some/dir/target`, if not ran from root, creates an erroneous folder.

* Run pre-commit

* Changie

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2022-12-02 17:23:37 -05:00
Michelle Ark
e91863de59 Set Flags from UserConfig (#6266)
flags with user config, flags.WHICH from invoked_subcommand if available
2022-12-02 15:23:58 -05:00
leahwicz
85d0b5afc7 Reverting back to older ubuntu image (#6363)
* Reverting back to older ubuntu image

* Updating the structured logging workflow as well
2022-12-02 12:09:46 -05:00
Matthew McKnight
1fbcaa4484 reformatting of test after some spike investigation (#6314)
* reformatting of test after some spike investigation

* reformat code to pull tests back into base class definition, move a test to more appropriate spot
2022-12-01 16:54:58 -06:00
justbldwn
481235a943 clarify error log for number of allowed models in a Python file (#6251) 2022-12-01 14:43:36 -05:00
Michelle Ark
2289e45571 Exposures support metrics (#6342)
* exposures support metrics
2022-12-01 11:01:16 -05:00
dependabot[bot]
b5d303f12a Bump mashumaro[msgpack] from 3.0.4 to 3.1 in /core (#6108)
* Bump mashumaro[msgpack] from 3.0.4 to 3.1.1 in /core

Bumps [mashumaro[msgpack]](https://github.com/Fatal1ty/mashumaro) from 3.0.4 to 3.1.
- [Release notes](https://github.com/Fatal1ty/mashumaro/releases)
- [Commits](https://github.com/Fatal1ty/mashumaro/compare/v3.0.4...v3.1)

---
updated-dependencies:
- dependency-name: mashumaro[msgpack]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2022-11-30 17:32:47 -05:00
Mila Page
c3be975783 Ct 288/convert 070 incremental test (#6330)
* Convert incremental schema tests.

* Drop the old test.

* Bad git add. My disappoint is immeasurable and my day has been ruined.

* Adjustments for flake8.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2022-11-29 12:47:20 -08:00
Mila Page
47c2edb42a Ct 1518/convert 063 relation names tests (#6304)
* Convert old test.

Add documentation. Adapt and reenable previously skipped test.

* Convert test and adapt and comment for current standards.

* Remove old versions of tests.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2022-11-29 12:25:36 -08:00
Stu Kilgore
b3440417ad Add GHA workflow to build CLI API docs (#6187) 2022-11-29 13:30:47 -06:00
leahwicz
020f639c7a Update stale.yml (#6258) 2022-11-29 09:40:59 -05:00
Mila Page
55db15aba8 Convert test 067. (#6305)
* Convert test 067. One bug outstanding.

* Test now working! Schema needed renaming to avoid 63 char max problems

* Remove old test.

* Add some docs and rewrite.

* Add exception for when audit tables' schema runs over the db limit.

* Code cleanup.

* Revert exception.

* Round out comments.

* Rename what shouldn't be a base class.

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2022-11-29 00:06:07 -08:00
Itamar Hartstein
bce0e7c096 BaseContext: expose md5 function in context (#6247)
* BaseContext: expose md5 function in context

* BaseContext: add return value type

* Add changie entry

* rename "md5" to "local_md5"

* fix test_context.py
2022-11-28 10:23:40 -05:00
Michelle Ark
44b457c191 DepsTask with Click (#6260)
* deps with click, default --vars param, PartialProject.render_package_metadata
2022-11-22 17:52:59 -05:00
Gerda Shank
7d7066466d CT 1537 fix event test and rename a couple of fields (#6293)
* Rename MacroEvent to JinjaLog

* Rename ConnectionClosed/2

* Fix LogSeedResult

* Rename ConnectionLeftOpen events, fix test_events.py

* Update events README.md, add "category" to EventInfo

* Rename GeneralMacroWarning to JinjaLogWarning
2022-11-22 14:54:20 -05:00
Emily Rockman
517576c088 add back in conditional node length check (#6298) 2022-11-21 21:20:55 -08:00
leahwicz
987764858b Revert "Bump python from 3.10.7-slim-bullseye to 3.11.0-slim-bullseye in /docker (#6180)" (#6281)
This reverts commit 8e28f5906e.
2022-11-17 09:14:22 -05:00
FishtownBuildBot
a235abd176 Add new index.html and changelog yaml files from dbt-docs (#6265) 2022-11-16 17:00:33 +01:00
dependabot[bot]
9297e4d55c Update pathspec requirement from ~=0.9.0 to >=0.9,<0.11 in /core (#5917)
* Update pathspec requirement from ~=0.9.0 to >=0.9,<0.11 in /core

Updates the requirements on [pathspec](https://github.com/cpburnz/python-pathspec) to permit the latest version.
- [Release notes](https://github.com/cpburnz/python-pathspec/releases)
- [Changelog](https://github.com/cpburnz/python-pathspec/blob/master/CHANGES.rst)
- [Commits](https://github.com/cpburnz/python-pathspec/compare/v0.9.0...v0.10.1)

---
updated-dependencies:
- dependency-name: pathspec
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-11-15 22:02:37 -05:00
Chenyu Li
a0ec0b6f9d Project working with Click (#6142)
Co-authored-by: MichelleArk <michelle.ark@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2022-11-15 15:54:05 -08:00
Michelle Ark
eae98677b9 s/gitlab/github for flake8 precommit repo (#6252) 2022-11-15 10:30:00 -05:00
Matthew McKnight
66ac107409 [CT-1262] Convert dbt_debug (#6125)
* init pr for dbt_debug test conversion

* removal of old test

* minor test format change

* add new Base class and Test classes

* reformatting test, new method for capsys and error messgae to check, todo fix badproject

* refomatting tests, ready for review

* checking yaml file, and small reformat

* modifying since update wasn't working in ci/cd
2022-11-14 14:22:48 -06:00
Michelle Ark
39c5c42215 converting 044_test_run_operations (#6122)
* converting 044_test_run_operations
2022-11-14 10:39:57 -05:00
dependabot[bot]
9f280a8469 Update colorama requirement from <0.4.6,>=0.3.9 to >=0.3.9,<0.4.7 in /core (#6144)
* Update colorama requirement in /core

Updates the requirements on [colorama](https://github.com/tartley/colorama) to permit the latest version.
- [Release notes](https://github.com/tartley/colorama/releases)
- [Changelog](https://github.com/tartley/colorama/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/tartley/colorama/compare/0.3.9...0.4.6)

---
updated-dependencies:
- dependency-name: colorama
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2022-11-13 09:57:33 -05:00
Ian Knox
1ec54abdc4 Logging works with Click (#6088) 2022-10-25 14:14:08 -05:00
Ian Knox
5efc4aa066 Fix CI issue with tox (#6137) 2022-10-25 13:32:15 -05:00
Chenyu Li
847c0b9644 Tracking works with Click (#5972)
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
2022-10-25 10:41:57 -07:00
972 changed files with 85282 additions and 43360 deletions

View File

@@ -1,13 +1,19 @@
[bumpversion]
current_version = 1.4.0a1
parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
((?P<prekind>a|b|rc)
(?P<pre>\d+) # pre-release version num
current_version = 1.6.0b5
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
(?P<prerelease> # optional pre-release - ex: a1, b2, rc25
(?P<prekind>a|b|rc) # pre-release type
(?P<num>[\d]+) # pre-release version number
)?
( # optional nightly release indicator
\.(?P<nightly>dev[0-9]+) # ex: .dev02142023
)? # expected matches: `1.15.0`, `1.5.0a11`, `1.5.0a1.dev123`, `1.5.0.dev123457`, expected failures: `1`, `1.5`, `1.5.2-a1`, `text1.5.0`
serialize =
{major}.{minor}.{patch}{prekind}{pre}
{major}.{minor}.{patch}{prekind}{num}.{nightly}
{major}.{minor}.{patch}.{nightly}
{major}.{minor}.{patch}{prekind}{num}
{major}.{minor}.{patch}
commit = False
tag = False
@@ -21,9 +27,11 @@ values =
rc
final
[bumpversion:part:pre]
[bumpversion:part:num]
first_value = 1
[bumpversion:part:nightly]
[bumpversion:file:core/setup.py]
[bumpversion:file:core/dbt/version.py]

View File

@@ -3,6 +3,8 @@
For information on prior major and minor releases, see their changelogs:
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
* [1.3](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
* [1.2](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
* [1.1](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)

1
.changes/1.6.0-a1.md Normal file
View File

@@ -0,0 +1 @@
## dbt-core 1.6.0-a1 - April 17, 2023

50
.changes/1.6.0-b1.md Normal file
View File

@@ -0,0 +1,50 @@
## dbt-core 1.6.0-b1 - May 12, 2023
### Features
- Skip catalog generation ([#6980](https://github.com/dbt-labs/dbt-core/issues/6980))
- Publication artifacts and cross-project ref ([#7227](https://github.com/dbt-labs/dbt-core/issues/7227))
- Add graph structure summaries to target path output ([#7357](https://github.com/dbt-labs/dbt-core/issues/7357))
- Allow duplicate manifest node (models, seeds, analyses, snapshots) names across packages ([#7446](https://github.com/dbt-labs/dbt-core/issues/7446))
- Detect breaking changes to enforced constraints ([#7065](https://github.com/dbt-labs/dbt-core/issues/7065))
- Check for project dependency cycles ([#7468](https://github.com/dbt-labs/dbt-core/issues/7468))
### Fixes
- Persist timing info in run results for failed nodes ([#5476](https://github.com/dbt-labs/dbt-core/issues/5476))
- fix typo in unpacking statically parsed ref ([#7364](https://github.com/dbt-labs/dbt-core/issues/7364))
- safe version attribute access in _check_resource_uniqueness ([#7375](https://github.com/dbt-labs/dbt-core/issues/7375))
- Fix dbt command missing target-path param ([# 7411](https://github.com/dbt-labs/dbt-core/issues/ 7411))
- Fix v0 ref resolution ([#7408](https://github.com/dbt-labs/dbt-core/issues/7408))
- Add --target-path to dbt snapshot command. ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- dbt build selection of tests' descendants ([#7289](https://github.com/dbt-labs/dbt-core/issues/7289))
- fix groupable node partial parsing, raise DbtReferenceError at runtime for safety ([#7437](https://github.com/dbt-labs/dbt-core/issues/7437))
- Fix partial parsing of latest_version changes for downstream references ([#7369](https://github.com/dbt-labs/dbt-core/issues/7369))
- Use "add_node" to update depends_on.nodes ([#7453](https://github.com/dbt-labs/dbt-core/issues/7453))
- Fix var precedence in configs: root vars override package vars ([#6705](https://github.com/dbt-labs/dbt-core/issues/6705))
- Fix inverted `--print/--no-print` flag ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
- Back-compat for previous return type of 'collect_freshness' macro ([#7489](https://github.com/dbt-labs/dbt-core/issues/7489))
- print model version in dbt show if specified ([#7407](https://github.com/dbt-labs/dbt-core/issues/7407))
- enable dbt show for seeds ([#7273](https://github.com/dbt-labs/dbt-core/issues/7273))
- push down limit filtering to adapter ([#7390](https://github.com/dbt-labs/dbt-core/issues/7390))
- Allow missing `profiles.yml` for `dbt deps` and `dbt init` ([#7511](https://github.com/dbt-labs/dbt-core/issues/7511))
- `run_results.json` is now written after every node completes. ([#7302](https://github.com/dbt-labs/dbt-core/issues/7302))
- Do not rewrite manifest.json during 'docs serve' command ([#7553](https://github.com/dbt-labs/dbt-core/issues/7553))
- Pin protobuf to greater than 4.0.0 ([#7565](https://github.com/dbt-labs/dbt-core/issues/7565))
- inject sql header in query for show ([#7413](https://github.com/dbt-labs/dbt-core/issues/7413))
- Pin urllib3 to ~=1.0 ([#7573](https://github.com/dbt-labs/dbt-core/issues/7573))
- Throw error for duplicated versioned and unversioned models ([#7487](https://github.com/dbt-labs/dbt-core/issues/7487))
### Under the Hood
- Update docs link in ContractBreakingChangeError message ([#7366](https://github.com/dbt-labs/dbt-core/issues/7366))
- Reduce memory footprint of cached statement results. ([#7281](https://github.com/dbt-labs/dbt-core/issues/7281))
- Remove noisy parsing events: GenericTestFileParse, MacroFileParse, Note events for static model parsing ([#6671](https://github.com/dbt-labs/dbt-core/issues/6671))
- Update --help text for cache-related parameters ([#7381](https://github.com/dbt-labs/dbt-core/issues/7381))
- Small UX improvements to model versions: Support defining latest_version in unsuffixed file by default. Notify on unpinned ref when a prerelease version is available. ([#7443](https://github.com/dbt-labs/dbt-core/issues/7443))
### Contributors
- [@AndyBys](https://github.com/AndyBys) ([#6980](https://github.com/dbt-labs/dbt-core/issues/6980))
- [@b-luu](https://github.com/b-luu) ([#7289](https://github.com/dbt-labs/dbt-core/issues/7289))
- [@dwreeves](https://github.com/dwreeves) ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- [@thomasgjerdekog](https://github.com/thomasgjerdekog) ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))

38
.changes/1.6.0-b2.md Normal file
View File

@@ -0,0 +1,38 @@
## dbt-core 1.6.0-b2 - May 25, 2023
### Features
- Added warnings for model and ref deprecations ([#7433](https://github.com/dbt-labs/dbt-core/issues/7433))
- Update drop_relation macro to allow for configuration of drop statement separately from object name ([#7625](https://github.com/dbt-labs/dbt-core/issues/7625))
- accept publications in dbt.invoke ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Fixes
- Honor `--skip-profile-setup` parameter when inside an existing project ([#7594](https://github.com/dbt-labs/dbt-core/issues/7594))
- Fix: Relative project paths weren't working with deps ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491))
- Exclude password fields from Jinja rendering. ([#7629](https://github.com/dbt-labs/dbt-core/issues/7629))
- Add --target-path to more CLI subcommands ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- Stringify flag paths for Jinja context ([#7495](https://github.com/dbt-labs/dbt-core/issues/7495))
- write run_results.json for run operation ([#7502](https://github.com/dbt-labs/dbt-core/issues/7502))
### Under the Hood
- Add ability to instantiate Flags class from dict ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607))
- Add other relation to reffable nodes ([#7550](https://github.com/dbt-labs/dbt-core/issues/7550))
- Move node patch method to schema parser patch_node_properties and refactor schema parsing ([#7430](https://github.com/dbt-labs/dbt-core/issues/7430))
- Remove legacy file logger code ([#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
- Break up integration tests as a short term fix for Windows CI runs ([#7668](https://github.com/dbt-labs/dbt-core/issues/7668))
- Include null checks in utils test base ([#7670](https://github.com/dbt-labs/dbt-core/issues/7670))
- Write pub artifact to log ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Dependencies
- Bump ubuntu from 23.04 to 23.10 ([#7675](https://github.com/dbt-labs/dbt-core/pull/7675))
- ([#7681](https://github.com/dbt-labs/dbt-core/pull/7681))
### Contributors
- [@dradnan89@hotmail.com](https://github.com/dradnan89@hotmail.com) ([#7681](https://github.com/dbt-labs/dbt-core/pull/7681))
- [@dwreeves](https://github.com/dwreeves) ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- [@iknox-fa](https://github.com/iknox-fa) ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491), [#NA](https://github.com/dbt-labs/dbt-core/issues/NA))
- [@sdebruyn](https://github.com/sdebruyn) ([#7670](https://github.com/dbt-labs/dbt-core/issues/7670))
- [@stu-k](https://github.com/stu-k) ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607), [#7550](https://github.com/dbt-labs/dbt-core/issues/7550))

46
.changes/1.6.0-b3.md Normal file
View File

@@ -0,0 +1,46 @@
## dbt-core 1.6.0-b3 - June 08, 2023
### Breaking Changes
- Dropped support for Python 3.7 ([#7082](https://github.com/dbt-labs/dbt-core/issues/7082))
- Switch from dbt-metrics to dbt-semantic-interfaces for MetricNode definitions ([#7500](https://github.com/dbt-labs/dbt-core/issues/7500), [#7404](https://github.com/dbt-labs/dbt-core/issues/7404))
### Features
- Add support for materialized views ([#6911](https://github.com/dbt-labs/dbt-core/issues/6911))
- Optimize template rendering for common parse scenarios ([#7449](https://github.com/dbt-labs/dbt-core/issues/7449))
- nodes in packages respect custom generate_alias_name, generate_schema_name, generate_database_name macro overrides defined in packages ([#7444](https://github.com/dbt-labs/dbt-core/issues/7444))
- Enable state for deferral to be separate from state for selectors ([#7300](https://github.com/dbt-labs/dbt-core/issues/7300))
- add access selection syntax ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- add project_name to manifest metadata ([#7752](https://github.com/dbt-labs/dbt-core/issues/7752))
- dbt retry ([#7299](https://github.com/dbt-labs/dbt-core/issues/7299))
- Revamp debug, add --connection flag. Prepare for future refactors/interface changes. ([#7104](https://github.com/dbt-labs/dbt-core/issues/7104))
- Validate public models are not materialized as ephemeral ([#7226](https://github.com/dbt-labs/dbt-core/issues/7226))
- Added support for parsing and serializaing semantic models ([#7499](https://github.com/dbt-labs/dbt-core/issues/7499), [#7503](https://github.com/dbt-labs/dbt-core/issues/7503))
### Fixes
- Constraint rendering fixes: wrap check expression in parentheses, foreign key 'references', support expression in all constraint types ([#7417](https://github.com/dbt-labs/dbt-core/issues/7417), [#7480](https://github.com/dbt-labs/dbt-core/issues/7480), [#7416](https://github.com/dbt-labs/dbt-core/issues/7416))
- Fix warning messages for deprecated dbt_project.yml configs ([#7424](https://github.com/dbt-labs/dbt-core/issues/7424))
- Add `%` to adapter suite test cases for `persist_docs` ([#7698](https://github.com/dbt-labs/dbt-core/issues/7698))
- Improve warnings for constraints and materialization types ([#7335](https://github.com/dbt-labs/dbt-core/issues/7335))
- Incorrect paths used for "target" and "state" directories ([#7465](https://github.com/dbt-labs/dbt-core/issues/7465))
- fix StopIteration error when publication for project not found ([#7711](https://github.com/dbt-labs/dbt-core/issues/7711))
- Using version 0 works when resolving single model ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
- Fix empty --warn-error-options error message ([#7730](https://github.com/dbt-labs/dbt-core/issues/7730))
- send sql header on contract enforcement ([#7714](https://github.com/dbt-labs/dbt-core/issues/7714))
- Fixed doc link in selector.py ([#7533](https://github.com/dbt-labs/dbt-core/issues/7533))
- Fix null-safe equals comparison via `equals` ([#7778](https://github.com/dbt-labs/dbt-core/issues/7778))
- Log PublicationArtifactAvailable even when partially parsing unchanged public models ([#7782](https://github.com/dbt-labs/dbt-core/issues/7782))
- fix RuntimeError when removing project dependency from dependencies.yml ([#7743](https://github.com/dbt-labs/dbt-core/issues/7743))
### Under the Hood
- Fix flaky test for --fail-fast ([#7744](https://github.com/dbt-labs/dbt-core/issues/7744))
- Create `add_from_artifact` to populate `state_relation` field of nodes ([#7551](https://github.com/dbt-labs/dbt-core/issues/7551))
### Contributors
- [@dave-connors-3](https://github.com/dave-connors-3) ([#7738](https://github.com/dbt-labs/dbt-core/issues/7738))
- [@quazi-irfan](https://github.com/quazi-irfan) ([#7533](https://github.com/dbt-labs/dbt-core/issues/7533))
- [@sdebruyn](https://github.com/sdebruyn) ([#7082](https://github.com/dbt-labs/dbt-core/issues/7082))
- [@stu-k](https://github.com/stu-k) ([#7299](https://github.com/dbt-labs/dbt-core/issues/7299), [#7551](https://github.com/dbt-labs/dbt-core/issues/7551))

21
.changes/1.6.0-b4.md Normal file
View File

@@ -0,0 +1,21 @@
## dbt-core 1.6.0-b4 - June 13, 2023
### Fixes
- Respect column 'quote' config in model contracts ([#7370](https://github.com/dbt-labs/dbt-core/issues/7370))
- Fix path selector when using project-dir ([#7819](https://github.com/dbt-labs/dbt-core/issues/7819))
- Allow project dependencies to use miscellaneous keys ([#7497](https://github.com/dbt-labs/dbt-core/issues/7497))
- Updating this error message to point to the correct URL ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))
- Update SemanticModel node to properly impelment the DSI 0.1.0dev3 SemanticModel protocol spec ([#7833](https://github.com/dbt-labs/dbt-core/issues/7833), [#7827](https://github.com/dbt-labs/dbt-core/issues/7827))
### Docs
- Fix for column tests not rendering on quoted columns ([dbt-docs/#201](https://github.com/dbt-labs/dbt-docs/issues/201))
### Under the Hood
- Rm space from NodeType strings ([#7841](https://github.com/dbt-labs/dbt-core/issues/7841))
### Contributors
- [@drewbanin](https://github.com/drewbanin) ([#201](https://github.com/dbt-labs/dbt-core/issues/201))
- [@mirnawong1](https://github.com/mirnawong1) ([#7789](https://github.com/dbt-labs/dbt-core/issues/7789))

28
.changes/1.6.0-b5.md Normal file
View File

@@ -0,0 +1,28 @@
## dbt-core 1.6.0-b5 - June 22, 2023
### Features
- Enable setting packages in dependencies.yml ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372), [#7736](https://github.com/dbt-labs/dbt-core/issues/7736))
- Add AdapterRegistered event log message ([#7038](https://github.com/dbt-labs/dbt-core/issues/7038))
- Further integrate semantic models into the DAG and partial parsing module ([#7800](https://github.com/dbt-labs/dbt-core/issues/7800))
### Fixes
- Fix regression in `run-operation` to not require the name of the package to run ([#7753](https://github.com/dbt-labs/dbt-core/issues/7753))
- Allow dbt show --inline preview of private models ([#7837](https://github.com/dbt-labs/dbt-core/issues/7837))
- Skip jinja parsing of metric filters ([#7864](https://github.com/dbt-labs/dbt-core/issues/7864))
- Fix a bad implicit string conversion regression in debug --config-dir code. ([#7774](https://github.com/dbt-labs/dbt-core/issues/7774))
- Fix UninstalledPackagesFoundError error message to use correct packages specified path ([#7921](https://github.com/dbt-labs/dbt-core/issues/7921))
- Fix: safe remove of external nodes from nodes.depends_on ([#7924](https://github.com/dbt-labs/dbt-core/issues/7924))
### Under the Hood
- Upgrade to dbt-semantic-interfaces v0.1.0dev5 ([#7853](https://github.com/dbt-labs/dbt-core/issues/7853))
- Refactoring: consolidating public_nodes and nodes ([#7890](https://github.com/dbt-labs/dbt-core/issues/7890))
- Resolve SemanticModel ref is the same way as other refs ([#7822](https://github.com/dbt-labs/dbt-core/issues/7822))
- Move from dbt-semantic-intefaces 0.1.0dev5 to 0.1.0dev7 ([#7898](https://github.com/dbt-labs/dbt-core/issues/7898))
- Don't jinja render packages from dependencies.yml ([#7905](https://github.com/dbt-labs/dbt-core/issues/7905))
### Dependencies
- Bump mypy from 0.981 to 1.0.1 ([#7027](https://github.com/dbt-labs/dbt-core/pull/7027))

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Dropped support for Python 3.7
time: 2023-05-15T05:31:48.375649+02:00
custom:
Author: sdebruyn
Issue: "7082"

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Switch from dbt-metrics to dbt-semantic-interfaces for MetricNode definitions
time: 2023-06-07T19:03:09.680189-07:00
custom:
Author: QMalcolm
Issue: 7500 7404

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Updating this error message to point to the correct URL
time: 2023-06-12T16:11:59.181953+01:00
custom:
Author: mirnawong1
Issue: "7789"

View File

@@ -0,0 +1,6 @@
kind: "Dependencies"
body: "Bump mypy from 0.981 to 1.0.1"
time: 2023-02-22T16:28:07.00000Z
custom:
Author: dependabot[bot]
PR: 7027

View File

@@ -0,0 +1,6 @@
kind: "Dependencies"
body: "Bump ubuntu from 23.04 to 23.10"
time: 2023-05-22T00:59:48.00000Z
custom:
Author: dependabot[bot]
PR: 7675

View File

@@ -0,0 +1,5 @@
kind: Dependencies
time: 2023-05-22T21:22:01.336704+02:00
custom:
Author: dradnan89@hotmail.com
PR: "7681"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Fix for column tests not rendering on quoted columns
time: 2023-05-31T11:54:19.687363-04:00
custom:
Author: drewbanin
Issue: "201"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Skip catalog generation
time: 2023-03-21T21:33:38.513443Z
custom:
Author: AndyBys
Issue: "6980"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add support for materialized views
time: 2023-03-29T12:03:13.862041-04:00
custom:
Author: mikealfare McKnight-42
Issue: "6911"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Publication artifacts and cross-project ref
time: 2023-04-20T12:47:56.92683-04:00
custom:
Author: gshank
Issue: "7227"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Optimize template rendering for common parse scenarios
time: 2023-04-24T16:36:11.24088-04:00
custom:
Author: peterallenwebb
Issue: "7449"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add graph structure summaries to target path output
time: 2023-04-25T14:25:22.269051-04:00
custom:
Author: peterallenwebb
Issue: "7357"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Allow duplicate manifest node (models, seeds, analyses, snapshots) names across
packages
time: 2023-04-29T15:50:57.757832-04:00
custom:
Author: MichelleArk
Issue: "7446"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Detect breaking changes to enforced constraints
time: 2023-05-03T10:11:00.617936-05:00
custom:
Author: emmyoop
Issue: "7065"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Check for project dependency cycles
time: 2023-05-09T09:41:47.2-04:00
custom:
Author: gshank
Issue: "7468"

View File

@@ -0,0 +1,7 @@
kind: Features
body: nodes in packages respect custom generate_alias_name, generate_schema_name,
generate_database_name macro overrides defined in packages
time: 2023-05-09T21:29:35.557761-04:00
custom:
Author: michelleark
Issue: "7444"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Added warnings for model and ref deprecations
time: 2023-05-09T23:33:29.679333-04:00
custom:
Author: peterallenwebb
Issue: "7433"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Update drop_relation macro to allow for configuration of drop statement separately
from object name
time: 2023-05-15T12:23:04.177141-04:00
custom:
Author: mikealfare
Issue: "7625"

View File

@@ -0,0 +1,6 @@
kind: Features
body: accept publications in dbt.invoke
time: 2023-05-17T18:56:27.585784-04:00
custom:
Author: MichelleArk
Issue: "7372"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Enable state for deferral to be separate from state for selectors
time: 2023-05-23T22:59:55.920975-07:00
custom:
Author: aranke
Issue: "7300"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add access selection syntax
time: 2023-05-30T16:48:47.740037-05:00
custom:
Author: dave-connors-3
Issue: "7738"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add project_name to manifest metadata
time: 2023-06-01T13:22:23.259448-04:00
custom:
Author: michelleark jtcohen6
Issue: "7752"

View File

@@ -0,0 +1,6 @@
kind: Features
body: dbt retry
time: 2023-06-02T08:33:02.410456-07:00
custom:
Author: stu-k aranke
Issue: "7299"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Revamp debug, add --connection flag. Prepare for future refactors/interface changes.
time: 2023-06-04T02:59:56.28283-07:00
custom:
Author: versusfacit
Issue: 7104

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix null-safe equals comparison via `equals`
time: 2023-06-04T08:00:52.537967-06:00
custom:
Author: dbeatty10
Issue: "7778"

View File

@@ -0,0 +1,6 @@
kind: Features
body: ' Validate public models are not materialized as ephemeral'
time: 2023-06-05T22:20:39.382019-04:00
custom:
Author: michelleark
Issue: "7226"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Added support for parsing and serializaing semantic models
time: 2023-06-06T16:53:51.117429-04:00
custom:
Author: peterallenwebb
Issue: 7499 7503

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add AdapterRegistered event log message
time: 2023-06-13T15:15:07.367371-07:00
custom:
Author: colin-rogers-dbt
Issue: "7038"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Enable setting packages in dependencies.yml
time: 2023-06-13T15:19:17.199297-04:00
custom:
Author: gshank
Issue: 7372 7736

View File

@@ -0,0 +1,6 @@
kind: Features
body: Further integrate semantic models into the DAG and partial parsing module
time: 2023-06-16T16:30:45.513314-04:00
custom:
Author: peterallenwebb
Issue: "7800"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Persist timing info in run results for failed nodes
time: 2023-04-13T13:31:57.938847-05:00
custom:
Author: stu-k
Issue: "5476"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix typo in unpacking statically parsed ref
time: 2023-04-14T16:36:42.279838-04:00
custom:
Author: MichelleArk
Issue: "7364"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: safe version attribute access in _check_resource_uniqueness
time: 2023-04-18T13:52:57.367108-04:00
custom:
Author: MichelleArk
Issue: "7375"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix dbt command missing target-path param
time: 2023-04-19T14:21:50.959786-07:00
custom:
Author: ChenyuLInx
Issue: "\t7411"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix v0 ref resolution
time: 2023-04-19T22:09:10.155137-04:00
custom:
Author: MichelleArk
Issue: "7408"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add --target-path to dbt snapshot command.
time: 2023-04-20T10:42:54.17972-04:00
custom:
Author: dwreeves
Issue: "7418"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: dbt build selection of tests' descendants
time: 2023-04-21T17:24:28.335866975+02:00
custom:
Author: b-luu
Issue: "7289"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix groupable node partial parsing, raise DbtReferenceError at runtime for safety
time: 2023-04-24T16:18:43.130637-04:00
custom:
Author: MichelleArk
Issue: "7437"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix partial parsing of latest_version changes for downstream references
time: 2023-04-24T16:46:49.721231-04:00
custom:
Author: MichelleArk
Issue: "7369"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Use "add_node" to update depends_on.nodes
time: 2023-04-24T17:34:04.37479-04:00
custom:
Author: gshank
Issue: "7453"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Fix var precedence in configs: root vars override package vars'
time: 2023-04-27T23:07:14.992529-04:00
custom:
Author: MichelleArk
Issue: "6705"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: 'Constraint rendering fixes: wrap check expression in parentheses, foreign key
''references'', support expression in all constraint types'
time: 2023-05-04T14:06:42.545193-04:00
custom:
Author: MichelleArk
Issue: 7417 7480 7416

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix inverted `--print/--no-print` flag
time: 2023-05-05T13:25:45.949997-06:00
custom:
Author: dbeatty10 thomasgjerdekog
Issue: "7517"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Back-compat for previous return type of 'collect_freshness' macro
time: 2023-05-06T17:33:15.104953+02:00
custom:
Author: jtcohen6
Issue: "7489"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix warning messages for deprecated dbt_project.yml configs
time: 2023-05-06T18:09:00.361961+02:00
custom:
Author: jtcohen6
Issue: "7424"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Respect column 'quote' config in model contracts
time: 2023-05-06T19:18:13.351819+02:00
custom:
Author: jtcohen6
Issue: "7370"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: print model version in dbt show if specified
time: 2023-05-08T04:27:07.9965-07:00
custom:
Author: aranke
Issue: "7407"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: enable dbt show for seeds
time: 2023-05-08T04:49:22.82093-07:00
custom:
Author: aranke
Issue: "7273"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: push down limit filtering to adapter
time: 2023-05-08T06:09:26.455524-07:00
custom:
Author: aranke
Issue: "7390"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Allow missing `profiles.yml` for `dbt deps` and `dbt init`
time: 2023-05-08T07:29:29.873793-06:00
custom:
Author: dbeatty10
Issue: "7511"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: '`run_results.json` is now written after every node completes.'
time: 2023-05-08T09:37:32.809356-05:00
custom:
Author: iknox-fa
Issue: "7302"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Do not rewrite manifest.json during 'docs serve' command
time: 2023-05-08T14:25:18.376379-04:00
custom:
Author: jtcohen6
Issue: "7553"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: inject sql header in query for show
time: 2023-05-09T10:29:32.12947-07:00
custom:
Author: aranke
Issue: "7413"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Pin protobuf to greater than 4.0.0
time: 2023-05-09T12:31:59.019718-04:00
custom:
Author: gshank
Issue: "7565"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Pin urllib3 to ~=1.0
time: 2023-05-09T15:35:30.504674-04:00
custom:
Author: mikealfare
Issue: "7573"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Throw error for duplicated versioned and unversioned models
time: 2023-05-09T16:50:07.530882-04:00
custom:
Author: gshank
Issue: "7487"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Honor `--skip-profile-setup` parameter when inside an existing project
time: 2023-05-11T14:04:41.382181-06:00
custom:
Author: dbeatty10
Issue: "7594"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Fix: Relative project paths weren''t working with deps'
time: 2023-05-15T12:36:54.807413-05:00
custom:
Author: iknox-fa
Issue: "7491"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Exclude password fields from Jinja rendering.
time: 2023-05-15T14:28:51.400321-04:00
custom:
Author: peterallenwebb
Issue: "7629"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add --target-path to more CLI subcommands
time: 2023-05-16T15:26:44.557072-04:00
custom:
Author: dwreeves
Issue: "7646"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: write run_results.json for run operation
time: 2023-05-22T13:29:24.182612-07:00
custom:
Author: aranke
Issue: "7502"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Stringify flag paths for Jinja context
time: 2023-05-22T13:50:07.897354-04:00
custom:
Author: gshank
Issue: "7495"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add `%` to adapter suite test cases for `persist_docs`
time: 2023-05-24T16:06:48.477708-06:00
custom:
Author: dbeatty10
Issue: "7698"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Improve warnings for constraints and materialization types
time: 2023-05-25T07:36:51.855641-05:00
custom:
Author: emmyoop
Issue: "7335"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Incorrect paths used for "target" and "state" directories
time: 2023-05-25T16:50:53.718564-04:00
custom:
Author: gshank
Issue: "7465"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix StopIteration error when publication for project not found
time: 2023-05-26T15:37:38.952939-04:00
custom:
Author: michelleark
Issue: "7711"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Using version 0 works when resolving single model
time: 2023-05-26T16:47:27.6065-04:00
custom:
Author: gshank
Issue: "7372"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix empty --warn-error-options error message
time: 2023-05-30T10:42:28.382804-04:00
custom:
Author: michelleark
Issue: "7730"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: send sql header on contract enforcement
time: 2023-05-31T13:19:19.801391-04:00
custom:
Author: michelleark
Issue: "7714"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fixed doc link in selector.py
time: 2023-06-01T13:05:49.952-05:00
custom:
Author: quazi-irfan
Issue: "7533"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Log PublicationArtifactAvailable even when partially parsing unchanged public
models
time: 2023-06-05T12:11:27.739183-04:00
custom:
Author: michelleark
Issue: "7782"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix RuntimeError when removing project dependency from dependencies.yml
time: 2023-06-05T12:44:25.978022-04:00
custom:
Author: michelleark
Issue: "7743"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Fix regression in `run-operation` to not require the name of the package to
run
time: 2023-06-06T14:52:17.38538-07:00
custom:
Author: aranke
Issue: "7753"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix path selector when using project-dir
time: 2023-06-08T13:59:52.95775-04:00
custom:
Author: gshank
Issue: "7819"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Allow project dependencies to use miscellaneous keys
time: 2023-06-09T12:19:30.469487-05:00
custom:
Author: emmyoop
Issue: "7497"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Allow dbt show --inline preview of private models
time: 2023-06-09T19:15:46.716379-04:00
custom:
Author: jtcohen6
Issue: "7837"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Update SemanticModel node to properly impelment the DSI 0.1.0dev3 SemanticModel
protocol spec
time: 2023-06-12T17:58:54.289704-07:00
custom:
Author: QMalcolm
Issue: 7833 7827

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Skip jinja parsing of metric filters
time: 2023-06-15T14:29:49.900201-07:00
custom:
Author: QMalcolm
Issue: "7864"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix a bad implicit string conversion regression in debug --config-dir code.
time: 2023-06-21T03:07:33.815966-07:00
custom:
Author: versusfacit
Issue: "7774"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Fix UninstalledPackagesFoundError error message to use correct packages specified
path
time: 2023-06-22T10:58:29.576179-04:00
custom:
Author: michelleark
Issue: "7921"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Fix: safe remove of external nodes from nodes.depends_on'
time: 2023-06-22T12:19:07.657855-04:00
custom:
Author: michelleark
Issue: "7924"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update docs link in ContractBreakingChangeError message
time: 2023-04-17T11:45:01.005104+02:00
custom:
Author: jtcohen6
Issue: "7366"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Reduce memory footprint of cached statement results.
time: 2023-04-17T12:27:21.972268-05:00
custom:
Author: iknox-fa
Issue: "7281"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: 'Remove noisy parsing events: GenericTestFileParse, MacroFileParse, Note events
for static model parsing'
time: 2023-04-18T12:12:36.928665+02:00
custom:
Author: jtcohen6
Issue: "6671"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update --help text for cache-related parameters
time: 2023-04-18T12:23:23.276693+02:00
custom:
Author: jtcohen6
Issue: "7381"

View File

@@ -0,0 +1,8 @@
kind: Under the Hood
body: 'Small UX improvements to model versions: Support defining latest_version in
unsuffixed file by default. Notify on unpinned ref when a prerelease version is
available. '
time: 2023-04-24T13:53:00.484916+02:00
custom:
Author: jtcohen6
Issue: "7443"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add ability to instantiate Flags class from dict
time: 2023-05-15T09:51:16.291352-05:00
custom:
Author: stu-k
Issue: "7607"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add other relation to reffable nodes
time: 2023-05-15T15:21:07.808892-05:00
custom:
Author: stu-k
Issue: "7550"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move node patch method to schema parser patch_node_properties and refactor schema
parsing
time: 2023-05-16T09:42:41.793503-04:00
custom:
Author: gshank
Issue: "7430"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove legacy file logger code
time: 2023-05-18T11:42:51.825799-05:00
custom:
Author: iknox-fa
Issue: NA

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Break up integration tests as a short term fix for Windows CI runs
time: 2023-05-19T15:30:59.138043-04:00
custom:
Author: mikealfare
Issue: "7668"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Include null checks in utils test base
time: 2023-05-21T12:57:20.659726+02:00
custom:
Author: sdebruyn
Issue: "7670"

Some files were not shown because too many files have changed in this diff Show More