Compare commits

...

181 Commits

Author SHA1 Message Date
Gerda Shank
893a79508a Remove ref to python 3.8 2024-10-16 15:22:23 -04:00
Gerda Shank
78c05718c5 Remove Python 3.8 from various places (#10861)
* Remove Python 3.8 from various places

* Add changelog entry.

---------

Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
2024-10-16 15:15:27 -04:00
Quigley Malcolm
d18f50bbb8 Ensure consistent current_time across microbatch models in an invocation (#10830)
* Add test that checks microbatch models are all operating with the same `current_time`

* Set an `invocated_at` on the `RuntimeConfig` and plumb to `MicrobatchBuilder`

* Add changie doc

* Rename `invocated_at` to `invoked_at`

* Simply conditional logic for setting MicrobatchBuilder.batch_current_time

* Rename `batch_current_time` to `default_end_time` for MicrobatchBuilder
2024-10-15 16:55:19 -05:00
Gerda Shank
ffa75ca9ff Refactor code to properly handle reference deprecations (#10852) 2024-10-15 16:44:27 -04:00
Gerda Shank
8f847167fa Remove dbt_valid_to_current test (will go in adapter zone) (#10854) 2024-10-15 15:39:01 -04:00
Kshitij Aranke
cd6bb9e782 Fix #2578: Allow instances of generic data tests to be documented (#10850) 2024-10-15 18:53:51 +01:00
Kshitij Aranke
ef9abe6c06 [Tidy-First] Fix node_status for hooks (#10845) 2024-10-14 21:27:34 +01:00
Peter Webb
40c350ff21 Add better typing in jinja_static.py (#10835)
* Add better typing in jinja_static.py, remove commented code, clarify names

* Avoid circular dependency.

* Actually work around the circular dependency.
2024-10-11 12:19:37 -04:00
Gerda Shank
c7d8693f70 Enable setting datetime value for dbt_valid_to when the record is current (#10780) 2024-10-10 18:41:03 -04:00
Colin Rogers
6743e32574 add builder config to test node config (#10767)
* add builder config to node config

* add changie

* raise expected exceptions

* add code comment and additional tests

* update tests

* update tests
2024-10-10 14:32:19 -07:00
Quigley Malcolm
f6cdacc61e Stop making microbatch batches with filters that will never have any rows (#10826) 2024-10-08 18:56:10 -05:00
Quigley Malcolm
5db0b81da1 Track batch execution time for microbatch models (#10828)
* Begin testing that microbatch execution times are being tracked and set

* Begin tracking the execution time of batches for microbatch models

* Add changie doc

* Additional assertions in microbatch testing
2024-10-08 14:32:58 -05:00
Quigley Malcolm
fc8eb820aa Validate --event-time-start is before --event-time-end (#10820)
* Validate that `event_time_start` is before `event_time_end` when passed from CLI

Sometimes CLI options have restrictions based on other CLI options. This is the case
for `--event-time-start` and `--event-time-end`. Unfortunately, click doesn't provide
a good way for validating this, at least not that I found. Additionaly I'm not sure
if we have had anything like this previously. In any case, I couldn't find a
centralized validation area for such occurances. Thus I've gone and added one,
`validate_option_interactions`. Long term if more validations are added, we should
add this wrapper to each CLI command. For now I've only added it to the commands that
support `event_time_start` and `event_time_end`, specifically `build` and `run`.

* Add changie doc

* If `--event-time-end` is not specififed, ensure `--event-time-start` is less than the current time

* Fixup error message about event_time_start and event_time_end

* Move logic to validate `event_time` cli flags to `flags.py`

* Update validation of `--event-time-start` against `datetime.now` to use UTC
2024-10-07 14:34:42 -05:00
FishtownBuildBot
fc83f5edfa [Automated] Merged prep-release/1.9.0b2_11213923466 into target main during release process 2024-10-07 07:17:39 -04:00
Github Build Bot
8248d1eb53 Bumping version to 1.9.0b2 and generate changelog 2024-10-07 10:50:33 +00:00
Kshitij Aranke
6b9c1da1ae Revert "state:modified vars, behind "state_modified_compare_vars" behaviour flag" (#10793) (#10813) 2024-10-02 21:00:48 +01:00
Courtney Holcomb
7940ad5c78 Fix case-sensitivity in validation warning (#10807) 2024-10-01 15:40:51 -05:00
Doug Beatty
3ec8fa79bd --inline-direct is an internal CLI flag (#10806) 2024-10-01 14:28:41 -06:00
FishtownBuildBot
396cf2d683 [Automated] Merged prep-release/1.9.0b1_11131260909 into target main during release process 2024-10-01 14:57:06 -04:00
Github Build Bot
87b1143a62 Bumping version to 1.9.0b1 and generate changelog 2024-10-01 18:30:08 +00:00
Kshitij Aranke
75a09621cd [tidy_first] Set default for STATE_MODIFIED_COMPARE_VARS flag, mark TestProjInfo as not a test class (#10805) 2024-10-01 17:27:44 +01:00
Kshitij Aranke
5e9f1b515f [Round 2] Fix #9005: Allow singular tests to be documented in properties.yml (#10792) 2024-10-01 08:05:36 +01:00
Quigley Malcolm
25a68a990c When retrying microbatch models, propagate prior successful state (#10802)
* When retrying microbatch models, propagate prior successful state

* Changie doc for microbatch dbt retry fixes

* Fix test_manifest unit tests for batch_info key

* Add functional test for when a microbatch model has multiple retries

* Add comment about when batch_info will be something other than None
2024-10-01 00:16:05 -05:00
Michelle Ark
a86e2b4ffc [state:modified] store unrendered_database and unrendered_schema on source definition for state:modified comparisons (#10675) 2024-09-30 17:50:33 +02:00
Michelle Ark
94917432f9 add model_incremental_strategy to track_model_run (#10758) 2024-09-30 17:35:33 +02:00
Michelle Ark
d1857b39ca state:modified vars, behind "state_modified_compare_vars" behaviour flag (#10793) 2024-09-30 16:32:37 +02:00
Kshitij Aranke
2ff3f20863 Create skip_nodes_if_on_run_start_fails behavior change flag (#10699) 2024-09-30 13:53:08 +01:00
Courtney Holcomb
5e3d418264 Add new validations for custom granularities (#10789)
* Bump DSI to latest version to ensure mantle users have new validations for custom granularities

* Changelog
2024-09-27 08:37:04 -05:00
Chenyu Li
5d32aa8b62 Revert "Fix #9005: Allow singular tests to be documented in properties.yml" (#10790)
This reverts commit 3ac20ce7a8.
2024-09-26 17:03:10 -07:00
Quigley Malcolm
d8b1bf53f7 [CT-10785] Microbatch models should respect full_refresh model config (#10788)
* Add tests to check how microbatch models respect `full_refresh` model configs

* Fix `_is_incremental` to properly respect `full_refresh` model config

In dbt-core, it is generally expected that values passed via CLI flags take
precedence over model level configs. However, `full_refresh` on a model is an
exception to this rule, where in the model config takes precedence. This
config exists specifically to _prevent_ accidental full refreshes of large
incremental models, as doing so can be costly. **_It is actually best
practice_** to set `full_refresh=False` on incremental models.

Prior to this commit, for microbatch models, the above was not happening. The
CLI flag `--full-refresh` was taking precedence over the model config
`full_refresh`. That meant that if `--full-refresh` was supplied, then the
microbatch model **_would full refresh_** even if `full_refresh=False` was
set on the model. This commit solves that problem.

* Add changie doc for microbatch `full_refresh` config handling
2024-09-26 16:43:14 -05:00
Kshitij Aranke
1076352293 [CORE-388] Add group metadata info to LogModelResult and LogTestResult (#10775) 2024-09-26 20:57:06 +01:00
Gerda Shank
1fe9c1bbfe Attempt to skip saved query processing when no semantic manifest changes (#10784) 2024-09-26 13:04:19 -04:00
Quigley Malcolm
41e4836c0f Fix changie doc for microbatch retry functionality (#10787)
The changie log was referencing the microbatch epic instead of the specific
issues it resolved
2024-09-26 11:34:41 -05:00
Michelle Ark
b590045b9f [state:modified] persist unrendered_config from schema.yml, and more reliably compute unrendered_config from .sql files (#10487) 2024-09-26 16:03:40 +01:00
Quigley Malcolm
1fd4d2eae6 Enable retry support for Microbatch models (#10751)
* Add `PartialSuccess` status type and use it for microbatch models with mixed results

* Handle `PartialSuccess` in `interpret_run_result`

* Add `BatchResults` object to `BaseResult` and begin tracking during microbatch runs

* Ensure batch_results being propagated to `run_results` artifact

* Move `batch_results` from `BaseResult` class to `RunResult` class

* Move `BatchResults` and `BatchType` to separate arifacts file to avoid circular imports

In our next commit we're gonna modify `dbt/contracts/graph/nodes.py` to import the
`BatchType` as part of our work to implement dbt retry for microbatch model nodes.
Unfortunately, the import in `nodes.py` creates a circular dependency because
`dbt/artifacts/schemas/results.py` imports from `nodes.py` and `dbt/artifacts/schemas/run/v5/run.py`
imports from that `results.py`. Thus the new import creates a circular import. Now this
_shouldn't_ be necessary as nothing in artifacts should import from the rest of dbt-core.
However, we do. We should fix this, but this is also out of scope for this segement of work.

* Add `PartialSuccess` as a retry-able status, and use batches to retry microbatch models

* Fix BatchType type so that the first datetime is no longer Optional

* Ensure `PartialSuccess` causes skipping of downstream nodes

* Alter `PartialSuccess` status to be considered an error in `interpret_run_result`

* Update schemas and test artifacts to include new batch_results run results key

* Add functional test to check that 'dbt retry' retries 'PartialSuccess' models

* Update partition failure test to assert downstream models are skipped

* Improve `success`/`error`/`partial success` messaging for microbatch models

* Include `PartialSuccess` in status that `--fail-fast` counts as a failure

* Update `LogModelResult` to handle partial successes

* Update `EndOfRunSummary` to handle partial successes

* Cleanup TODO comment

* Raise a DbtInternalError if we get a batch run result without `batch_results`

* When running a microbatch model with supplied batches, force non full-refresh behavior

This is necessary because of retry. Say on the initial run the microbatch model
succeeds on 97% of it's batches. Then on retry it does the last 3%. If the retry
of the microbatch model executes in full refresh mode it _might_ blow away the
97% of work that has been done. This edge case seems to be adapter specific.

* Only pass batches to retry for microbatch model when there was a PartialSuccess

In the previous commit we made it so that retries of microbatch models wouldn't
run in full refresh mode when the microbatch model to retry has batches already
specified from the prior run. This is only problematic when the run being retried
was a full refresh AND all the batches for a given microbatch model failed. In
that case WE DO want to do a full refresh for the given microbatch model. To better
outline the problem, consider the following:

* a microbatch model had a begin of `2020-01-01` and has been running this way for awhile
* the begin config has changed to `2024-01-01` and  dbt run --full-refresh gets run
* every batch for an microbatch model fails
* on dbt retry the the relation is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purged

To avoid this, all we have to do is ONLY pass the batch information for partially successful microbatch
models. Note: microbatch models only have a partially successful status IFF they have both
successful and failed batches.

* Fix test_manifest unit tests to know about model 'batches' key

* Add some console output assertions to microbatch functional tests

* add batch_results: None to expected_run_results

* Add changie doc for microbatch retry functionality

* maintain protoc version 5.26.1

* Cleanup extraneous comment in LogModelResult

---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2024-09-26 08:45:47 -05:00
Gerda Shank
ac66f91351 Improve performance of infer primary key (#10782) 2024-09-25 18:16:33 -04:00
Peter Webb
359a2c0cc5 Add '--inline-direct' parameter to 'dbt show'. (#10770)
* Add '--inline-direct' parameter to 'dbt show'.

* Add changelog entry.

* Update core/dbt/cli/main.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Add test of failure for --inline-direct

---------

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2024-09-25 12:31:05 -04:00
Quigley Malcolm
bbdb98fa5d Microbatch Config Validation (#10752) 2024-09-24 19:56:34 +01:00
Katsuya Shimabukuro
a8d4ba2b4a Fix unit tests for incremental models with alias (#10755) 2024-09-24 17:57:37 +01:00
Doug Beatty
09e973d24a Test case for merge_exclude_columns (#8268)
* Test case for `merge_exclude_columns`

* Update expected output for `merge_exclude_columns`

* Skip TestMergeExcludeColumns test

* Enable this test since PostgreSQL 15+ is available in CI now

* Undo modification to expected output
2024-09-24 08:10:44 -06:00
Michelle Ark
730e40a867 Add required 'begin' config support for microbatch models (#10756) 2024-09-24 14:13:02 +01:00
Michelle Ark
a1e4753020 Write microbatch compiled + run code to separate target files (#10743) 2024-09-24 14:12:44 +01:00
Kshitij Aranke
3ac20ce7a8 Fix #9005: Allow singular tests to be documented in properties.yml (#10744) 2024-09-24 13:15:03 +01:00
Michelle Ark
aa23af98e5 ignore --empty in unit test ref/source calls (#10764) 2024-09-23 23:36:28 +01:00
Peter Webb
46da967115 Allow snapshots to be defined with YAML only. (#10762) 2024-09-23 16:33:30 -04:00
Gerda Shank
db694731c9 Allow configuration of snapshot column names (#10608) 2024-09-20 19:31:05 -04:00
Doug Beatty
7016cd3085 Standardize returning ResourceTypeSelector instances in dbt list and dbt build (#10739)
* Remove duplicated constructor for `ResourceTypeSelector`

* Add type annotation for `ResourceTypeSelector`

* Standardize on constructor for `ResourceTypeSelector` where `include_empty_nodes=True`

* Changelog entry
2024-09-19 16:53:16 -06:00
Doug Beatty
9ca10fbfd9 Enable --resource-type and --exclude-resource-type CLI flags and environment variables for dbt test (#10706)
* Adding logic to TestSelector to remove unit tests if they are in excluded_resource_types

* Adding change log

* Respect `--resource-type` and `--exclude-resource-type` CLI flags and associated environment variables

* Test CLI flag for excluding unit tests for the `dbt test` command

* Satisy isort pre-commit hook

* Fix mypy for positional argument "resource_types" in call to "TestSelector"

* Replace `TestSelector` with `ResourceTypeSelector`

* Add co-author

* Update changelog description

* Add functional tests for new feature

* Compare the actual results, not just the count

* Remove test case covered elsewhere

* Test for `DBT_EXCLUDE_RESOURCE_TYPES` environment variable for `dbt test`

* Update per pre-commit hook

* Restore to original form (until we refactor extraneous `ResourceTypeSelector` references later)

---------

Co-authored-by: Matthew Cooper <asimov.1st@gmail.com>
2024-09-19 16:52:59 -06:00
Quigley Malcolm
3308a4365e Split out model vs microbatch execution (#10737) 2024-09-18 22:13:50 -05:00
Mike Alfare
f8bfd32ed6 fix mock in unit test after removing deprecated snowplow method (#10738) 2024-09-18 15:45:11 -07:00
Mike Alfare
3e437a6734 snowplow is deprecating track_struct_event; use StructuredEvent instead (#10736) 2024-09-18 16:29:18 -04:00
Doug Beatty
9e633f6178 Replace TestSelector with ResourceTypeSelector (#10718)
* [tidy first] Replace `TestSelector` with `ResourceTypeSelector`

* Changelog entry

* Fully preserve current behavior

* Revert "Fully preserve current behavior"

This reverts commit ceecfec96d.
2024-09-18 14:13:50 -06:00
Mike Alfare
d182d06644 Add Snowplow tracking for behavior flags (#10721)
* add behavior deprecation snowplow callback
* update tests for new callback
* update test input with the new required field
2024-09-18 15:09:23 -04:00
Peter Webb
054c6fde37 Fix calls to pre_model_hook which were apparently passing an object of the wrong type. (#10735) 2024-09-18 14:09:02 -04:00
Ly Nguyen
4c326e40b5 Update template for docs issues (#10731)
* Update template for docs issues

* Update .github/workflows/docs-issue.yml

* Update .github/workflows/docs-issue.yml

* Update .github/workflows/docs-issue.yml
2024-09-18 13:02:26 -05:00
Michelle Ark
8fe5ea1ee7 Microbatch: batched execution (#10677)
* initial rough-in with CLI flags

* dbt-adapters testing against event-time-ref-filtering

* fix TestList

* Checkpoint

* fix tests

* add event_time_start params to build

* rename configs

* Gate resolve_event_time_filter via micro batch strategy and fix strptime usage

* Add unit test for resolve_event_time_filter

* Additional unit tests for `resolve_event_time_filter` to ensure lookback + batch_size work

* Remove extraneous comments and print statements from resolve_event_time_filter

* Fixup microbatch functional tests to use microbatch strategy

* Gate microbatch functionality behind env_var while in beta

* Add comment about how _is_incremental should be removed

* Improve `event_time_start/end` cli parameters to auto convert to datetime objects

* for testing: dbt-postgres 'microbatch' strategy

* rough in: chunked backfills

* partial failure of microbatch runs

* decouple run result methods

* initial refactor

* rename configs to __dbt_internal

* update compiled_code in context after re-compilation

* finish rename of context vars

* changelog entry

* fix patch_microbatch_end_time

* refactor into MicrobatchBuilder

* fix provider unit tests + add unit tests for MicrobatchBuilder

* add TestMicrobatchJinjaContextVarsAvailable

* unit test offset + truncate timestamp methods

* Remove pairing.md file

* Add tying to microbatch specific functions added in `task/run.py`

* Add doc strings to microbatch.py functions and classes

* Set microbatch node status to `ERROR` if all batches for node failed

* Fire an event for batch exceptions instead of directly printing

* Fix firing of failed microbatch log event

---------

Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
2024-09-18 11:46:06 -05:00
Doug Beatty
16f5023f4d Fix --resource-type test for dbt list and dbt build (#10730)
* Update functional tests to cover this case

* Revert "Update functional tests to cover this case"

This reverts commit 4c78e816f6.

* New functional tests to cover the resource_type config

* Separate data tests from unit tests for `resource_types` config of `dbt list` and `dbt build`

* Changelog entry
2024-09-18 10:21:14 -06:00
Courtney Holcomb
c6b8f7e595 Add custom granularities to YAML spec (#10664)
* Add custom granularities to YAML spec

* Changelog

* Add tests

* Remove unneeded duplicate classes
2024-09-17 13:02:56 -05:00
Michelle Ark
77aeb3ea68 Add functional tests for custom incremental strategies names 'microbatch' (#10716)
* Add functional tests for custom incremental strategies names 'microbatch'

* Point dev-requirement of `dbt-adapters` back to the main branch

The associated branch/PR in `dbt-adapters` that we were previously
pointing to has been merged. Thus we can point back to `main` again.

---------

Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
2024-09-16 14:38:14 -05:00
Emily Rockman
1e20772d33 Update docs-issue.yml (#10720) 2024-09-16 11:59:15 -05:00
Doug Beatty
8ce2c46a2f [tidy first] Fix a couple misspellings (#10717) 2024-09-16 10:02:30 -06:00
Gerda Shank
aeaaedcaa1 Update docs and title for schemas.getdbt.com changes (#10719) 2024-09-16 11:11:08 -04:00
Michelle Ark
6c111f2e31 add patch_microbatch_event_time (#10712) 2024-09-13 22:01:31 -04:00
Michelle Ark
139b9ac54f delete extraneous EventCatcher (#10710) 2024-09-13 18:19:33 -04:00
Michelle Ark
cc8541c05f Microbatch: event_time ref + source filtering (#10594) 2024-09-12 18:16:04 -04:00
Michelle Ark
ab500a9709 install postgres-16 in CI (#10687) 2024-09-11 12:43:33 -04:00
Mike Alfare
1d3d315249 Add flags from dbt_project.yml to the Project and RuntimeConfig objects (#10644)
* add flags from dbt_project.yml to the Project and RuntimeConfig objects
2024-09-06 15:42:29 -04:00
Gerda Shank
b35ad46e3f Remove deprecation warning to change "tests:" config to "data_tests:" (#10670) 2024-09-05 20:35:28 -04:00
Gerda Shank
c28cb92af5 Warn if timestamp updated_at field uses incompatible timestamp (#10352)
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2024-09-04 14:42:14 -04:00
RyoAriyama
b56d96df5e Fix/changes current working dir when using a dbt project dir (#9596)
* made class changing directory a context manager.

* add change log

* fix conflict

* made base as a context manager

* add assertion

* Remove index.html

* add it test to testDbtRunner

* fix deps args order

* fix test

---------

Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2024-09-03 13:41:53 -07:00
Jeremy Cohen
37d382c8e7 Filter out empty nodes after graph selection (#10580)
* Add unit test

* Filter out empty nodes after graph selection

* Add changie

* Add --indirect-selection empty check to unit test
2024-09-03 18:48:03 +02:00
Gerda Shank
9b7f4ff842 use full manifest in adapter instead of macro_manifest (#10609)
* use full manifest in adapter instead of macro_manifest

* Add test case

* Add changelog entry

* Remove commented code.

---------

Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
2024-08-29 11:32:30 -04:00
Emily Rockman
555ff8091f update dep for psycopg (#10633) 2024-08-29 09:44:49 -05:00
Emily Rockman
98fddcf54f rework test to ignore utils version (#10625) 2024-08-28 15:25:09 -05:00
Emily Rockman
d652359c61 add typing (#10619) 2024-08-28 13:26:18 -05:00
Peter Webb
f7d21e012e Add More Typing to the dbt.task Module (#10622)
* Add typing to task module.

* More typing in the task module

* Still more types for task module
2024-08-28 11:18:01 -04:00
Quigley Malcolm
e1fa461186 [TIDY FIRST] Fix typing issues in dbt/core/tasks/clean.py (#10617) 2024-08-27 17:37:16 -05:00
Quigley Malcolm
1153597970 Fix typing errors in core/dbt/contracts/sql.py (#10615) 2024-08-27 17:37:00 -05:00
Quigley Malcolm
09f9febc25 [TIDY FIRST] Fix core/dbt/version.py type hinting (#10613) 2024-08-27 17:36:31 -05:00
Doug Beatty
22181409f6 Enable calling a macro in a pre- or post-hook config in properties.yml (#10603)
* Tests for calling a macro in a pre- or post-hook config in properties.yml

* Late render pre- and post-hooks configs in properties / schema YAML files

* Changelog entry
2024-08-27 11:08:56 -06:00
William Deng
f25a474f75 updated saved query tests and fixtures (#10610) 2024-08-26 17:39:35 -04:00
aliceliu
3c55806203 Fix state:modified check for exports (#10565) 2024-08-23 15:22:38 -04:00
Gerda Shank
bba020fcc0 Add test for source names with quotes (#10588) 2024-08-21 11:57:34 -04:00
Courtney Holcomb
84eb0ff672 Bump DSI version (#10585)
* Bump DSI version

* Changelog
2024-08-20 16:37:52 -04:00
Kshitij Aranke
3695698e22 [CORE-364] Add group info to RunResultError, RunResultFailure, RunResultWarning log lines (#10535) 2024-08-19 11:26:00 -07:00
Courtney Holcomb
9ca1bc5b4c Remove unneeded TODO (#10568) 2024-08-14 14:49:47 -07:00
Gerda Shank
5f66678f6d Incremental models with a contract don't need their columns modified (#10371) 2024-08-14 08:15:25 -07:00
Jean Cochrane
63262e93cb Use model alias for the CTE identifier generated during ephemeral materialization (#10290)
* Use alias instead of name when adding ephemeral model prefixes

* Adjust TestCustomSchemaWithCustomMacroFromModelName to test ephemeral models

* Add changelog entry for ephemeral model CTE identifier fix

* Reference model.identifier and model.name where appropriate to resolve typing errors

* Move test for ephemeral model with alias to dedicated test in test_compile.py
2024-08-09 15:00:55 -07:00
Tobie Tusing
374412af53 Improve tree traversal of select_children (#10526)
* update children search

* update search to include children in original selector

* add changie

* remove unused function

* fix wrong function call

* fix depth
2024-08-09 17:38:15 -04:00
Kshitij Aranke
47848b8ea8 Fix add_ephemeral_prefix to identifier instead of name (#10550) 2024-08-09 13:58:37 -05:00
Michelle Ark
3d09872a56 reset deprecations prior to usage in unit tests (#10545) 2024-08-08 12:25:22 -04:00
Colin Rogers
dfa7d06526 Revert "Remove Undocumented Property" (#10544)
* Revert "Remove undocumented property which does not pass mypy checks after an…"

This reverts commit 21a46332f1.

* add code comment
2024-08-07 20:01:55 -07:00
aliceliu
7f57dd5a30 Support using the measure label when using create_metric option (#10536) 2024-08-07 15:06:46 -05:00
Peter Webb
56bfbeaedd Depend on snowplow-tracker rather than our old fork, minimal-snowplow-tracker. (#10530) 2024-08-07 14:55:42 -04:00
Michelle Ark
1dd26e79af deprecations.buffer: respect --quiet and --warn-error-options for deprecations (#10534) 2024-08-07 11:16:40 -04:00
Quigley Malcolm
86223609dd Parameterized testing examples utilizing happy path fixture (#10480)
* sketch

* Bring back the happy path fixture snapshot file

The commit c783a86 removed the snapshot file from the happy path fixture.
This was done because the snapshot was breaking the tests we were adding,
`test_run_commands`. However this broke `test_ls` in `test_list.py`. In order
to move forward, we need everything to be working. Maybe the idea was to delete
the `test_list.py` file, however that is not noted anywhere, and was not done.
Thus this commit ensures that test is not broken nor or new tests.

* Create conftest for `functional` tests so that happy path fixtures are accessible

* Format `test_commands.py` and update imports to appease pre-commit hooks

* Parametrize `test_run_command` to make it easier to see which command is failing (if any)

* Update the setup for `TestRunCommands.test_run_command` to be more formulaic

* Add test to ensure resource types are selectable

* Fix docstring formatting in TestRunCommands

* Fixup documentation for test_commands.py

---------

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2024-08-07 10:05:25 -05:00
Peter Webb
21a46332f1 Remove undocumented property which does not pass mypy checks after annotations in dbt-common. (#10529) 2024-08-06 11:04:54 -04:00
Michelle Ark
ff2726c3b5 more defensive node.all_constraints access (#10508) 2024-07-31 20:02:27 -04:00
Courtney Holcomb
014444dc18 Bump DSI version to release new time spine validations (#10507) 2024-07-30 15:52:40 -07:00
Kshitij Aranke
25c2042dc9 Bump dbt-adapters to 1.3.0 (#10499) 2024-07-29 18:57:24 +01:00
Courtney Holcomb
0a160fc27a Support time spine configs for sub-daily granularity (#10483) 2024-07-29 13:39:39 -04:00
Michelle Ark
c598741262 Bump dbt common 1.6 (#10489) 2024-07-26 13:51:34 -04:00
Courtney Holcomb
f9c2b9398f Remove newlines from JSON schema files (#10486) 2024-07-26 13:36:13 -04:00
Michelle Ark
cab6dabbc7 parse + compile constraint.to and constraint.to_columns on foreign key constraints (#10414) 2024-07-25 10:56:17 -04:00
nakamichi
e1621ebc54 Fix typing for artifact schemas (#10443) 2024-07-24 18:22:21 -04:00
Michelle Ark
cd90d4493c add predicate to EventCatcher test util (#10482) 2024-07-23 17:55:30 -04:00
Kshitij Aranke
560d151dcd [Tidy First] Update PR template punctuation (#10479) 2024-07-23 19:32:52 +01:00
Doug Beatty
229c537748 Update pytest examples for contributors (#10478) 2024-07-23 11:42:31 -06:00
Mila Page
79ad0a3243 Support for redshift 821 (#10448)
* Add breakpoint.

* Move breakpoint.

* Add fix

* Add changelog.

* Avoid sorting for the string case.

* Add unit test.

* Fix test.

* add good unit tests for coverage of sort method.

* add sql format coverage.

* Modify behavior to log a warning and proceed.

* code review comments.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2024-07-22 13:54:46 -07:00
Gerda Shank
c668846404 Modify snapshot config to allow using schema/database/alias macros (#10439) 2024-07-18 15:56:51 -04:00
igorvoltaic
c4958de166 test: cover additional listagg cases (#10445)
* test: cover addidtional listagg cases

* fix embarrasement

---------

Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
2024-07-18 10:42:22 -04:00
Gerda Shank
33161a3035 Fix warn error options being invalid when warn or error set to none (#10453)
* Fix exclusive_primary_alt_value_setting to set warn_error_options correctly

* Add test

* Changie

* Fix unit test

* Replace conversion method

* Refactor normalize_warn_error_options
2024-07-17 14:13:34 -07:00
Gerda Shank
471b816dcd Fix multiple semantic models with generated metrics (#10451) 2024-07-17 16:09:06 -04:00
Michelle Ark
bef2d20c21 bump black in dev-requirements and pre-commit-config (#10407) 2024-07-17 12:01:14 -04:00
Michelle Ark
2a26fabfdf [tidy first] generate protos w protoc version 5.26.1 (#10456) 2024-07-17 10:35:38 -04:00
Courtney Holcomb
4c7d922a6d Add Metric.time_granularity to metric spec (#10378) 2024-07-16 13:35:20 -04:00
Michelle Ark
b03291548a update test__model_contract_true (#10449) 2024-07-16 11:32:00 -04:00
Chenyu Li
a7af3b3831 Revert "Support for redshift 821" (#10446) 2024-07-15 17:21:38 +01:00
Mila Page
6e4564ab05 Support for redshift 821 (#10366)
* Add changelog.

* Avoid sorting for the string case.

* add good unit tests for coverage of sort method.

* add sql format coverage.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2024-07-12 14:55:34 -07:00
Mirna Wong
1aeff2c58f GitHub issue template for documentation internal to dbt-core (#10404) 2024-07-11 18:38:25 +01:00
Gerda Shank
601fee0d5f Global CLI flag should override env var flag (#10423) 2024-07-10 15:25:19 -04:00
Emily Rockman
88b8b10df1 Update README.md (#10420)
* Update README.md

* update workflow
2024-07-08 20:44:36 -05:00
Gerda Shank
4ea0e1007c [Tidy first] Add some mypy types to parser files (#10380) 2024-07-08 10:19:39 -04:00
Gerda Shank
a309283a7c Add additional logging information to capture skipped ephemeral model info (#10390) 2024-07-03 12:28:34 -04:00
Emily Rockman
b10fa79ae8 add testing slack notification (#10400) 2024-07-03 09:10:27 -05:00
Quigley Malcolm
37e2725038 [TIDY-FIRST] Fix isort for dbt-semantic-interfaces (#10394)
* Correct `isort` configuration to include dbt-semantic-interfaces as internal

We thought we were already doing this. However, we accidentally missed the last
`s` of `dbt-semantic-interfaces`, so imports from dbt-semantic-interfaces were not
being identified as an internal package by isort. This fixes that.

* Run isort using updated configs to mark `dbt-semantic-interfaces` as included
2024-07-02 11:36:27 -07:00
Emily Rockman
37fd299ad0 skip more tests (#10392) 2024-07-02 10:31:16 -05:00
Courtney Holcomb
a94027acea Add CumulativeTypeParams & sub-daily granularities to semantic manifest (#10350) 2024-07-01 14:54:23 -07:00
Gerda Shank
b59c9075e2 Tweak project-test-config deprecation test for misc deprecations (#10391) 2024-07-01 14:52:14 -04:00
Gerda Shank
c215697a02 Limit data test deprecation warning to root_project (#10375) 2024-06-28 09:43:55 -04:00
aliceliu
d936a630c1 Make time spine model name a constant (#10363) 2024-06-25 16:11:49 -07:00
Quigley Malcolm
11ee2b9c42 Fix setting silence of warn_error_options via dbt_project.yaml flags (#10359)
* Fix `test_can_silence` tests in `test_warn_error_options.py` to ensure silencing

We're fixing an issue wherein `silence` specifications in the `dbt_project.yaml`
weren't being respected. This was odd since we had tests specifically for this.
It turns out the tests were broken. Essentially the warning was instead being raised
as an error due to `include: 'all'`. Then because it was being raised as an error,
the event wasn't going through the logger. We were only asserting in these tests that
the silenced event wasn't going through the logger (which it wasn't) so everything
"appeared" to be working. Unfortunately everything wasn't actually working. This is now
highlighted because `test_warn_error_options::TestWarnErrorOptionsFromProject:test_can_silence`
is now failing with this commit.

* Fix setting `warn_error_options` via `dbt_project.yaml` flags.

Back when I did the work for #10058 (specifically c52d6531) I thought that
the `warn_error_options` would automatically be converted from the yaml
to the `WarnErrorOptions` object as we were building the `ProjectFlags` object,
which holds `warn_error_options`, via `ProjectFlags.from_dict`. And I thought
this was validated by the `test_can_silence` test added in c52d6531. However,
there were two problems:

1. The definition of `warn_error_options` on `PrjectFlags` is a dict, not a
`WarnErrorOptions` object
2. The `test_can_silence` test was broken, and not testing what I thought

The quick fix (this commit) is to ensure `silence` is passed to `WarnErrorOptions`
instantiation in `dbt.cli.flags.convert_config`. The better fix would be to make
the `warn_error_options` of `ProjectFlags` a `WarnErrorOptions` object instead of
a dict. However, to do this we first need to update dbt-common's `WarnErrorOptions`
definition to default `include` to an empty list. Doing so would allow us to get rid
of `convert_config` entirely.
2024-06-25 13:53:18 -07:00
Emily Rockman
64c59476f4 pin versions so CI and local dev match (#10358) 2024-06-24 11:25:48 -05:00
Chenyu Li
2bae05b8ed Configure Codecov to check by components 2024-06-21 15:11:59 -07:00
Emily Rockman
ca163c3d6e skip test until we can rewrite the strategy (#10348) 2024-06-20 17:22:58 -05:00
Quigley Malcolm
9a796aa202 Add unit tests for Graph class (#10338) 2024-06-20 13:17:21 -07:00
Emily Rockman
51ff85bb2d Support DIFF as a recording mode (#10263)
* add diffing

* WIP

* change file name

* Update core/dbt/cli/requires.py
2024-06-20 14:02:28 -05:00
Gerda Shank
d389ff1450 Remove unnecessary IntermediateSnapshotNode and EmptySnapshotConfig (#10332) 2024-06-19 16:21:03 -04:00
Kshitij Aranke
4415731da4 Add get_catalog_for_single_relation to MockAdapter (#10336) 2024-06-18 22:09:44 +01:00
Michelle Ark
0fdc83af9d move non-circular import to top of manifest.py (#10333) 2024-06-18 16:57:12 -04:00
Gerda Shank
71a8a41104 Implement state:modified for saved queries (#10295) 2024-06-17 23:18:56 -04:00
Quigley Malcolm
da19d7ba9f Add initial unit tests for ModelRunner class (#10196)
* Add unit test for `ModelRunner.print_result_line`

* Add (and skip) unit test for `ModelRunner.execute`

An attempt at testing `ModelRunner.execute`. We should probably also be
asserting that the model has been executed. However before we could get there,
we're running into runtime errors during `ModelRunner.execute`. Currently the
struggle is ensuring the adapter exists in the global factory when `execute`
goes looking for. The error we're getting looks like the following:
```
    def test_execute(self, table_model: ModelNode, manifest: Manifest, model_runner: ModelRunner) -> None:
>       model_runner.execute(model=table_model, manifest=manifest)

tests/unit/task/test_run.py:121:
----
core/dbt/task/run.py:259: in execute
    context = generate_runtime_model_context(model, self.config, manifest)
core/dbt/context/providers.py:1636: in generate_runtime_model_context
    ctx = ModelContext(model, config, manifest, RuntimeProvider(), None)
core/dbt/context/providers.py:834: in __init__
    self.adapter = get_adapter(self.config)
venv/lib/python3.10/site-packages/dbt/adapters/factory.py:207: in get_adapter
    return FACTORY.lookup_adapter(config.credentials.type)
---`
self = <dbt.adapters.factory.AdapterContainer object at 0x106e73280>, adapter_name = 'postgres'

    def lookup_adapter(self, adapter_name: str) -> Adapter:
>       return self.adapters[adapter_name]
E       KeyError: 'postgres'

venv/lib/python3.10/site-packages/dbt/adapters/factory.py:132: KeyError
```

* Add `postgres_adapter` fixture for use in `TestModelRunner`

Previously we were running into an issue where the during `ModelRunner.execute`
the mock_adapter we were using wouldn't be found in the global adapter
factory. We've gotten past this error by supply a "real" adapter, a
`PostgresAdapter` instance. However we're now running into a new error
in which the materialization macro can't be found. This error looks like
```
model_runner = <dbt.task.run.ModelRunner object at 0x106746650>

    def test_execute(
        self, table_model: ModelNode, manifest: Manifest, model_runner: ModelRunner
    ) -> None:
>       model_runner.execute(model=table_model, manifest=manifest)

tests/unit/task/test_run.py:129:
----
self = <dbt.task.run.ModelRunner object at 0x106746650>
model = ModelNode(database='dbt', schema='dbt_schema', name='table_model', resource_type=<NodeType.Model: 'model'>, package_na...ected'>, constraints=[], version=None, latest_version=None, deprecation_date=None, defer_relation=None, primary_key=[])
manifest = Manifest(nodes={'seed.pkg.seed': SeedNode(database='dbt', schema='dbt_schema', name='seed', resource_type=<NodeType.Se...s(show=True, node_color=None), patch_path=None, arguments=[], created_at=1718229810.21914, supported_languages=None)}})

    def execute(self, model, manifest):
        context = generate_runtime_model_context(model, self.config, manifest)

        materialization_macro = manifest.find_materialization_macro_by_name(
            self.config.project_name, model.get_materialization(), self.adapter.type()
        )

        if materialization_macro is None:
>           raise MissingMaterializationError(
                materialization=model.get_materialization(), adapter_type=self.adapter.type()
            )
E           dbt.adapters.exceptions.compilation.MissingMaterializationError: Compilation Error
E             No materialization 'table' was found for adapter postgres! (searched types 'default' and 'postgres')

core/dbt/task/run.py:266: MissingMaterializationError
```

* Add spoofed macro fixture `materialization_table_default` for `test_execute` test

Previously the `TestModelRunner:test_execute` test was running into a runtime error
do to the macro `materialization_table_default` macro not existing in the project. This
commit adds that macro to the project (though it should ideally get loaded via interactions
between the manifest and adapter). Manually adding it resolved our previous issue, but created
a new one. The macro appears to not be properly loaded into the manifest, and thus isn't
discoverable later on when getting the macros for the jinja context. This leads to an error
that looks like the following:
```
model_runner = <dbt.task.run.ModelRunner object at 0x1080a4f70>

    def test_execute(
        self, table_model: ModelNode, manifest: Manifest, model_runner: ModelRunner
    ) -> None:
>       model_runner.execute(model=table_model, manifest=manifest)

tests/unit/task/test_run.py:129:
----
core/dbt/task/run.py:287: in execute
    result = MacroGenerator(
core/dbt/clients/jinja.py:82: in __call__
    return self.call_macro(*args, **kwargs)
venv/lib/python3.10/site-packages/dbt_common/clients/jinja.py:294: in call_macro
    macro = self.get_macro()
---
self = <dbt.clients.jinja.MacroGenerator object at 0x1080f3130>

    def get_macro(self):
        name = self.get_name()
        template = self.get_template()
        # make the module. previously we set both vars and local, but that's
        # redundant: They both end up in the same place
        # make_module is in jinja2.environment. It returns a TemplateModule
        module = template.make_module(vars=self.context, shared=False)
>       macro = module.__dict__[get_dbt_macro_name(name)]
E       KeyError: 'dbt_macro__materialization_table_default'

venv/lib/python3.10/site-packages/dbt_common/clients/jinja.py:277: KeyError
```

It's becoming apparent that we need to find a better way to either mock or legitimately
load the default and adapter macros. At this point I think I've exausted the time box
I should be using to figure out if testing the `ModelRunner` class is possible currently,
with the result being more work has yet to be done.

* Begin adding the `LogModelResult` event catcher to event manager class fixture
2024-06-17 17:06:52 -07:00
Michelle Ark
1475abb1cb add empty to context flags (#10315) 2024-06-17 12:09:45 -04:00
Matthew McKnight
27b2f965dd Call adapter.pre_model_hook + adapter.post_model_hook within tests (#10258)
* init push for issue 10198

* add changelog

* add unit tests based on michelle example

* add data_tests, and post_hook unit tests

* pull creating macro_func out of try call

* revert last commit

* pull macro_func definition back out of try

* update code formatting
2024-06-14 14:17:09 -05:00
Quigley Malcolm
100352d6b4 Stop mis-identifying unit_test config paths in dbt_project.yaml as unused (#10312)
* Add basic semantic layer fixture nodes to unit test `manifest` fixture

We're doing this in preperation to a for a unit test which will be testing
these nodes (as well as others) and thus we want them in the manifest.

* Add `WhereFilterInteresection` to `QueryParams` of `saved_query` fixture

In the previous commit, 58990aa450, we added
the `saved_query` fixture to the `manifest` fixture. This broke the test
`tests/unit/parser/test_manifest.py::TestPartialParse::test_partial_parse_by_version`.
It broke because the `Manifest.deepcopy` manifest basically dictifies things. When we were
dictifying the `QueryParams` of the `saved_query` before, the `where` key was getting
dropped because it was `None`. We'd then run into a runtime error on instantiation of the
`QueryParams` because although `where` is declared as _optional_, we don't set a default
value for it. And thus, kaboom :(

We should probably provide a default value for `where`. However that is out of scope
for this branch of work.

* Fix `test_select_fqn` to account for newly included semantic model

In 58990aa450 we added a semantic model
to the `manifest` fixture. This broke the test
`tests/unit/graph/test_selector_methods.py::test_select_fqn` because in
the test it selects nodes based on the string `*.*.*_model`. The newly
added semantic model matches this, and thus needed to be added to the
expected results.

* Add unit tests for `_warn_for_unused_resource_config_paths` method

Note: At this point the test when run with for a `unit_test` config
taht should be considered used, fails. This is because it is not being
properly identified as used.

* Include `unit_tests` in `Manifest.get_resouce_fqns`

Because `unit_tests` weren't being included in calls to `Manifest.get_resource.fqns`,
it always appeared to `_warn_for_unused_resource_config_paths` that there were no
unit tests in the manifest. Because of this `_warn_for_unused_resource_config_paths` thought
that _any_ `unit_test` config in `dbt_project.yaml` was unused.
2024-06-14 09:27:45 -07:00
FishtownBuildBot
8ee8b2560a Add most recent dbt-docs changes (#10307)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2024-06-14 16:29:11 +01:00
Michelle Ark
d4a6482091 deflake source freshness hooks test (#10308) 2024-06-13 11:25:48 -04:00
Chenyu Li
8639290108 update on_skip to adjust for node that do not have schema (#10302)
* update on_skip to adjust for node that do not have schema

* changelog
2024-06-12 14:16:54 -07:00
Chenyu Li
e699f5d042 create a happy path project fixture (#10291) 2024-06-11 14:41:11 -07:00
Kshitij Aranke
e977b3eee5 Move CatalogKey, ColumnMetadata, ColumnMap, CatalogTable to dbt-common (#10253) 2024-06-11 05:07:57 +01:00
Gerda Shank
c5be8e2a93 Improve handling of error when loading schema file which is a list (#10285) 2024-06-10 15:53:24 -04:00
Quigley Malcolm
bff116dbed Migrate selector tests to pytest (#10282)
* Rename `maniest` fixture in `test_selector` to `mock_manifest`

We have a globally available `manifest` fixture in our unit tests. In the
coming commits we're going to add tests to the file which use the gloablly
available `manifest` fixture. Prior to this commit, the locally defined
`manifest` fixture was taking precidence. To get around this, the easiest
solution was to rename the locally defined fixture.

I had tried to isolate the locally defined fixture by moving it, and the relevant
tests to a test class like `TestNodeSelector`. However because of _how_ the relevant
tests were parameterized, this proved difficult. Basically for readability we define
a variable which holds a list of all the parameterization variables. By moving to a
test class, the definition of the variables would have had to be defined directly in
the parameterization macro call. Although possible, it made the readability slighty
worse. It might be worth doing anyway in the long run, but instead I used a less heavy
handed alternative (already described)

* Improve type hinting in `tests/unit/utils/manifest.py`

* Ensure `args` get set from global flags for `runtime_config` fixture in unit tests

The `Compiler.compile` method accesses `self.config.args.which`. The `config`
is the `RuntimeConfig` the `Compiler` was instantiated with. Our `runtime_config`
fixture was being instatiated with an empty dict for the `args` property. Thus
a `which` property of the args wasn't being made avaiable, and if `compile` was run
a runtime error would occur. To solve this, we've begun instantiating the args from
the global flags via `get_flags()`. This works because we ensure the `set_test_flags`
fixture is run first which calls `set_from_args`.

* Create a `make_manifest` utility function for use in unit tests and fixture creation

* Refactor `Compiler` and `NodeSelector` tests in `test_selector.py` to use pytesting methodology

* Remove parsing tests that exist in `test_selector.py`

We had some tests in `test_selector.py::GraphTest` that didn't add
anything ontop of what was already being tested else where in the file
except the parsing of models. However, the tests in `test_parser.py::ModelParserTest`
cover everything being tested here (and then some). Thus these tests
in `test_selector.py::GraphTest` are unnecessary and can be deleted.

* Move `test__partial_parse` from `test_selector.py` to `test_manifest.py`

There was a test `test__partial_parse` in `test_selector.py` which tested
the functionality of `is_partial_parsable` of the `ManifestLoader`. This
doesn't really make sense to exist in `test_selector.py` where we are
testing selectors. We test the `ManifestLoader` class in `test_manifest.py`
which seemed like a more appropriate place for the test. Additionally we
renamed the test to `test_is_partial_parsable_by_version` to more accurately
describe what is being tested.

* Make `test_selector`'s manifest fixture name even more specific

* Add type hint to `expected_nodes` in `test_selector.py` tests

In the test `tests/unit/graph/test_selector.py::TestCompiler::test_two_models_simple_ref`
we have a variable `expected_nodes` that we are setting via a list comprehension.
At a glance it isn't immediately obvious what `expected_nodes` actually is. It's a
list, but a list of what? One suggestion was to explicitly write out the list of strings.
However, I worry about the brittleness of doing so. That might be the way we head long
term, but as a compromise for now, I've added a type hint the the variable definition.
2024-06-10 10:09:33 -07:00
Gerda Shank
4df120e40e Fix snapshot config to work in yaml files (#10275) 2024-06-07 17:22:05 -04:00
Chenyu Li
e53420c1d0 Update semantic interface requirement (#10276)
* Update semantic interface requirement

* Update setup.py
2024-06-07 13:27:46 -07:00
Nev Delap
88ccc8a447 Fire skipped events at debug level (#10244)
* Fire skipped events at debug level

Closes https://github.com/dbt-labs/dbt-core/issues/8774

* add changelog entry

* Update to work with 1.9.*.

* Add tests for --fail-fast not showing skip messages unless --debug.

* Update test that works by itself, but assumes to much to work in integration tests.

---------

Co-authored-by: Scott Gigante <scottgigante@users.noreply.github.com>
2024-06-06 16:47:35 -07:00
Gerda Shank
a98059967d Fix issues with inline nodes and selectors (#10264) 2024-06-06 16:18:07 -04:00
Matthew McKnight
b680c7ae95 [Feature] add generic config for generic tests (#10245)
* init push arbitrary configs for generic tests pr

* iterative work

* initial test design attempts

* test reformatting

* test rework, have basic structure for 3 of 4 passing, need to figure out how to best represent same key error, failing correctly though

* swap up test formats for new config dict and mixed varitey to run of dbt parse and inspecting the manifest

* modify tests to get passing, then modify the TestBuilder class work from earlier to be more dry

* add changelog

* modify code to match suggested changes around seperate methods and test id fix

* add column_name reference to init for deeper nested _render_values can use the input

* add type annotations

* feedback based on mike review
2024-06-05 10:37:37 -05:00
Chenyu Li
a677abd5e8 Move to test concrete class over abstract class (#10243) 2024-06-04 14:48:09 -07:00
Kshitij Aranke
8c850b58cb update dbt_utils version to 1.2.0 (#10260) 2024-06-04 22:24:41 +01:00
Quigley Malcolm
a34267f54b Create RuntimeConfig fixture (#10242)
* Create `runtime_config` fixture and necessary upstream fixtures

* Check for better scoped `ProjectContractError` in test_runtime tests

Previously in `test_unsupported_version_extra_config` and
`test_archive_not_allowed` we were checking for `DbtProjectError`. This
worked because `ProjectContractError` is a subclass of `DbtProjectError`.
However, if we check for `DbtProjectError` in these tests than, some tangential
failure which raises a `DbtProejctError` type error would go undetected. As
we plan on modifying these tests to be pytest in the coming commits, we want to
ensure that the tests are succeeding for the right reason.

* Convert `test_str` of `TestRuntimeConfig` to a pytest test using fixtures

* Convert `test_from_parts` of `TestRuntimeConfig` to a pytest test using fixtures

While converting `test_from_parts` I noticed the comment
>  TODO(jeb): Adapters must assert that quoting is populated?

This led me to beleive that `quoting` shouldn't be "fully" realized
in our project fixture unless we're saying that it's gone through
adapter instantiation. Thus I update the `quoting` on our project
fixture to be an empty dict. This change affected `test__str__` in
`test_project.py` which we thus needed to update accordingly.

* Convert runtime version specifier tests to pytest tests and move to test_project

We've done two things in this commit, which arguably _should_ have been done in
two commits. First we moved the version specifier tests from `test_runtime.py::TestRuntimeConfig`
to `test_project.py::TestGetRequiredVersion` this is because what is really being
tested is the `_get_required_version` method. Doing it via `RuntimeConfig.from_parts` method
made actually testing it a lot harder as it requires setting up more of the world and
running with a _full_ project config dict.

The second thing we did was convert it from the old unittest implementation to a pytest
implementation. This saves us from having to create most of the world as we were doing
previously in these tests.

Of note, I did not move the test `test_unsupported_version_range_bad_config`. This test
is a bit different from the rest of the version specifier tests. It was introduced in
[1eb5857811](1eb5857811)
of [#2726](https://github.com/dbt-labs/dbt-core/pull/2726) to resolve [#2638](https://github.com/dbt-labs/dbt-core/issues/2638).
The focus of #2726 was to ensure the version specifier checks were run _before_ the validation
of the `dbt_project.yml`. Thus what this test is actually testing for is order of
operations at parse time. As such, this is really more a _functional_ test than a
unit test. In the next commit we'll get this test moved (and renamed)

* Create a better test for checking that version checks come before project schema validation

* Convert `test_get_metadata` to pytest test

* Refactor `test_archive_not_allowed` to functional test

We do already have tests that ensure "extra" keys aren't allowed in
the dbt_project.yaml. This test is different because it's checking that
a specific key, `archive`, isn't allowed. We do this because at one point
in time `archive` _was_ an allowed key. Specifically, we stopped allowing
`archive` in dbt-core 0.15.0 via commit [f26948dd](f26948dde2).
Given that it's been 5 years and a major version, we could probably remove
this test, but let's keep it around unless we start supporting `archive` again.

* Convert `warn_for_unused_resource_config_paths` tests to use pytest
2024-05-31 13:47:28 -07:00
Doug Beatty
155482851a Button to open an issue for the docs.getdbt.com documentation repo (#10239) 2024-05-30 06:14:34 -06:00
Kshitij Aranke
81386a7a43 Move StatsItem, StatsDict, TableMetadata to dbt-common (#10238) 2024-05-30 02:22:41 +01:00
Emily Rockman
d8e38c1a1d add recording types (#10175)
* add new mode

* reflect dbt-common changes

* changelog

* rename var
2024-05-29 14:53:06 -05:00
Michelle Ark
3e37d77780 [Fix] Renaming or removing a contracted model raises a BreakingChange warning/error (#10221) 2024-05-28 15:48:21 -04:00
Michelle Ark
e0783c2922 Fix: prefer disabled project nodes to external node (#10223) 2024-05-28 11:37:33 -04:00
Michelle Ark
c2d4643f9d add --host flag to docs serve (#10228) 2024-05-28 11:11:56 -04:00
Michelle Ark
84456f50f6 Fix: Order-insensitive unit test equality assertion for expected/actual with multiple nulls (#10202) 2024-05-24 10:12:26 -04:00
Michelle Ark
fb10bb4aea Maximally parallelize dbt clone (#10129) 2024-05-23 11:06:04 -04:00
FishtownBuildBot
366d4ad04a Add most recent dbt-docs changes (#10217) 2024-05-22 15:41:57 -07:00
Michelle Ark
0c08d7a19a change port bind and add a unittest (#10208) 2024-05-22 10:00:54 -04:00
Chenyu Li
eb71ad9a89 Remove Print and use event system (#10131) 2024-05-21 15:12:00 -07:00
Chenyu Li
ad6120080a Reorganize tests to match their code location (#10191) 2024-05-21 14:50:51 -07:00
Chenyu Li
6813007e28 add saved query resource type (#10168)
* add saved query resource type

* nits
2024-05-21 13:57:30 -07:00
Quigley Malcolm
09243d1e5d Improved flags fixturing for for repository unit tests (#10190)
* Add fixtures for setting and resettign flags for unit tests

* Remove unnecessary `set_from_args` in non `unittest.TestCase` based unit tests

In the previous commit we added a pytest fixture which sets and tears down
the global flags arg via `set_from_args` for every pytest based unit test.
Previously we had added a `set_from_args` in tests or test files to reset
the global flags from if they were modified by a previous test. This is no
longer necessary because of the work done in the previous commit.

Note: We did not modify any tests that use the `unittest.TestCase` class
because they don't use pytest fixtures. Thus those tests need to continue
operating as they currently do until we shift them to pytest unit tests.

* Utilize the new `args_for_flags` fixture for setting of flags in `test_contracts_graph_parsed.py`

* Convert `test_compilation.py` from `TestCase` tests to pytest tests

We did this so in the next commit we can drop the unnecessary `set_from_args`
in the next commit. That will be it's own commit because converting these
tests is a restructuring that doing separately makes things easier to follow.
That is to say, all changes in this commit were just to convert the tests to
pytest, no other changes were made.

* Drop unnecessary `set_from_args` in `test_compilation.py`

* Add return types to all methods in `test_compilation.py`

* Reduce imports from `compilation` in `test_compilation.py`

* Update `test_logging.py` now that we don't need to worry about global flags

* Conditionally import `Generator` type for python 3.8

In python 3.9 `Generator` was moved to `collections.abc` and deprecated
in `typing`. We still support 3.8 and thus need to be conditionally
importing `Generator`. We should remove this in the future when we drop
support for 3.8.
2024-05-21 09:51:13 -07:00
Peter Webb
0d297c2537 Add more accurate RSS high water mark measurement for Linux (#10177)
* Add more accurate RSS high water mark measurement for Linux

* Add changelog entry.

* Checks to avoid exception based flow control, per review.

* Fix review nit.
2024-05-21 11:01:22 -04:00
Quigley Malcolm
341803d784 Standardize event catching during repository unit tests (#10153)
* Add unit test to assert `setup_config_logger` clears the event manager state

* Move `setup_event_logger` tests from `test_functions.py` to `test_logging.py`

* Move `EventCatcher` to `tests.utils` for use in unit and functional tests

* Update fixture mocking global event manager to instead clear it

Previously we had started _mocking_ the global event manager. We did this
because we though that meant anything we did to the global event manager,
including modifying it via things like `setup_event_logger`, would be
automatically cleaned up at the end of any test using the fixture because
the mock would go away. However, this understanding of fixtures and mocking
was incorrect, and the global event manager wouldn't be properly isolated/reset.
Thus we changed to fixture to instead cleanup the global event manager before
any test that uses it and by using `autouse=True` in the fixture definition
we made it so that every unit test uses the fixture.

Note this will no longer be viabled if we every multi-thread our unit testing as
the event manager isn't actually isolated, and thus two tests could both modify
the event manager at the same time.
2024-05-16 15:52:19 -07:00
Quigley Malcolm
32a7f82772 Repository unit tests for ManifestLoader.get_full_manifest (#10147)
* Add test for different `write_perf_info` values to `get_full_manifest`

* Add test for different `reset` values to `get_full_manifest`

* Abstract required mocks for `get_full_manifest` tests to reduce duplication

There are a set of required mocks that `get_full_manifest` unit tests need.
Instead of doing these mocks in each test, we've abstracted these mocks into
a reusable function. I did try to do this as a fixture, but for some reaosn
the mocks didn't actually propagate when I did that.

* Add test for different `PARTIAL_PARSE_FILE_DIFF` values to `get_full_manifest`

* Refactor mock fixtures in `test_manifest.py` to make them more widely available

* Convert `set_required_mocks` of `TestGetFullManifest` into a fixture

This wasn't working before, but it does now. Not sure why.
2024-05-15 12:02:37 -07:00
Doug Beatty
d21ff38ba1 Rename the "Explain It Like I'm 64" folder to "guides" (#10141) 2024-05-14 10:41:02 -06:00
Kshitij Aranke
2f7ef5f120 [Tidy First] Remove wheel and twine from dev-requirements.txt (#10140) 2024-05-14 17:31:02 +01:00
Kshitij Aranke
751139d8c1 Fix #9907: Add retry to tox to reduce flaky tests due to network failures (#10137) 2024-05-14 12:57:18 +01:00
Emily Rockman
03b621f390 remove logbook (#10119)
* remove logbook

* some cleanup, some bug fixes
2024-05-13 10:37:32 -05:00
475 changed files with 17717 additions and 7386 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.9.0a1
current_version = 1.9.0b2
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

136
.changes/1.9.0-b1.md Normal file
View File

@@ -0,0 +1,136 @@
## dbt-core 1.9.0-b1 - October 01, 2024
### Breaking Changes
- Fix changing the current working directory when using dpt deps, clean and init. ([#8997](https://github.com/dbt-labs/dbt-core/issues/8997))
### Features
- serialize inferred primary key ([#9824](https://github.com/dbt-labs/dbt-core/issues/9824))
- Add unit_test: selection method ([#10053](https://github.com/dbt-labs/dbt-core/issues/10053))
- Maximally parallelize dbt clone in clone command" ([#7914](https://github.com/dbt-labs/dbt-core/issues/7914))
- Add --host flag to dbt docs serve, defaulting to '127.0.0.1' ([#10229](https://github.com/dbt-labs/dbt-core/issues/10229))
- Update data_test to accept arbitrary config options ([#10197](https://github.com/dbt-labs/dbt-core/issues/10197))
- add pre_model and post_model hook calls to data and unit tests to be able to provide extra config options ([#10198](https://github.com/dbt-labs/dbt-core/issues/10198))
- add --empty value to jinja context as flags.EMPTY ([#10317](https://github.com/dbt-labs/dbt-core/issues/10317))
- Warning message for snapshot timestamp data types ([#10234](https://github.com/dbt-labs/dbt-core/issues/10234))
- Support cumulative_type_params & sub-daily granularities in semantic manifest. ([#10360](https://github.com/dbt-labs/dbt-core/issues/10360))
- Add time_granularity to metric spec. ([#10376](https://github.com/dbt-labs/dbt-core/issues/10376))
- Support standard schema/database fields for snapshots ([#10301](https://github.com/dbt-labs/dbt-core/issues/10301))
- Support ref and source in foreign key constraint expressions, bump dbt-common minimum to 1.6 ([#8062](https://github.com/dbt-labs/dbt-core/issues/8062))
- Support new semantic layer time spine configs to enable sub-daily granularity. ([#10475](https://github.com/dbt-labs/dbt-core/issues/10475))
- Include models that depend on changed vars in state:modified, add state:modified.vars selection method ([#4304](https://github.com/dbt-labs/dbt-core/issues/4304))
- Add support for behavior flags ([#10618](https://github.com/dbt-labs/dbt-core/issues/10618))
- Enable `--resource-type` and `--exclude-resource-type` CLI flags and environment variables for `dbt test` ([#10656](https://github.com/dbt-labs/dbt-core/issues/10656))
- Allow configuring snapshot column names ([#10185](https://github.com/dbt-labs/dbt-core/issues/10185))
- Add custom_granularities to YAML spec for time spines. ([#9265](https://github.com/dbt-labs/dbt-core/issues/9265))
- Add basic functionality for creating microbatch incremental models ([#9490](https://github.com/dbt-labs/dbt-core/issues/9490), [#10635](https://github.com/dbt-labs/dbt-core/issues/10635), [#10637](https://github.com/dbt-labs/dbt-core/issues/10637), [#10638](https://github.com/dbt-labs/dbt-core/issues/10638), [#10636](https://github.com/dbt-labs/dbt-core/issues/10636), [#10662](https://github.com/dbt-labs/dbt-core/issues/10662), [#10639](https://github.com/dbt-labs/dbt-core/issues/10639))
- Execute microbatch models in batches ([#10700](https://github.com/dbt-labs/dbt-core/issues/10700))
- Create 'skip_nodes_if_on_run_start_fails' behavior change flag ([#7387](https://github.com/dbt-labs/dbt-core/issues/7387))
- Allow snapshots to be defined in YAML. ([#10246](https://github.com/dbt-labs/dbt-core/issues/10246))
- Write microbatch compiled/run targets to separate files, one per batch ([#10714](https://github.com/dbt-labs/dbt-core/issues/10714))
- Track incremental_strategy as part of model_run tracking event ([#10761](https://github.com/dbt-labs/dbt-core/issues/10761))
- Support required 'begin' config for microbatch models ([#10701](https://github.com/dbt-labs/dbt-core/issues/10701))
- Parse-time validation of microbatch configs: require event_time, batch_size, lookback and validate input event_time ([#10709](https://github.com/dbt-labs/dbt-core/issues/10709))
- Added the --inline-direct parameter to 'dbt show' ([#10770](https://github.com/dbt-labs/dbt-core/issues/10770))
- Enable `retry` support for microbatch models ([#10715](https://github.com/dbt-labs/dbt-core/issues/10715), [#10729](https://github.com/dbt-labs/dbt-core/issues/10729))
- Use unrendered database and schema source properties during state:modified, behind state_modified_compare_more_unrendered_values behavoiur flag ([#9573](https://github.com/dbt-labs/dbt-core/issues/9573))
- Ensure microbatch models respect `full_refresh` model config ([#10785](https://github.com/dbt-labs/dbt-core/issues/10785))
- Adds validations for custom_granularities to ensure unique naming. ([#9265](https://github.com/dbt-labs/dbt-core/issues/9265))
### Fixes
- Remove unused check_new method ([#7586](https://github.com/dbt-labs/dbt-core/issues/7586))
- Test case for `merge_exclude_columns` ([#8267](https://github.com/dbt-labs/dbt-core/issues/8267))
- Convert "Skipping model due to fail_fast" message to DEBUG level ([#8774](https://github.com/dbt-labs/dbt-core/issues/8774))
- Restore previous behavior for --favor-state: only favor defer_relation if not selected in current command" ([#10107](https://github.com/dbt-labs/dbt-core/issues/10107))
- Unit test fixture (csv) returns null for empty value ([#9881](https://github.com/dbt-labs/dbt-core/issues/9881))
- Fix json format log and --quiet for ls and jinja print by converting print call to fire events ([#8756](https://github.com/dbt-labs/dbt-core/issues/8756))
- Add resource type to saved_query ([#10168](https://github.com/dbt-labs/dbt-core/issues/10168))
- Fix: Order-insensitive unit test equality assertion for expected/actual with multiple nulls ([#10167](https://github.com/dbt-labs/dbt-core/issues/10167))
- Renaming or removing a contracted model should raise a BreakingChange warning/error ([#10116](https://github.com/dbt-labs/dbt-core/issues/10116))
- prefer disabled project nodes to external node ([#10224](https://github.com/dbt-labs/dbt-core/issues/10224))
- Fix issues with selectors and inline nodes ([#8943](https://github.com/dbt-labs/dbt-core/issues/8943), [#9269](https://github.com/dbt-labs/dbt-core/issues/9269))
- Fix snapshot config to work in yaml files ([#4000](https://github.com/dbt-labs/dbt-core/issues/4000))
- Improve handling of error when loading schema file list ([#10284](https://github.com/dbt-labs/dbt-core/issues/10284))
- Use model alias for the CTE identifier generated during ephemeral materialization ([#5273](https://github.com/dbt-labs/dbt-core/issues/5273))
- Implement state:modified for saved queries ([#10294](https://github.com/dbt-labs/dbt-core/issues/10294))
- Saved Query node fail during skip ([#10029](https://github.com/dbt-labs/dbt-core/issues/10029))
- DOn't warn on `unit_test` config paths that are properly used ([#10311](https://github.com/dbt-labs/dbt-core/issues/10311))
- Fix setting `silence` of `warn_error_options` via `dbt_project.yaml` flags ([#10160](https://github.com/dbt-labs/dbt-core/issues/10160))
- Attempt to provide test fixture tables with all values to set types correctly for comparisong with source tables ([#10365](https://github.com/dbt-labs/dbt-core/issues/10365))
- Limit data_tests deprecation to root_project ([#9835](https://github.com/dbt-labs/dbt-core/issues/9835))
- CLI flags should take precedence over env var flags ([#10304](https://github.com/dbt-labs/dbt-core/issues/10304))
- Fix typing for artifact schemas ([#10442](https://github.com/dbt-labs/dbt-core/issues/10442))
- Fix over deletion of generated_metrics in partial parsing ([#10450](https://github.com/dbt-labs/dbt-core/issues/10450))
- Fix error constructing warn_error_options ([#10452](https://github.com/dbt-labs/dbt-core/issues/10452))
- Do not update varchar column definitions if a contract exists ([#10362](https://github.com/dbt-labs/dbt-core/issues/10362))
- fix all_constraints access, disabled node parsing of non-uniquely named resources ([#10509](https://github.com/dbt-labs/dbt-core/issues/10509))
- respect --quiet and --warn-error-options for flag deprecations ([#10105](https://github.com/dbt-labs/dbt-core/issues/10105))
- Propagate measure label when using create_metrics ([#10536](https://github.com/dbt-labs/dbt-core/issues/10536))
- Fix state:modified check for exports ([#10138](https://github.com/dbt-labs/dbt-core/issues/10138))
- Filter out empty nodes after graph selection to support consistent selection of nodes that depend on upstream public models ([#8987](https://github.com/dbt-labs/dbt-core/issues/8987))
- Late render pre- and post-hooks configs in properties / schema YAML files ([#10603](https://github.com/dbt-labs/dbt-core/issues/10603))
- Allow the use of env_var function in certain macros in which it was previously unavailable. ([#10609](https://github.com/dbt-labs/dbt-core/issues/10609))
- Remove deprecation for tests: to data_tests: change ([#10564](https://github.com/dbt-labs/dbt-core/issues/10564))
- Fix `--resource-type test` for `dbt list` and `dbt build` ([#10730](https://github.com/dbt-labs/dbt-core/issues/10730))
- Fix unit tests for incremental model with alias ([#10754](https://github.com/dbt-labs/dbt-core/issues/10754))
- Allow singular tests to be documented in properties.yml ([#9005](https://github.com/dbt-labs/dbt-core/issues/9005))
- Ignore --empty in unit test ref/source rendering ([#10516](https://github.com/dbt-labs/dbt-core/issues/10516))
- Ignore rendered jinja in configs for state:modified, behind state_modified_compare_more_unrendered_values behaviour flag ([#9564](https://github.com/dbt-labs/dbt-core/issues/9564))
- Improve performance of infer primary key ([#10781](https://github.com/dbt-labs/dbt-core/issues/10781))
- Attempt to skip saved query processing when no semantic manifest changes ([#10563](https://github.com/dbt-labs/dbt-core/issues/10563))
- Ensure dbt retry of microbatch models doesn't lose prior successful state ([#10800](https://github.com/dbt-labs/dbt-core/issues/10800))
### Docs
- Enable display of unit tests ([dbt-docs/#501](https://github.com/dbt-labs/dbt-docs/issues/501))
- Unit tests not rendering ([dbt-docs/#506](https://github.com/dbt-labs/dbt-docs/issues/506))
- Add support for Saved Query node ([dbt-docs/#486](https://github.com/dbt-labs/dbt-docs/issues/486))
- Fix npm security vulnerabilities as of June 2024 ([dbt-docs/#513](https://github.com/dbt-labs/dbt-docs/issues/513))
### Under the Hood
- Clear error message for Private package in dbt-core ([#10083](https://github.com/dbt-labs/dbt-core/issues/10083))
- Enable use of context in serialization ([#10093](https://github.com/dbt-labs/dbt-core/issues/10093))
- Make RSS high water mark measurement more accurate on Linux ([#10177](https://github.com/dbt-labs/dbt-core/issues/10177))
- Enable record filtering by type. ([#10240](https://github.com/dbt-labs/dbt-core/issues/10240))
- Remove IntermediateSnapshotNode ([#10326](https://github.com/dbt-labs/dbt-core/issues/10326))
- Additional logging for skipped ephemeral models ([#10389](https://github.com/dbt-labs/dbt-core/issues/10389))
- bump black to 24.3.0 ([#10454](https://github.com/dbt-labs/dbt-core/issues/10454))
- generate protos with protoc version 5.26.1 ([#10457](https://github.com/dbt-labs/dbt-core/issues/10457))
- Move from minimal-snowplow-tracker fork back to snowplow-tracker ([#8409](https://github.com/dbt-labs/dbt-core/issues/8409))
- Add group info to RunResultError, RunResultFailure, RunResultWarning log lines ([#](https://github.com/dbt-labs/dbt-core/issues/))
- Improve speed of tree traversal when finding children, increasing build speed for some selectors ([#10434](https://github.com/dbt-labs/dbt-core/issues/10434))
- Add test for sources tables with quotes ([#10582](https://github.com/dbt-labs/dbt-core/issues/10582))
- Additional type hints for `core/dbt/version.py` ([#10612](https://github.com/dbt-labs/dbt-core/issues/10612))
- Fix typing issues in core/dbt/contracts/sql.py ([#10614](https://github.com/dbt-labs/dbt-core/issues/10614))
- Fix type errors in `dbt/core/task/clean.py` ([#10616](https://github.com/dbt-labs/dbt-core/issues/10616))
- Add Snowplow tracking for behavior flag deprecations ([#10552](https://github.com/dbt-labs/dbt-core/issues/10552))
- Add test utility patch_microbatch_end_time for adapters testing ([#10713](https://github.com/dbt-labs/dbt-core/issues/10713))
- Replace `TestSelector` with `ResourceTypeSelector` ([#10718](https://github.com/dbt-labs/dbt-core/issues/10718))
- Standardize returning `ResourceTypeSelector` instances in `dbt list` and `dbt build` ([#10739](https://github.com/dbt-labs/dbt-core/issues/10739))
- Add group metadata info to LogModelResult and LogTestResult ([#10775](https://github.com/dbt-labs/dbt-core/issues/10775))
### Dependencies
- Remove logbook dependency ([#8027](https://github.com/dbt-labs/dbt-core/issues/8027))
- Increase supported version range for dbt-semantic-interfaces. Needed to support custom calendar features. ([#9265](https://github.com/dbt-labs/dbt-core/issues/9265))
### Security
- Explicitly bind to localhost in docs serve ([#10209](https://github.com/dbt-labs/dbt-core/issues/10209))
### Contributors
- [@McKnight-42](https://github.com/McKnight-42) ([#10197](https://github.com/dbt-labs/dbt-core/issues/10197), [#10198](https://github.com/dbt-labs/dbt-core/issues/10198))
- [@TowardOliver](https://github.com/TowardOliver) ([#10656](https://github.com/dbt-labs/dbt-core/issues/10656))
- [@aliceliu](https://github.com/aliceliu) ([#10536](https://github.com/dbt-labs/dbt-core/issues/10536), [#10138](https://github.com/dbt-labs/dbt-core/issues/10138))
- [@courtneyholcomb](https://github.com/courtneyholcomb) ([#10360](https://github.com/dbt-labs/dbt-core/issues/10360), [#10376](https://github.com/dbt-labs/dbt-core/issues/10376), [#10475](https://github.com/dbt-labs/dbt-core/issues/10475), [#9265](https://github.com/dbt-labs/dbt-core/issues/9265), [#9265](https://github.com/dbt-labs/dbt-core/issues/9265), [#9265](https://github.com/dbt-labs/dbt-core/issues/9265))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#9824](https://github.com/dbt-labs/dbt-core/issues/9824))
- [@jeancochrane](https://github.com/jeancochrane) ([#5273](https://github.com/dbt-labs/dbt-core/issues/5273))
- [@katsugeneration](https://github.com/katsugeneration) ([#10754](https://github.com/dbt-labs/dbt-core/issues/10754))
- [@kevinneville](https://github.com/kevinneville) ([#7586](https://github.com/dbt-labs/dbt-core/issues/7586))
- [@nakamichiworks](https://github.com/nakamichiworks) ([#10442](https://github.com/dbt-labs/dbt-core/issues/10442))
- [@rariyama](https://github.com/rariyama) ([#8997](https://github.com/dbt-labs/dbt-core/issues/8997))
- [@scottgigante,nevdelap](https://github.com/scottgigante,nevdelap) ([#8774](https://github.com/dbt-labs/dbt-core/issues/8774))
- [@ttusing](https://github.com/ttusing) ([#10434](https://github.com/dbt-labs/dbt-core/issues/10434))

1
.changes/1.9.0-b2.md Normal file
View File

@@ -0,0 +1 @@
## dbt-core 1.9.0-b2 - October 07, 2024

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Fix changing the current working directory when using dpt deps, clean and init.
time: 2023-12-06T19:24:42.575372+09:00
custom:
Author: rariyama
Issue: "8997"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Remove logbook dependency
time: 2024-05-09T09:37:17.745129-05:00
custom:
Author: emmyoop
Issue: "8027"

View File

@@ -0,0 +1,7 @@
kind: Dependencies
body: Increase supported version range for dbt-semantic-interfaces. Needed to support
custom calendar features.
time: 2024-08-20T13:19:09.015225-07:00
custom:
Author: courtneyholcomb
Issue: "9265"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Enable display of unit tests
time: 2024-03-11T14:03:44.490834-04:00
custom:
Author: gshank
Issue: "501"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Unit tests not rendering
time: 2024-05-01T02:10:50.987412+02:00
custom:
Author: aranke
Issue: "506"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Add support for Saved Query node
time: 2024-05-16T22:30:36.206492-07:00
custom:
Author: ChenyuLInx
Issue: "486"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Fix npm security vulnerabilities as of June 2024
time: 2024-06-13T15:10:48.301989+01:00
custom:
Author: aranke
Issue: "513"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Maximally parallelize dbt clone
in clone command"
time: 2024-05-22T00:03:09.765977-04:00
custom:
Author: michelleark
Issue: "7914"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add --host flag to dbt docs serve, defaulting to '127.0.0.1'
time: 2024-05-27T12:44:05.040843-04:00
custom:
Author: michelleark
Issue: "10229"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Update data_test to accept arbitrary config options
time: 2024-05-31T15:08:16.431966-05:00
custom:
Author: McKnight-42
Issue: "10197"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add pre_model and post_model hook calls to data and unit tests to be able to provide extra config options
time: 2024-06-06T11:23:34.758675-05:00
custom:
Author: McKnight-42
Issue: "10198"

View File

@@ -0,0 +1,6 @@
kind: Features
body: add --empty value to jinja context as flags.EMPTY
time: 2024-06-17T10:39:48.275801-04:00
custom:
Author: michelleark
Issue: "10317"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Warning message for snapshot timestamp data types
time: 2024-06-21T14:16:35.717637-04:00
custom:
Author: gshank
Issue: "10234"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support cumulative_type_params & sub-daily granularities in semantic manifest.
time: 2024-06-25T09:51:07.983248-07:00
custom:
Author: courtneyholcomb
Issue: "10360"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add time_granularity to metric spec.
time: 2024-06-27T16:29:53.500917-07:00
custom:
Author: courtneyholcomb
Issue: "10376"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support standard schema/database fields for snapshots
time: 2024-07-12T21:45:46.06011-04:00
custom:
Author: gshank
Issue: "10301"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support ref and source in foreign key constraint expressions, bump dbt-common minimum to 1.6
time: 2024-07-19T16:18:41.434278-04:00
custom:
Author: michelleark
Issue: "8062"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support new semantic layer time spine configs to enable sub-daily granularity.
time: 2024-07-22T20:22:38.258249-07:00
custom:
Author: courtneyholcomb
Issue: "10475"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add support for behavior flags
time: 2024-08-29T13:53:20.16122-04:00
custom:
Author: mikealfare
Issue: "10618"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Enable `--resource-type` and `--exclude-resource-type` CLI flags and environment variables for `dbt test`
time: 2024-09-03T13:24:28.592837+01:00
custom:
Author: TowardOliver dbeatty10
Issue: "10656"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Allow configuring snapshot column names
time: 2024-09-03T15:41:33.167097-04:00
custom:
Author: gshank
Issue: "10185"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add custom_granularities to YAML spec for time spines.
time: 2024-09-04T18:23:20.234952-07:00
custom:
Author: courtneyholcomb
Issue: "9265"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add basic functionality for creating microbatch incremental models
time: 2024-09-11T12:10:29.822189-05:00
custom:
Author: MichelleArk QMalcolm
Issue: 9490 10635 10637 10638 10636 10662 10639

View File

@@ -0,0 +1,6 @@
kind: Features
body: Execute microbatch models in batches
time: 2024-09-13T23:21:11.935434-04:00
custom:
Author: michelleark
Issue: "10700"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Create 'skip_nodes_if_on_run_start_fails' behavior change flag
time: 2024-09-18T16:29:59.268422+01:00
custom:
Author: aranke
Issue: "7387"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Allow snapshots to be defined in YAML.
time: 2024-09-20T11:04:47.703117-04:00
custom:
Author: peterallenwebb
Issue: "10246"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Write microbatch compiled/run targets to separate files, one per batch
time: 2024-09-20T17:24:19.219556+01:00
custom:
Author: michelleark
Issue: "10714"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Track incremental_strategy as part of model_run tracking event
time: 2024-09-23T15:53:35.898928+01:00
custom:
Author: michelleark
Issue: "10761"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support required 'begin' config for microbatch models
time: 2024-09-23T15:59:03.924079+01:00
custom:
Author: michelleark
Issue: "10701"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Added the --inline-direct parameter to 'dbt show'
time: 2024-09-24T15:29:22.874496-04:00
custom:
Author: aranke peterallenwebb
Issue: "10770"

View File

@@ -0,0 +1,7 @@
kind: Features
body: 'Parse-time validation of microbatch configs: require event_time, batch_size,
lookback and validate input event_time'
time: 2024-09-24T15:46:39.83112+01:00
custom:
Author: michelleark
Issue: "10709"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Enable `retry` support for microbatch models
time: 2024-09-25T16:50:02.105069-05:00
custom:
Author: QMalcolm MichelleArk
Issue: 10715 10729

View File

@@ -0,0 +1,7 @@
kind: Features
body: Use unrendered database and schema source properties during state:modified,
behind state_modified_compare_more_unrendered_values behavoiur flag
time: 2024-09-26T14:02:06.732068+01:00
custom:
Author: michelleark
Issue: "9573"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Adds validations for custom_granularities to ensure unique naming.
time: 2024-09-26T15:10:57.907694-07:00
custom:
Author: courtneyholcomb
Issue: "9265"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Ensure microbatch models respect `full_refresh` model config
time: 2024-09-26T15:32:10.202789-05:00
custom:
Author: QMalcolm
Issue: "10785"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Test case for `merge_exclude_columns`
time: 2023-08-01T09:46:26.829362-06:00
custom:
Author: dbeatty10
Issue: "8267"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Convert "Skipping model due to fail_fast" message to DEBUG level
time: 2024-01-13T07:36:15.836294-00:00
custom:
Author: scottgigante,nevdelap
Issue: "8774"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Fix json format log and --quiet for ls and jinja print by converting print call
to fire events
time: 2024-05-16T15:39:13.896723-07:00
custom:
Author: ChenyuLInx
Issue: "8756"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add resource type to saved_query
time: 2024-05-16T22:35:10.287514-07:00
custom:
Author: ChenyuLInx
Issue: "10168"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: 'Fix: Order-insensitive unit test equality assertion for expected/actual with
multiple nulls'
time: 2024-05-22T18:28:55.91733-04:00
custom:
Author: michelleark
Issue: "10167"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Renaming or removing a contracted model should raise a BreakingChange warning/error
time: 2024-05-23T20:42:51.033946-04:00
custom:
Author: michelleark
Issue: "10116"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: prefer disabled project nodes to external node
time: 2024-05-24T13:11:35.440443-04:00
custom:
Author: michelleark
Issue: "10224"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix issues with selectors and inline nodes
time: 2024-06-05T11:16:52.187667-04:00
custom:
Author: gshank
Issue: 8943 9269

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix snapshot config to work in yaml files
time: 2024-06-07T13:46:48.383215-04:00
custom:
Author: gshank
Issue: "4000"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Improve handling of error when loading schema file list
time: 2024-06-10T13:21:30.963371-04:00
custom:
Author: gshank
Issue: "10284"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Use model alias for the CTE identifier generated during ephemeral materialization
time: 2024-06-10T20:05:22.510814008Z
custom:
Author: jeancochrane
Issue: "5273"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Saved Query node fail during skip
time: 2024-06-12T12:42:56.329073-07:00
custom:
Author: ChenyuLInx
Issue: "10029"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Implement state:modified for saved queries
time: 2024-06-12T15:21:39.851426-04:00
custom:
Author: gshank
Issue: "10294"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: DOn't warn on `unit_test` config paths that are properly used
time: 2024-06-13T18:31:17.486497-07:00
custom:
Author: QMalcolm
Issue: "10311"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix setting `silence` of `warn_error_options` via `dbt_project.yaml` flags
time: 2024-06-24T17:17:29.464865-07:00
custom:
Author: QMalcolm
Issue: "10160"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Attempt to provide test fixture tables with all values to set types correctly
for comparisong with source tables
time: 2024-06-25T17:17:37.514619-07:00
custom:
Author: versusfacit
Issue: "10365"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Limit data_tests deprecation to root_project
time: 2024-06-27T15:44:48.579869-04:00
custom:
Author: gshank
Issue: "9835"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: CLI flags should take precedence over env var flags
time: 2024-07-09T17:24:40.918977-04:00
custom:
Author: gshank
Issue: "10304"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix typing for artifact schemas
time: 2024-07-14T10:02:54.452099+09:00
custom:
Author: nakamichiworks
Issue: "10442"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix over deletion of generated_metrics in partial parsing
time: 2024-07-16T13:37:03.49651-04:00
custom:
Author: gshank
Issue: "10450"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix error constructing warn_error_options
time: 2024-07-16T17:14:27.837171-04:00
custom:
Author: gshank
Issue: "10452"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Do not update varchar column definitions if a contract exists
time: 2024-07-28T22:14:21.67712-04:00
custom:
Author: gshank
Issue: "10362"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix all_constraints access, disabled node parsing of non-uniquely named resources
time: 2024-07-31T09:51:52.751135-04:00
custom:
Author: michelleark gshank
Issue: "10509"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Propagate measure label when using create_metrics
time: 2024-08-06T17:21:10.265494-07:00
custom:
Author: aliceliu
Issue: "10536"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: respect --quiet and --warn-error-options for flag deprecations
time: 2024-08-06T19:48:43.399453-04:00
custom:
Author: michelleark
Issue: "10105"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix state:modified check for exports
time: 2024-08-13T15:42:35.471685-07:00
custom:
Author: aliceliu
Issue: "10138"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Filter out empty nodes after graph selection to support consistent selection of nodes that depend on upstream public models
time: 2024-08-16T14:08:07.426235-07:00
custom:
Author: jtcohen6
Issue: "8987"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Late render pre- and post-hooks configs in properties / schema YAML files
time: 2024-08-24T21:09:03.252733-06:00
custom:
Author: dbeatty10
Issue: "10603"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Allow the use of env_var function in certain macros in which it was previously
unavailable.
time: 2024-08-29T10:57:01.160613-04:00
custom:
Author: peterallenwebb
Issue: "10609"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Remove deprecation for tests: to data_tests: change'
time: 2024-09-05T18:02:48.086421-04:00
custom:
Author: gshank
Issue: "10564"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix `--resource-type test` for `dbt list` and `dbt build`
time: 2024-09-17T17:44:46.121032-06:00
custom:
Author: dbeatty10
Issue: "10730"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix unit tests for incremental model with alias
time: 2024-09-22T13:35:27.991398741Z
custom:
Author: katsugeneration
Issue: "10754"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Allow singular tests to be documented in properties.yml
time: 2024-09-23T19:07:58.151069+01:00
custom:
Author: aranke
Issue: "9005"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Ignore --empty in unit test ref/source rendering
time: 2024-09-23T20:20:24.151285+01:00
custom:
Author: michelleark
Issue: "10516"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Ignore rendered jinja in configs for state:modified, behind state_modified_compare_more_unrendered_values
behaviour flag
time: 2024-09-25T13:10:28.490042+01:00
custom:
Author: michelleark
Issue: "9564"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Improve performance of infer primary key
time: 2024-09-25T16:05:43.59536-04:00
custom:
Author: gshank
Issue: "10781"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Attempt to skip saved query processing when no semantic manifest changes
time: 2024-09-26T10:12:20.193453-04:00
custom:
Author: gshank
Issue: "10563"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Ensure dbt retry of microbatch models doesn't lose prior successful state
time: 2024-09-30T15:31:58.541656-05:00
custom:
Author: QMalcolm
Issue: "10800"

View File

@@ -0,0 +1,6 @@
kind: Security
body: Explicitly bind to localhost in docs serve
time: 2024-05-22T09:45:40.748185-04:00
custom:
Author: ChenyuLInx michelleark
Issue: "10209"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Make RSS high water mark measurement more accurate on Linux
time: 2024-05-19T15:59:46.700842315-04:00
custom:
Author: peterallenwebb
Issue: "10177"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Enable record filtering by type.
time: 2024-05-29T10:28:14.547624-05:00
custom:
Author: emmyoop
Issue: "10240"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove IntermediateSnapshotNode
time: 2024-06-18T14:06:52.618602-04:00
custom:
Author: gshank
Issue: "10326"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Additional logging for skipped ephemeral models
time: 2024-07-01T13:17:50.827788-04:00
custom:
Author: gshank
Issue: "10389"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: bump black to 24.3.0
time: 2024-07-16T18:48:59.651834-04:00
custom:
Author: michelleark
Issue: "10454"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: generate protos with protoc version 5.26.1
time: 2024-07-16T20:57:03.332448-04:00
custom:
Author: michelleark
Issue: "10457"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move from minimal-snowplow-tracker fork back to snowplow-tracker
time: 2024-08-06T15:54:06.422444-04:00
custom:
Author: peterallenwebb
Issue: "8409"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Add group info to RunResultError, RunResultFailure, RunResultWarning log lines
time: 2024-08-07T15:56:52.171199-05:00
custom:
Author: aranke
Issue: ""
JiraID: "364"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Improve speed of tree traversal when finding children, increasing build speed for some selectors
time: 2024-08-09T13:02:34.759905-07:00
custom:
Author: ttusing
Issue: "10434"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add test for sources tables with quotes
time: 2024-08-21T09:55:16.038101-04:00
custom:
Author: gshank
Issue: "10582"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Additional type hints for `core/dbt/version.py`
time: 2024-08-27T10:50:14.047859-05:00
custom:
Author: QMalcolm
Issue: "10612"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix typing issues in core/dbt/contracts/sql.py
time: 2024-08-27T11:31:23.749912-05:00
custom:
Author: QMalcolm
Issue: "10614"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix type errors in `dbt/core/task/clean.py`
time: 2024-08-27T11:48:10.438173-05:00
custom:
Author: QMalcolm
Issue: "10616"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add Snowplow tracking for behavior flag deprecations
time: 2024-09-11T16:27:30.293832-04:00
custom:
Author: mikealfare
Issue: "10552"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add test utility patch_microbatch_end_time for adapters testing
time: 2024-09-13T21:33:12.482336-04:00
custom:
Author: michelleark
Issue: "10713"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Replace `TestSelector` with `ResourceTypeSelector`
time: 2024-09-16T10:22:01.339462-06:00
custom:
Author: dbeatty10
Issue: "10718"

Some files were not shown because too many files have changed in this diff Show More