Compare commits

...

63 Commits

Author SHA1 Message Date
Colin
3ce8d1490e Merge branch 'main' into externalCatalogConfig 2024-10-10 14:44:35 -07:00
Colin
d76fc5751a add catalog to provider and components 2024-10-10 12:51:30 -07:00
Colin
418ce4d62f add functional test and add catalogs to source resolution 2024-10-09 12:55:45 -07:00
Quigley Malcolm
f6cdacc61e Stop making microbatch batches with filters that will never have any rows (#10826) 2024-10-08 18:56:10 -05:00
Quigley Malcolm
5db0b81da1 Track batch execution time for microbatch models (#10828)
* Begin testing that microbatch execution times are being tracked and set

* Begin tracking the execution time of batches for microbatch models

* Add changie doc

* Additional assertions in microbatch testing
2024-10-08 14:32:58 -05:00
Quigley Malcolm
fc8eb820aa Validate --event-time-start is before --event-time-end (#10820)
* Validate that `event_time_start` is before `event_time_end` when passed from CLI

Sometimes CLI options have restrictions based on other CLI options. This is the case
for `--event-time-start` and `--event-time-end`. Unfortunately, click doesn't provide
a good way for validating this, at least not that I found. Additionaly I'm not sure
if we have had anything like this previously. In any case, I couldn't find a
centralized validation area for such occurances. Thus I've gone and added one,
`validate_option_interactions`. Long term if more validations are added, we should
add this wrapper to each CLI command. For now I've only added it to the commands that
support `event_time_start` and `event_time_end`, specifically `build` and `run`.

* Add changie doc

* If `--event-time-end` is not specififed, ensure `--event-time-start` is less than the current time

* Fixup error message about event_time_start and event_time_end

* Move logic to validate `event_time` cli flags to `flags.py`

* Update validation of `--event-time-start` against `datetime.now` to use UTC
2024-10-07 14:34:42 -05:00
FishtownBuildBot
fc83f5edfa [Automated] Merged prep-release/1.9.0b2_11213923466 into target main during release process 2024-10-07 07:17:39 -04:00
Github Build Bot
8248d1eb53 Bumping version to 1.9.0b2 and generate changelog 2024-10-07 10:50:33 +00:00
Colin
288be96754 explore where/how to ingest the catalog config 2024-10-03 15:52:06 -07:00
Kshitij Aranke
6b9c1da1ae Revert "state:modified vars, behind "state_modified_compare_vars" behaviour flag" (#10793) (#10813) 2024-10-02 21:00:48 +01:00
Colin
706ff326e9 add catalog config to manifest.py 2024-10-02 08:58:51 -07:00
Colin
b5bf57b910 add catalog config to manifest.py 2024-10-02 08:57:51 -07:00
Colin
666cc3bb72 add py.typed 2024-10-01 13:58:54 -07:00
Courtney Holcomb
7940ad5c78 Fix case-sensitivity in validation warning (#10807) 2024-10-01 15:40:51 -05:00
Doug Beatty
3ec8fa79bd --inline-direct is an internal CLI flag (#10806) 2024-10-01 14:28:41 -06:00
FishtownBuildBot
396cf2d683 [Automated] Merged prep-release/1.9.0b1_11131260909 into target main during release process 2024-10-01 14:57:06 -04:00
Github Build Bot
87b1143a62 Bumping version to 1.9.0b1 and generate changelog 2024-10-01 18:30:08 +00:00
Kshitij Aranke
75a09621cd [tidy_first] Set default for STATE_MODIFIED_COMPARE_VARS flag, mark TestProjInfo as not a test class (#10805) 2024-10-01 17:27:44 +01:00
Kshitij Aranke
5e9f1b515f [Round 2] Fix #9005: Allow singular tests to be documented in properties.yml (#10792) 2024-10-01 08:05:36 +01:00
Quigley Malcolm
25a68a990c When retrying microbatch models, propagate prior successful state (#10802)
* When retrying microbatch models, propagate prior successful state

* Changie doc for microbatch dbt retry fixes

* Fix test_manifest unit tests for batch_info key

* Add functional test for when a microbatch model has multiple retries

* Add comment about when batch_info will be something other than None
2024-10-01 00:16:05 -05:00
Michelle Ark
a86e2b4ffc [state:modified] store unrendered_database and unrendered_schema on source definition for state:modified comparisons (#10675) 2024-09-30 17:50:33 +02:00
Michelle Ark
94917432f9 add model_incremental_strategy to track_model_run (#10758) 2024-09-30 17:35:33 +02:00
Michelle Ark
d1857b39ca state:modified vars, behind "state_modified_compare_vars" behaviour flag (#10793) 2024-09-30 16:32:37 +02:00
Kshitij Aranke
2ff3f20863 Create skip_nodes_if_on_run_start_fails behavior change flag (#10699) 2024-09-30 13:53:08 +01:00
Courtney Holcomb
5e3d418264 Add new validations for custom granularities (#10789)
* Bump DSI to latest version to ensure mantle users have new validations for custom granularities

* Changelog
2024-09-27 08:37:04 -05:00
Chenyu Li
5d32aa8b62 Revert "Fix #9005: Allow singular tests to be documented in properties.yml" (#10790)
This reverts commit 3ac20ce7a8.
2024-09-26 17:03:10 -07:00
Quigley Malcolm
d8b1bf53f7 [CT-10785] Microbatch models should respect full_refresh model config (#10788)
* Add tests to check how microbatch models respect `full_refresh` model configs

* Fix `_is_incremental` to properly respect `full_refresh` model config

In dbt-core, it is generally expected that values passed via CLI flags take
precedence over model level configs. However, `full_refresh` on a model is an
exception to this rule, where in the model config takes precedence. This
config exists specifically to _prevent_ accidental full refreshes of large
incremental models, as doing so can be costly. **_It is actually best
practice_** to set `full_refresh=False` on incremental models.

Prior to this commit, for microbatch models, the above was not happening. The
CLI flag `--full-refresh` was taking precedence over the model config
`full_refresh`. That meant that if `--full-refresh` was supplied, then the
microbatch model **_would full refresh_** even if `full_refresh=False` was
set on the model. This commit solves that problem.

* Add changie doc for microbatch `full_refresh` config handling
2024-09-26 16:43:14 -05:00
Kshitij Aranke
1076352293 [CORE-388] Add group metadata info to LogModelResult and LogTestResult (#10775) 2024-09-26 20:57:06 +01:00
Gerda Shank
1fe9c1bbfe Attempt to skip saved query processing when no semantic manifest changes (#10784) 2024-09-26 13:04:19 -04:00
Quigley Malcolm
41e4836c0f Fix changie doc for microbatch retry functionality (#10787)
The changie log was referencing the microbatch epic instead of the specific
issues it resolved
2024-09-26 11:34:41 -05:00
Michelle Ark
b590045b9f [state:modified] persist unrendered_config from schema.yml, and more reliably compute unrendered_config from .sql files (#10487) 2024-09-26 16:03:40 +01:00
Quigley Malcolm
1fd4d2eae6 Enable retry support for Microbatch models (#10751)
* Add `PartialSuccess` status type and use it for microbatch models with mixed results

* Handle `PartialSuccess` in `interpret_run_result`

* Add `BatchResults` object to `BaseResult` and begin tracking during microbatch runs

* Ensure batch_results being propagated to `run_results` artifact

* Move `batch_results` from `BaseResult` class to `RunResult` class

* Move `BatchResults` and `BatchType` to separate arifacts file to avoid circular imports

In our next commit we're gonna modify `dbt/contracts/graph/nodes.py` to import the
`BatchType` as part of our work to implement dbt retry for microbatch model nodes.
Unfortunately, the import in `nodes.py` creates a circular dependency because
`dbt/artifacts/schemas/results.py` imports from `nodes.py` and `dbt/artifacts/schemas/run/v5/run.py`
imports from that `results.py`. Thus the new import creates a circular import. Now this
_shouldn't_ be necessary as nothing in artifacts should import from the rest of dbt-core.
However, we do. We should fix this, but this is also out of scope for this segement of work.

* Add `PartialSuccess` as a retry-able status, and use batches to retry microbatch models

* Fix BatchType type so that the first datetime is no longer Optional

* Ensure `PartialSuccess` causes skipping of downstream nodes

* Alter `PartialSuccess` status to be considered an error in `interpret_run_result`

* Update schemas and test artifacts to include new batch_results run results key

* Add functional test to check that 'dbt retry' retries 'PartialSuccess' models

* Update partition failure test to assert downstream models are skipped

* Improve `success`/`error`/`partial success` messaging for microbatch models

* Include `PartialSuccess` in status that `--fail-fast` counts as a failure

* Update `LogModelResult` to handle partial successes

* Update `EndOfRunSummary` to handle partial successes

* Cleanup TODO comment

* Raise a DbtInternalError if we get a batch run result without `batch_results`

* When running a microbatch model with supplied batches, force non full-refresh behavior

This is necessary because of retry. Say on the initial run the microbatch model
succeeds on 97% of it's batches. Then on retry it does the last 3%. If the retry
of the microbatch model executes in full refresh mode it _might_ blow away the
97% of work that has been done. This edge case seems to be adapter specific.

* Only pass batches to retry for microbatch model when there was a PartialSuccess

In the previous commit we made it so that retries of microbatch models wouldn't
run in full refresh mode when the microbatch model to retry has batches already
specified from the prior run. This is only problematic when the run being retried
was a full refresh AND all the batches for a given microbatch model failed. In
that case WE DO want to do a full refresh for the given microbatch model. To better
outline the problem, consider the following:

* a microbatch model had a begin of `2020-01-01` and has been running this way for awhile
* the begin config has changed to `2024-01-01` and  dbt run --full-refresh gets run
* every batch for an microbatch model fails
* on dbt retry the the relation is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purged

To avoid this, all we have to do is ONLY pass the batch information for partially successful microbatch
models. Note: microbatch models only have a partially successful status IFF they have both
successful and failed batches.

* Fix test_manifest unit tests to know about model 'batches' key

* Add some console output assertions to microbatch functional tests

* add batch_results: None to expected_run_results

* Add changie doc for microbatch retry functionality

* maintain protoc version 5.26.1

* Cleanup extraneous comment in LogModelResult

---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2024-09-26 08:45:47 -05:00
Gerda Shank
ac66f91351 Improve performance of infer primary key (#10782) 2024-09-25 18:16:33 -04:00
Peter Webb
359a2c0cc5 Add '--inline-direct' parameter to 'dbt show'. (#10770)
* Add '--inline-direct' parameter to 'dbt show'.

* Add changelog entry.

* Update core/dbt/cli/main.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Add test of failure for --inline-direct

---------

Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2024-09-25 12:31:05 -04:00
Quigley Malcolm
bbdb98fa5d Microbatch Config Validation (#10752) 2024-09-24 19:56:34 +01:00
Katsuya Shimabukuro
a8d4ba2b4a Fix unit tests for incremental models with alias (#10755) 2024-09-24 17:57:37 +01:00
Doug Beatty
09e973d24a Test case for merge_exclude_columns (#8268)
* Test case for `merge_exclude_columns`

* Update expected output for `merge_exclude_columns`

* Skip TestMergeExcludeColumns test

* Enable this test since PostgreSQL 15+ is available in CI now

* Undo modification to expected output
2024-09-24 08:10:44 -06:00
Michelle Ark
730e40a867 Add required 'begin' config support for microbatch models (#10756) 2024-09-24 14:13:02 +01:00
Michelle Ark
a1e4753020 Write microbatch compiled + run code to separate target files (#10743) 2024-09-24 14:12:44 +01:00
Kshitij Aranke
3ac20ce7a8 Fix #9005: Allow singular tests to be documented in properties.yml (#10744) 2024-09-24 13:15:03 +01:00
Michelle Ark
aa23af98e5 ignore --empty in unit test ref/source calls (#10764) 2024-09-23 23:36:28 +01:00
Peter Webb
46da967115 Allow snapshots to be defined with YAML only. (#10762) 2024-09-23 16:33:30 -04:00
Gerda Shank
db694731c9 Allow configuration of snapshot column names (#10608) 2024-09-20 19:31:05 -04:00
Doug Beatty
7016cd3085 Standardize returning ResourceTypeSelector instances in dbt list and dbt build (#10739)
* Remove duplicated constructor for `ResourceTypeSelector`

* Add type annotation for `ResourceTypeSelector`

* Standardize on constructor for `ResourceTypeSelector` where `include_empty_nodes=True`

* Changelog entry
2024-09-19 16:53:16 -06:00
Doug Beatty
9ca10fbfd9 Enable --resource-type and --exclude-resource-type CLI flags and environment variables for dbt test (#10706)
* Adding logic to TestSelector to remove unit tests if they are in excluded_resource_types

* Adding change log

* Respect `--resource-type` and `--exclude-resource-type` CLI flags and associated environment variables

* Test CLI flag for excluding unit tests for the `dbt test` command

* Satisy isort pre-commit hook

* Fix mypy for positional argument "resource_types" in call to "TestSelector"

* Replace `TestSelector` with `ResourceTypeSelector`

* Add co-author

* Update changelog description

* Add functional tests for new feature

* Compare the actual results, not just the count

* Remove test case covered elsewhere

* Test for `DBT_EXCLUDE_RESOURCE_TYPES` environment variable for `dbt test`

* Update per pre-commit hook

* Restore to original form (until we refactor extraneous `ResourceTypeSelector` references later)

---------

Co-authored-by: Matthew Cooper <asimov.1st@gmail.com>
2024-09-19 16:52:59 -06:00
Quigley Malcolm
3308a4365e Split out model vs microbatch execution (#10737) 2024-09-18 22:13:50 -05:00
Mike Alfare
f8bfd32ed6 fix mock in unit test after removing deprecated snowplow method (#10738) 2024-09-18 15:45:11 -07:00
Mike Alfare
3e437a6734 snowplow is deprecating track_struct_event; use StructuredEvent instead (#10736) 2024-09-18 16:29:18 -04:00
Doug Beatty
9e633f6178 Replace TestSelector with ResourceTypeSelector (#10718)
* [tidy first] Replace `TestSelector` with `ResourceTypeSelector`

* Changelog entry

* Fully preserve current behavior

* Revert "Fully preserve current behavior"

This reverts commit ceecfec96da00efeaca57922b4a3ed7eec639105.
2024-09-18 14:13:50 -06:00
Mike Alfare
d182d06644 Add Snowplow tracking for behavior flags (#10721)
* add behavior deprecation snowplow callback
* update tests for new callback
* update test input with the new required field
2024-09-18 15:09:23 -04:00
Peter Webb
054c6fde37 Fix calls to pre_model_hook which were apparently passing an object of the wrong type. (#10735) 2024-09-18 14:09:02 -04:00
Ly Nguyen
4c326e40b5 Update template for docs issues (#10731)
* Update template for docs issues

* Update .github/workflows/docs-issue.yml

* Update .github/workflows/docs-issue.yml

* Update .github/workflows/docs-issue.yml
2024-09-18 13:02:26 -05:00
Michelle Ark
8fe5ea1ee7 Microbatch: batched execution (#10677)
* initial rough-in with CLI flags

* dbt-adapters testing against event-time-ref-filtering

* fix TestList

* Checkpoint

* fix tests

* add event_time_start params to build

* rename configs

* Gate resolve_event_time_filter via micro batch strategy and fix strptime usage

* Add unit test for resolve_event_time_filter

* Additional unit tests for `resolve_event_time_filter` to ensure lookback + batch_size work

* Remove extraneous comments and print statements from resolve_event_time_filter

* Fixup microbatch functional tests to use microbatch strategy

* Gate microbatch functionality behind env_var while in beta

* Add comment about how _is_incremental should be removed

* Improve `event_time_start/end` cli parameters to auto convert to datetime objects

* for testing: dbt-postgres 'microbatch' strategy

* rough in: chunked backfills

* partial failure of microbatch runs

* decouple run result methods

* initial refactor

* rename configs to __dbt_internal

* update compiled_code in context after re-compilation

* finish rename of context vars

* changelog entry

* fix patch_microbatch_end_time

* refactor into MicrobatchBuilder

* fix provider unit tests + add unit tests for MicrobatchBuilder

* add TestMicrobatchJinjaContextVarsAvailable

* unit test offset + truncate timestamp methods

* Remove pairing.md file

* Add tying to microbatch specific functions added in `task/run.py`

* Add doc strings to microbatch.py functions and classes

* Set microbatch node status to `ERROR` if all batches for node failed

* Fire an event for batch exceptions instead of directly printing

* Fix firing of failed microbatch log event

---------

Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
2024-09-18 11:46:06 -05:00
Doug Beatty
16f5023f4d Fix --resource-type test for dbt list and dbt build (#10730)
* Update functional tests to cover this case

* Revert "Update functional tests to cover this case"

This reverts commit 4c78e816f6c370b7a39df774f2fab7328db5312b.

* New functional tests to cover the resource_type config

* Separate data tests from unit tests for `resource_types` config of `dbt list` and `dbt build`

* Changelog entry
2024-09-18 10:21:14 -06:00
Courtney Holcomb
c6b8f7e595 Add custom granularities to YAML spec (#10664)
* Add custom granularities to YAML spec

* Changelog

* Add tests

* Remove unneeded duplicate classes
2024-09-17 13:02:56 -05:00
Michelle Ark
77aeb3ea68 Add functional tests for custom incremental strategies names 'microbatch' (#10716)
* Add functional tests for custom incremental strategies names 'microbatch'

* Point dev-requirement of `dbt-adapters` back to the main branch

The associated branch/PR in `dbt-adapters` that we were previously
pointing to has been merged. Thus we can point back to `main` again.

---------

Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
2024-09-16 14:38:14 -05:00
Emily Rockman
1e20772d33 Update docs-issue.yml (#10720) 2024-09-16 11:59:15 -05:00
Doug Beatty
8ce2c46a2f [tidy first] Fix a couple misspellings (#10717) 2024-09-16 10:02:30 -06:00
Gerda Shank
aeaaedcaa1 Update docs and title for schemas.getdbt.com changes (#10719) 2024-09-16 11:11:08 -04:00
Michelle Ark
6c111f2e31 add patch_microbatch_event_time (#10712) 2024-09-13 22:01:31 -04:00
Michelle Ark
139b9ac54f delete extraneous EventCatcher (#10710) 2024-09-13 18:19:33 -04:00
Michelle Ark
cc8541c05f Microbatch: event_time ref + source filtering (#10594) 2024-09-12 18:16:04 -04:00
Michelle Ark
ab500a9709 install postgres-16 in CI (#10687) 2024-09-11 12:43:33 -04:00
240 changed files with 7198 additions and 994 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.9.0a1
current_version = 1.9.0b2
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

136
.changes/1.9.0-b1.md Normal file
View File

@@ -0,0 +1,136 @@
## dbt-core 1.9.0-b1 - October 01, 2024
### Breaking Changes
- Fix changing the current working directory when using dpt deps, clean and init. ([#8997](https://github.com/dbt-labs/dbt-core/issues/8997))
### Features
- serialize inferred primary key ([#9824](https://github.com/dbt-labs/dbt-core/issues/9824))
- Add unit_test: selection method ([#10053](https://github.com/dbt-labs/dbt-core/issues/10053))
- Maximally parallelize dbt clone in clone command" ([#7914](https://github.com/dbt-labs/dbt-core/issues/7914))
- Add --host flag to dbt docs serve, defaulting to '127.0.0.1' ([#10229](https://github.com/dbt-labs/dbt-core/issues/10229))
- Update data_test to accept arbitrary config options ([#10197](https://github.com/dbt-labs/dbt-core/issues/10197))
- add pre_model and post_model hook calls to data and unit tests to be able to provide extra config options ([#10198](https://github.com/dbt-labs/dbt-core/issues/10198))
- add --empty value to jinja context as flags.EMPTY ([#10317](https://github.com/dbt-labs/dbt-core/issues/10317))
- Warning message for snapshot timestamp data types ([#10234](https://github.com/dbt-labs/dbt-core/issues/10234))
- Support cumulative_type_params & sub-daily granularities in semantic manifest. ([#10360](https://github.com/dbt-labs/dbt-core/issues/10360))
- Add time_granularity to metric spec. ([#10376](https://github.com/dbt-labs/dbt-core/issues/10376))
- Support standard schema/database fields for snapshots ([#10301](https://github.com/dbt-labs/dbt-core/issues/10301))
- Support ref and source in foreign key constraint expressions, bump dbt-common minimum to 1.6 ([#8062](https://github.com/dbt-labs/dbt-core/issues/8062))
- Support new semantic layer time spine configs to enable sub-daily granularity. ([#10475](https://github.com/dbt-labs/dbt-core/issues/10475))
- Include models that depend on changed vars in state:modified, add state:modified.vars selection method ([#4304](https://github.com/dbt-labs/dbt-core/issues/4304))
- Add support for behavior flags ([#10618](https://github.com/dbt-labs/dbt-core/issues/10618))
- Enable `--resource-type` and `--exclude-resource-type` CLI flags and environment variables for `dbt test` ([#10656](https://github.com/dbt-labs/dbt-core/issues/10656))
- Allow configuring snapshot column names ([#10185](https://github.com/dbt-labs/dbt-core/issues/10185))
- Add custom_granularities to YAML spec for time spines. ([#9265](https://github.com/dbt-labs/dbt-core/issues/9265))
- Add basic functionality for creating microbatch incremental models ([#9490](https://github.com/dbt-labs/dbt-core/issues/9490), [#10635](https://github.com/dbt-labs/dbt-core/issues/10635), [#10637](https://github.com/dbt-labs/dbt-core/issues/10637), [#10638](https://github.com/dbt-labs/dbt-core/issues/10638), [#10636](https://github.com/dbt-labs/dbt-core/issues/10636), [#10662](https://github.com/dbt-labs/dbt-core/issues/10662), [#10639](https://github.com/dbt-labs/dbt-core/issues/10639))
- Execute microbatch models in batches ([#10700](https://github.com/dbt-labs/dbt-core/issues/10700))
- Create 'skip_nodes_if_on_run_start_fails' behavior change flag ([#7387](https://github.com/dbt-labs/dbt-core/issues/7387))
- Allow snapshots to be defined in YAML. ([#10246](https://github.com/dbt-labs/dbt-core/issues/10246))
- Write microbatch compiled/run targets to separate files, one per batch ([#10714](https://github.com/dbt-labs/dbt-core/issues/10714))
- Track incremental_strategy as part of model_run tracking event ([#10761](https://github.com/dbt-labs/dbt-core/issues/10761))
- Support required 'begin' config for microbatch models ([#10701](https://github.com/dbt-labs/dbt-core/issues/10701))
- Parse-time validation of microbatch configs: require event_time, batch_size, lookback and validate input event_time ([#10709](https://github.com/dbt-labs/dbt-core/issues/10709))
- Added the --inline-direct parameter to 'dbt show' ([#10770](https://github.com/dbt-labs/dbt-core/issues/10770))
- Enable `retry` support for microbatch models ([#10715](https://github.com/dbt-labs/dbt-core/issues/10715), [#10729](https://github.com/dbt-labs/dbt-core/issues/10729))
- Use unrendered database and schema source properties during state:modified, behind state_modified_compare_more_unrendered_values behavoiur flag ([#9573](https://github.com/dbt-labs/dbt-core/issues/9573))
- Ensure microbatch models respect `full_refresh` model config ([#10785](https://github.com/dbt-labs/dbt-core/issues/10785))
- Adds validations for custom_granularities to ensure unique naming. ([#9265](https://github.com/dbt-labs/dbt-core/issues/9265))
### Fixes
- Remove unused check_new method ([#7586](https://github.com/dbt-labs/dbt-core/issues/7586))
- Test case for `merge_exclude_columns` ([#8267](https://github.com/dbt-labs/dbt-core/issues/8267))
- Convert "Skipping model due to fail_fast" message to DEBUG level ([#8774](https://github.com/dbt-labs/dbt-core/issues/8774))
- Restore previous behavior for --favor-state: only favor defer_relation if not selected in current command" ([#10107](https://github.com/dbt-labs/dbt-core/issues/10107))
- Unit test fixture (csv) returns null for empty value ([#9881](https://github.com/dbt-labs/dbt-core/issues/9881))
- Fix json format log and --quiet for ls and jinja print by converting print call to fire events ([#8756](https://github.com/dbt-labs/dbt-core/issues/8756))
- Add resource type to saved_query ([#10168](https://github.com/dbt-labs/dbt-core/issues/10168))
- Fix: Order-insensitive unit test equality assertion for expected/actual with multiple nulls ([#10167](https://github.com/dbt-labs/dbt-core/issues/10167))
- Renaming or removing a contracted model should raise a BreakingChange warning/error ([#10116](https://github.com/dbt-labs/dbt-core/issues/10116))
- prefer disabled project nodes to external node ([#10224](https://github.com/dbt-labs/dbt-core/issues/10224))
- Fix issues with selectors and inline nodes ([#8943](https://github.com/dbt-labs/dbt-core/issues/8943), [#9269](https://github.com/dbt-labs/dbt-core/issues/9269))
- Fix snapshot config to work in yaml files ([#4000](https://github.com/dbt-labs/dbt-core/issues/4000))
- Improve handling of error when loading schema file list ([#10284](https://github.com/dbt-labs/dbt-core/issues/10284))
- Use model alias for the CTE identifier generated during ephemeral materialization ([#5273](https://github.com/dbt-labs/dbt-core/issues/5273))
- Implement state:modified for saved queries ([#10294](https://github.com/dbt-labs/dbt-core/issues/10294))
- Saved Query node fail during skip ([#10029](https://github.com/dbt-labs/dbt-core/issues/10029))
- DOn't warn on `unit_test` config paths that are properly used ([#10311](https://github.com/dbt-labs/dbt-core/issues/10311))
- Fix setting `silence` of `warn_error_options` via `dbt_project.yaml` flags ([#10160](https://github.com/dbt-labs/dbt-core/issues/10160))
- Attempt to provide test fixture tables with all values to set types correctly for comparisong with source tables ([#10365](https://github.com/dbt-labs/dbt-core/issues/10365))
- Limit data_tests deprecation to root_project ([#9835](https://github.com/dbt-labs/dbt-core/issues/9835))
- CLI flags should take precedence over env var flags ([#10304](https://github.com/dbt-labs/dbt-core/issues/10304))
- Fix typing for artifact schemas ([#10442](https://github.com/dbt-labs/dbt-core/issues/10442))
- Fix over deletion of generated_metrics in partial parsing ([#10450](https://github.com/dbt-labs/dbt-core/issues/10450))
- Fix error constructing warn_error_options ([#10452](https://github.com/dbt-labs/dbt-core/issues/10452))
- Do not update varchar column definitions if a contract exists ([#10362](https://github.com/dbt-labs/dbt-core/issues/10362))
- fix all_constraints access, disabled node parsing of non-uniquely named resources ([#10509](https://github.com/dbt-labs/dbt-core/issues/10509))
- respect --quiet and --warn-error-options for flag deprecations ([#10105](https://github.com/dbt-labs/dbt-core/issues/10105))
- Propagate measure label when using create_metrics ([#10536](https://github.com/dbt-labs/dbt-core/issues/10536))
- Fix state:modified check for exports ([#10138](https://github.com/dbt-labs/dbt-core/issues/10138))
- Filter out empty nodes after graph selection to support consistent selection of nodes that depend on upstream public models ([#8987](https://github.com/dbt-labs/dbt-core/issues/8987))
- Late render pre- and post-hooks configs in properties / schema YAML files ([#10603](https://github.com/dbt-labs/dbt-core/issues/10603))
- Allow the use of env_var function in certain macros in which it was previously unavailable. ([#10609](https://github.com/dbt-labs/dbt-core/issues/10609))
- Remove deprecation for tests: to data_tests: change ([#10564](https://github.com/dbt-labs/dbt-core/issues/10564))
- Fix `--resource-type test` for `dbt list` and `dbt build` ([#10730](https://github.com/dbt-labs/dbt-core/issues/10730))
- Fix unit tests for incremental model with alias ([#10754](https://github.com/dbt-labs/dbt-core/issues/10754))
- Allow singular tests to be documented in properties.yml ([#9005](https://github.com/dbt-labs/dbt-core/issues/9005))
- Ignore --empty in unit test ref/source rendering ([#10516](https://github.com/dbt-labs/dbt-core/issues/10516))
- Ignore rendered jinja in configs for state:modified, behind state_modified_compare_more_unrendered_values behaviour flag ([#9564](https://github.com/dbt-labs/dbt-core/issues/9564))
- Improve performance of infer primary key ([#10781](https://github.com/dbt-labs/dbt-core/issues/10781))
- Attempt to skip saved query processing when no semantic manifest changes ([#10563](https://github.com/dbt-labs/dbt-core/issues/10563))
- Ensure dbt retry of microbatch models doesn't lose prior successful state ([#10800](https://github.com/dbt-labs/dbt-core/issues/10800))
### Docs
- Enable display of unit tests ([dbt-docs/#501](https://github.com/dbt-labs/dbt-docs/issues/501))
- Unit tests not rendering ([dbt-docs/#506](https://github.com/dbt-labs/dbt-docs/issues/506))
- Add support for Saved Query node ([dbt-docs/#486](https://github.com/dbt-labs/dbt-docs/issues/486))
- Fix npm security vulnerabilities as of June 2024 ([dbt-docs/#513](https://github.com/dbt-labs/dbt-docs/issues/513))
### Under the Hood
- Clear error message for Private package in dbt-core ([#10083](https://github.com/dbt-labs/dbt-core/issues/10083))
- Enable use of context in serialization ([#10093](https://github.com/dbt-labs/dbt-core/issues/10093))
- Make RSS high water mark measurement more accurate on Linux ([#10177](https://github.com/dbt-labs/dbt-core/issues/10177))
- Enable record filtering by type. ([#10240](https://github.com/dbt-labs/dbt-core/issues/10240))
- Remove IntermediateSnapshotNode ([#10326](https://github.com/dbt-labs/dbt-core/issues/10326))
- Additional logging for skipped ephemeral models ([#10389](https://github.com/dbt-labs/dbt-core/issues/10389))
- bump black to 24.3.0 ([#10454](https://github.com/dbt-labs/dbt-core/issues/10454))
- generate protos with protoc version 5.26.1 ([#10457](https://github.com/dbt-labs/dbt-core/issues/10457))
- Move from minimal-snowplow-tracker fork back to snowplow-tracker ([#8409](https://github.com/dbt-labs/dbt-core/issues/8409))
- Add group info to RunResultError, RunResultFailure, RunResultWarning log lines ([#](https://github.com/dbt-labs/dbt-core/issues/))
- Improve speed of tree traversal when finding children, increasing build speed for some selectors ([#10434](https://github.com/dbt-labs/dbt-core/issues/10434))
- Add test for sources tables with quotes ([#10582](https://github.com/dbt-labs/dbt-core/issues/10582))
- Additional type hints for `core/dbt/version.py` ([#10612](https://github.com/dbt-labs/dbt-core/issues/10612))
- Fix typing issues in core/dbt/contracts/sql.py ([#10614](https://github.com/dbt-labs/dbt-core/issues/10614))
- Fix type errors in `dbt/core/task/clean.py` ([#10616](https://github.com/dbt-labs/dbt-core/issues/10616))
- Add Snowplow tracking for behavior flag deprecations ([#10552](https://github.com/dbt-labs/dbt-core/issues/10552))
- Add test utility patch_microbatch_end_time for adapters testing ([#10713](https://github.com/dbt-labs/dbt-core/issues/10713))
- Replace `TestSelector` with `ResourceTypeSelector` ([#10718](https://github.com/dbt-labs/dbt-core/issues/10718))
- Standardize returning `ResourceTypeSelector` instances in `dbt list` and `dbt build` ([#10739](https://github.com/dbt-labs/dbt-core/issues/10739))
- Add group metadata info to LogModelResult and LogTestResult ([#10775](https://github.com/dbt-labs/dbt-core/issues/10775))
### Dependencies
- Remove logbook dependency ([#8027](https://github.com/dbt-labs/dbt-core/issues/8027))
- Increase supported version range for dbt-semantic-interfaces. Needed to support custom calendar features. ([#9265](https://github.com/dbt-labs/dbt-core/issues/9265))
### Security
- Explicitly bind to localhost in docs serve ([#10209](https://github.com/dbt-labs/dbt-core/issues/10209))
### Contributors
- [@McKnight-42](https://github.com/McKnight-42) ([#10197](https://github.com/dbt-labs/dbt-core/issues/10197), [#10198](https://github.com/dbt-labs/dbt-core/issues/10198))
- [@TowardOliver](https://github.com/TowardOliver) ([#10656](https://github.com/dbt-labs/dbt-core/issues/10656))
- [@aliceliu](https://github.com/aliceliu) ([#10536](https://github.com/dbt-labs/dbt-core/issues/10536), [#10138](https://github.com/dbt-labs/dbt-core/issues/10138))
- [@courtneyholcomb](https://github.com/courtneyholcomb) ([#10360](https://github.com/dbt-labs/dbt-core/issues/10360), [#10376](https://github.com/dbt-labs/dbt-core/issues/10376), [#10475](https://github.com/dbt-labs/dbt-core/issues/10475), [#9265](https://github.com/dbt-labs/dbt-core/issues/9265), [#9265](https://github.com/dbt-labs/dbt-core/issues/9265), [#9265](https://github.com/dbt-labs/dbt-core/issues/9265))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#9824](https://github.com/dbt-labs/dbt-core/issues/9824))
- [@jeancochrane](https://github.com/jeancochrane) ([#5273](https://github.com/dbt-labs/dbt-core/issues/5273))
- [@katsugeneration](https://github.com/katsugeneration) ([#10754](https://github.com/dbt-labs/dbt-core/issues/10754))
- [@kevinneville](https://github.com/kevinneville) ([#7586](https://github.com/dbt-labs/dbt-core/issues/7586))
- [@nakamichiworks](https://github.com/nakamichiworks) ([#10442](https://github.com/dbt-labs/dbt-core/issues/10442))
- [@rariyama](https://github.com/rariyama) ([#8997](https://github.com/dbt-labs/dbt-core/issues/8997))
- [@scottgigante,nevdelap](https://github.com/scottgigante,nevdelap) ([#8774](https://github.com/dbt-labs/dbt-core/issues/8774))
- [@ttusing](https://github.com/ttusing) ([#10434](https://github.com/dbt-labs/dbt-core/issues/10434))

1
.changes/1.9.0-b2.md Normal file
View File

@@ -0,0 +1 @@
## dbt-core 1.9.0-b2 - October 07, 2024

View File

@@ -0,0 +1,6 @@
kind: Features
body: Enable `--resource-type` and `--exclude-resource-type` CLI flags and environment variables for `dbt test`
time: 2024-09-03T13:24:28.592837+01:00
custom:
Author: TowardOliver dbeatty10
Issue: "10656"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Allow configuring snapshot column names
time: 2024-09-03T15:41:33.167097-04:00
custom:
Author: gshank
Issue: "10185"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add custom_granularities to YAML spec for time spines.
time: 2024-09-04T18:23:20.234952-07:00
custom:
Author: courtneyholcomb
Issue: "9265"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add basic functionality for creating microbatch incremental models
time: 2024-09-11T12:10:29.822189-05:00
custom:
Author: MichelleArk QMalcolm
Issue: 9490 10635 10637 10638 10636 10662 10639

View File

@@ -0,0 +1,6 @@
kind: Features
body: Execute microbatch models in batches
time: 2024-09-13T23:21:11.935434-04:00
custom:
Author: michelleark
Issue: "10700"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Create 'skip_nodes_if_on_run_start_fails' behavior change flag
time: 2024-09-18T16:29:59.268422+01:00
custom:
Author: aranke
Issue: "7387"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Allow snapshots to be defined in YAML.
time: 2024-09-20T11:04:47.703117-04:00
custom:
Author: peterallenwebb
Issue: "10246"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Write microbatch compiled/run targets to separate files, one per batch
time: 2024-09-20T17:24:19.219556+01:00
custom:
Author: michelleark
Issue: "10714"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Track incremental_strategy as part of model_run tracking event
time: 2024-09-23T15:53:35.898928+01:00
custom:
Author: michelleark
Issue: "10761"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support required 'begin' config for microbatch models
time: 2024-09-23T15:59:03.924079+01:00
custom:
Author: michelleark
Issue: "10701"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Added the --inline-direct parameter to 'dbt show'
time: 2024-09-24T15:29:22.874496-04:00
custom:
Author: aranke peterallenwebb
Issue: "10770"

View File

@@ -0,0 +1,7 @@
kind: Features
body: 'Parse-time validation of microbatch configs: require event_time, batch_size,
lookback and validate input event_time'
time: 2024-09-24T15:46:39.83112+01:00
custom:
Author: michelleark
Issue: "10709"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Enable `retry` support for microbatch models
time: 2024-09-25T16:50:02.105069-05:00
custom:
Author: QMalcolm MichelleArk
Issue: 10715 10729

View File

@@ -0,0 +1,7 @@
kind: Features
body: Use unrendered database and schema source properties during state:modified,
behind state_modified_compare_more_unrendered_values behavoiur flag
time: 2024-09-26T14:02:06.732068+01:00
custom:
Author: michelleark
Issue: "9573"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Adds validations for custom_granularities to ensure unique naming.
time: 2024-09-26T15:10:57.907694-07:00
custom:
Author: courtneyholcomb
Issue: "9265"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Ensure microbatch models respect `full_refresh` model config
time: 2024-09-26T15:32:10.202789-05:00
custom:
Author: QMalcolm
Issue: "10785"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Test case for `merge_exclude_columns`
time: 2023-08-01T09:46:26.829362-06:00
custom:
Author: dbeatty10
Issue: "8267"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix `--resource-type test` for `dbt list` and `dbt build`
time: 2024-09-17T17:44:46.121032-06:00
custom:
Author: dbeatty10
Issue: "10730"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix unit tests for incremental model with alias
time: 2024-09-22T13:35:27.991398741Z
custom:
Author: katsugeneration
Issue: "10754"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Allow singular tests to be documented in properties.yml
time: 2024-09-23T19:07:58.151069+01:00
custom:
Author: aranke
Issue: "9005"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Ignore --empty in unit test ref/source rendering
time: 2024-09-23T20:20:24.151285+01:00
custom:
Author: michelleark
Issue: "10516"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Ignore rendered jinja in configs for state:modified, behind state_modified_compare_more_unrendered_values
behaviour flag
time: 2024-09-25T13:10:28.490042+01:00
custom:
Author: michelleark
Issue: "9564"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Improve performance of infer primary key
time: 2024-09-25T16:05:43.59536-04:00
custom:
Author: gshank
Issue: "10781"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Attempt to skip saved query processing when no semantic manifest changes
time: 2024-09-26T10:12:20.193453-04:00
custom:
Author: gshank
Issue: "10563"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Ensure dbt retry of microbatch models doesn't lose prior successful state
time: 2024-09-30T15:31:58.541656-05:00
custom:
Author: QMalcolm
Issue: "10800"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add Snowplow tracking for behavior flag deprecations
time: 2024-09-11T16:27:30.293832-04:00
custom:
Author: mikealfare
Issue: "10552"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add test utility patch_microbatch_end_time for adapters testing
time: 2024-09-13T21:33:12.482336-04:00
custom:
Author: michelleark
Issue: "10713"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Replace `TestSelector` with `ResourceTypeSelector`
time: 2024-09-16T10:22:01.339462-06:00
custom:
Author: dbeatty10
Issue: "10718"

Some files were not shown because too many files have changed in this diff Show More