* Add `batch_id` to jinja context of microbatch batches
* Add changie doc
* Update `format_batch_start` to assume `batch_start` is always provided
* Add "runtime only" property `batch_context` to `ModelNode`
By it being "runtime only" we mean that it doesn't exist on the artifact
and thus won't be written out to the manifest artifact.
* Begin populating `batch_context` during materialization execution for microbatch batches
* Fix circular import
* Fixup MicrobatchBuilder.batch_id property method
* Ensure MicrobatchModelRunner doesn't double compile batches
We were compiling the node for each batch _twice_. Besides making microbatch
models more expensive than they needed to be, double compiling wasn't
causing any issue. However the first compilation was happening _before_ we
had added the batch context information to the model node for the batch. This
was leading to models which try to access the `batch_context` information on the
model to blow up, which was undesirable. As such, we've now gone and skipped
the first compilation. We've done this similar to how SavedQuery nodes skip
compilation.
* Add `__post_serialize__` method to `BatchContext` to ensure correct dict shape
This is weird, but necessary, I apologize. Mashumaro handles the
dictification of this class via a compile time generated `to_dict`
method based off of the _typing_ of th class. By default `datetime`
types are converted to strings. We don't want that, we want them to
stay datetimes.
* Update tests to check for `batch_context`
* Update `resolve_event_time_filter` to use new `batch_context`
* Stop testing for batchless compiled code for microbatch models
In 45daec72f4 we stopped an extra compilation
that was happening per batch prior to the batch_context being loaded. Stopping
this extra compilation means that compiled sql for the microbatch model without
the event time filter / batch context is no longer produced. We have discussed
this and _believe_ it is okay given that this is a new node type that has not
hit GA yet.
* Rename `ModelNode.batch_context` to `ModelNode.batch`
* Rename `build_batch_context` to `build_jinja_context_for_batch`
The name `build_batch_context` was confusing as
1) We have a `BatchContext` object, which the method was not building
2) The method builds the jinja context for the batch
As such it felt appropriate to rename the method to more accurately
communicate what it does.
* Rename test macro `invalid_batch_context_macro_sql` to `invalid_batch_jinja_context_macro_sql`
This rename was to make it more clear that the jinja context for a
batch was being checked, as a batch_context has a slightly different
connotation.
* Update changie doc
* Rename `batch_info` to `previous_batch_results`
* Exclude `previous_batch_results` from serialization of model node to avoid jinja context bloat
* Drop `previous_batch_results` key from `test_manifest.py` unit tests
In 4050e377ec01c2f14dd9600fe704ddb34adb66fa we began excluding
`previous_batch_results` from the serialized representation of the
ModelNode. As such, we no longer need to check for it in `test_manifest.py`.
* Clean up changelog on main
* Bumping version to 1.10.0a1
* Code quality cleanup
* add 1.8,1.9 link
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Allow `dbt show` and `dbt compile` to output JSON without extra logs
* Add `quiet` attribute for ShowNode and CompiledNode messages
* Output of protoc compiler
* Utilize the `quiet` attribute for ShowNode and CompiledNode
* Reuse the `dbt list` approach when the `--quiet` flag is used
* Use PrintEvent to get to stdout even if the logger is set to ERROR
* Functional tests for quiet compile
* Functional tests for quiet show
* Fire event same way regardless if LOG_FORMAT is json or not
* Switch back to firing ShowNode and CompiledNode events
* Make `--inline-direct` to be quiet-compatible
* Temporarily change to dev branch for dbt-common
* Remove extraneous newline
* Functional test for `--quiet` for `--inline-direct` flag
* Update changelog entry
* Update `core_types_pb2.py`
* Restore the original branch in `dev-requirements.txt`
---------
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
This is needed for dbt-core + dbt-adapters to work properly in regards to
the microbatch project_flag/behavior flag `require_batched_execution_for_custom_microbatch_strategy`
* first pass: replace os env with project flag
* Fix `TestMicrobatchMultipleRetries` to not use `os.env`
* Turn off microbatch project flag for `TestMicrobatchCustomUserStrategyDefault` as it was prior to a9df50f
* Update `BaseMicrobatchTest` to turn on microbatch via project flags
* Add changie doc
* Fix functional tests after merging in main
* Add function to that determines whether the new microbatch functionality should be used
The new microbatch functionality is, unfortunately, potentially dangerous. That is
it adds a new materalization strategy `microbatch` which an end user could have
defined as a custom strategy previously. Additionally we added config keys to nodes,
and as `config` is just a Dict[str, Any], it could contain anything, thus meaning
people could already be using the configs we're adding for different purposes. Thus
we need some intellegent gating. Specifically something that adheres to the following:
cms = Custom Microbatch Strategy
abms = Adapter Builtin Microbatch Strategy
bf = Behavior flag
umb = Use Microbatch Batching
t/f/e = True/False/Error
| cms | abms | bf | umb |
| t | t | t | t |
| f | t | t | t |
| t | f | t | t |
| f | f | t | e |
| t | t | f | f |
| f | t | f | t |
| t | f | f | f |
| f | f | f | e |
(The above table assumes that there is a microbatch model present in the project)
In order to achieve this we need to check that either the microbatch behavior
flag is set to true OR microbatch materializaion being used is the _root_ microbatch
materialization (i.e. not custom). The function we added in this commit,
`use_microbatch_batches`, does just that.
* Gate microbatch functionality by `use_microbatch_batches` manifest function
* Rename microbatch behavior flag to `require_batched_execution_for_custom_microbatch_strategy`
* Extract logic of `find_macro_by_name` to `find_macro_candiate_by_name`
In 0349968c615444de05360509ddeaf6d75d41d826 I had done this for the function
`find_materialization_macro_by_name`, but that wasn't the right function to
do it to, and will be reverted shortly. `find_materialization_macro_by_name`
is used for finding the general materialization macro, whereas `find_macro_by_name`
is more general. For the work we're doing, we need to find the microbatch
macro, which is not a materialization macro.
* Use `find_macro_candidate_by_name` to find the microbatch macro
* Fix microbatch macro locality check to search for `core` locality instead of `root`
Previously were were checking for a locality of `root`. However, a locality
of `root` means it was provided by a `package`. We wnt to check for locality
of `core` which basically means `builtin via dbt-core/adapters`. There is
another locality `imported` which I beleive means it comes from another
package.
* Move the evaluation of `use_microbatch_batches` to the last position in boolean checks
The method `use_microbatch_batches` is always invoked to evaluate an `if`
statement. In most instances, it is part of a logic chain (i.e. there are
multiple things being evaluated in the `if` statement). In `if` statements
where there are multiple things being evaulated, `use_microbatch_batches`
should come _last_ (or as late as possible). This is because it is likely
the most costly thing to evaluate in the logic chain, and thus any shortcuts
cuts via other evaluations in the if statement failing (and thus avoiding
invoking `use_microbatch_batches) is desirable.
* Drop behavior flag setting for BaseMicrobatchTest tests
* Rename 'env_var' to 'project_flag' in test_microbatch.py
* Update microbatch tests to assert when we are/aren't running with batches
* Update `test_resolve_event_time_filter` to use `use_microbatch_batches`
* Fire deprecation warning for custom microbatch macros
* Add microbatch deprecation events to test_events.py
---------
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* Add new `ArtifactWritten` event
* Emit ArtifactWritten event whenever an artifact is written
* Get artifact_type from class name for ArtifactWritten event
* Add changie docs
* Add test to check that ArtifactWritten events are being emitted
* Regenerate core_types_pb2.py using correct protobuf version
* Regen core_types_pb2 again, using a more correct protoc version
* Add unit tests to check how `safe_run_hooks` handles exceptions
* Improve exception handling in `get_execution_status`
Previously in `get_execution_status` if a non `DbtRuntimeError` exception was
raised, the finally would be entered, but the `status`/`message` would not be
set, and thus a `status not defined` exception would get raised on attempting
to return. Tangentially, there is another issue where somehow the `node_status`
is becoming `None`. In all my playing with `get_execution_status` I found that
trying to return an undefined variable in the `finally` caused an undefined
variable exception. However, if in some python version, it instead just handed
back `None`, then this fix should also solve that.
* Add changie docs
* Ensure run_results get written if KeyboardInterrupt happens during end run hooks
* Bump minimum dbt-adpaters to 1.8.0
In https://github.com/dbt-labs/dbt-core/pull/10859 we started using the
`get_adapter_run_info` method provided by `dbt-adapters`. However that
function is only available in dbt-adapters >= 1.8.0. Thus 1.8.0 is our
new minimum for dbt-adapters.
* Add changie doc
* Add function to MicrobatchBuilder to get ceiling of timestamp by batch_size
* Update `MicrobatchBuilder.build_end_time` to use `ceiling_timestamp`
* fix TestMicrobatchBuilder.test_build_end_time by specifying a BatchSize + asserting actual is a ceiling timestamp
* Add changie
---------
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
* Stop validating that `--event-time-start` is before "current" time
In the next commit we'll be adding a validation that requires that `--event-time-start`
and `--event-time-end` are mutually required. That is, whenever one is specified,
the other is required. In that world, `--event-time-start` will never need to be compared
against the "current" time, because it'll never be run in conjunction with the "current"
time.
* Validate that `--event-time-start` and `--event-time-end` are mutually present
* Add changie doc for validation changes
* Alter functional microbatch tests to work with updated `event_time_start/end` reqs
We made it such that when `event_time_start` is specified, `event_time_end` must also
be specified (and vice versa). This broke numerous tests, in a few different ways:
1. There were tests that used `--event-time-start` without `--event-time-end` butg
were using event_time_start essentially as the `begin` time for models being initially
built or full refreshed. These tests could simply drop the `--event-time-start` and
instead rely on the `begin` value.
2. There was a test that was trying to load a subset of the data _excluding_ some
data which would be captured by using `begin`. In this test we added an appropriate
`--event-time-end` as the `--event-time-start` was necessary to statisfy what the
test was testing
3. There was a test which was trying to ensure that two microbatch models would be
given the same "current" time. Because we wanted to ensure the "current" time code
path was used, we couldn't add `--event-time-end` to resolve the problem, thus we
needed to remove the `--event-time-start` that was being used. However, this led to
the test being incredibly slow. This was resolved by switching the relevant microbatch
models from having `batch_size`s of `day` to instead have `year`. This solution should
be good enough for roughly ~40 years? We'll figure out a better solution then, so see ya
in 2064. Assuming I haven't died before my 70th birthday, feel free to ping me to get
this taken care of.
---------
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
* Add adapter telemetry to snowplow event.
* Temporary dev branch switch.
* Set tracking for overrideable adapter method.
* Do safer adapter ref.
* Improve comment.
* Code review comments.
* Don't call the asdict on a dict.
* Bump ci to pull in fix from base adapter.
* Add unit tests for coverage.
* Update field name from base adapter/schema change.
* remove breakpoint.
* Change `lookback` default from `0` to `1`
* Regen jsonschema manifest v12 to include `lookback` default change
* Regen saved state of v12 manifest for functional artifact testing
* Add changie doc for lookback default change
* Avoid a KeyError if `child_unique_id` is not found in the dictionary
* Changelog entry
* Functional test when an exposure references a deprecated model
dbt-adapters updated the incremental_strategy validation of incremental models such that
the validation now _always_ happens when an incremental model is executed. A test in dbt-core
`TestMicrobatchCustomUserStrategyEnvVarTrueInvalid` was previously set to _expect_ buggy behavior
where an incremental model would succeed on it's "first"/"refresh" run even if it had an invalid
incremental strategy. Thus we needed to update this test in dbt-core to expect the now correct
behavior of incremental model execution time validation
* [Tidy-First]: Fix `timings` object for hooks and macros, and make types of timings explicit
* cast literal to str
* change test
* change jsonschema to enum
* Discard changes to schemas/dbt/manifest/v12.json
* nits
---------
Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
* Add `order_by` and `limit` fields to saved queries.
* Update JSON schema
* Add change log for #10531.
* Check order by / limit in saved-query parsing test.
* Add test that checks microbatch models are all operating with the same `current_time`
* Set an `invocated_at` on the `RuntimeConfig` and plumb to `MicrobatchBuilder`
* Add changie doc
* Rename `invocated_at` to `invoked_at`
* Simply conditional logic for setting MicrobatchBuilder.batch_current_time
* Rename `batch_current_time` to `default_end_time` for MicrobatchBuilder
* Begin testing that microbatch execution times are being tracked and set
* Begin tracking the execution time of batches for microbatch models
* Add changie doc
* Additional assertions in microbatch testing
* Validate that `event_time_start` is before `event_time_end` when passed from CLI
Sometimes CLI options have restrictions based on other CLI options. This is the case
for `--event-time-start` and `--event-time-end`. Unfortunately, click doesn't provide
a good way for validating this, at least not that I found. Additionaly I'm not sure
if we have had anything like this previously. In any case, I couldn't find a
centralized validation area for such occurances. Thus I've gone and added one,
`validate_option_interactions`. Long term if more validations are added, we should
add this wrapper to each CLI command. For now I've only added it to the commands that
support `event_time_start` and `event_time_end`, specifically `build` and `run`.
* Add changie doc
* If `--event-time-end` is not specififed, ensure `--event-time-start` is less than the current time
* Fixup error message about event_time_start and event_time_end
* Move logic to validate `event_time` cli flags to `flags.py`
* Update validation of `--event-time-start` against `datetime.now` to use UTC
* When retrying microbatch models, propagate prior successful state
* Changie doc for microbatch dbt retry fixes
* Fix test_manifest unit tests for batch_info key
* Add functional test for when a microbatch model has multiple retries
* Add comment about when batch_info will be something other than None
* Add tests to check how microbatch models respect `full_refresh` model configs
* Fix `_is_incremental` to properly respect `full_refresh` model config
In dbt-core, it is generally expected that values passed via CLI flags take
precedence over model level configs. However, `full_refresh` on a model is an
exception to this rule, where in the model config takes precedence. This
config exists specifically to _prevent_ accidental full refreshes of large
incremental models, as doing so can be costly. **_It is actually best
practice_** to set `full_refresh=False` on incremental models.
Prior to this commit, for microbatch models, the above was not happening. The
CLI flag `--full-refresh` was taking precedence over the model config
`full_refresh`. That meant that if `--full-refresh` was supplied, then the
microbatch model **_would full refresh_** even if `full_refresh=False` was
set on the model. This commit solves that problem.
* Add changie doc for microbatch `full_refresh` config handling
* Add `PartialSuccess` status type and use it for microbatch models with mixed results
* Handle `PartialSuccess` in `interpret_run_result`
* Add `BatchResults` object to `BaseResult` and begin tracking during microbatch runs
* Ensure batch_results being propagated to `run_results` artifact
* Move `batch_results` from `BaseResult` class to `RunResult` class
* Move `BatchResults` and `BatchType` to separate arifacts file to avoid circular imports
In our next commit we're gonna modify `dbt/contracts/graph/nodes.py` to import the
`BatchType` as part of our work to implement dbt retry for microbatch model nodes.
Unfortunately, the import in `nodes.py` creates a circular dependency because
`dbt/artifacts/schemas/results.py` imports from `nodes.py` and `dbt/artifacts/schemas/run/v5/run.py`
imports from that `results.py`. Thus the new import creates a circular import. Now this
_shouldn't_ be necessary as nothing in artifacts should import from the rest of dbt-core.
However, we do. We should fix this, but this is also out of scope for this segement of work.
* Add `PartialSuccess` as a retry-able status, and use batches to retry microbatch models
* Fix BatchType type so that the first datetime is no longer Optional
* Ensure `PartialSuccess` causes skipping of downstream nodes
* Alter `PartialSuccess` status to be considered an error in `interpret_run_result`
* Update schemas and test artifacts to include new batch_results run results key
* Add functional test to check that 'dbt retry' retries 'PartialSuccess' models
* Update partition failure test to assert downstream models are skipped
* Improve `success`/`error`/`partial success` messaging for microbatch models
* Include `PartialSuccess` in status that `--fail-fast` counts as a failure
* Update `LogModelResult` to handle partial successes
* Update `EndOfRunSummary` to handle partial successes
* Cleanup TODO comment
* Raise a DbtInternalError if we get a batch run result without `batch_results`
* When running a microbatch model with supplied batches, force non full-refresh behavior
This is necessary because of retry. Say on the initial run the microbatch model
succeeds on 97% of it's batches. Then on retry it does the last 3%. If the retry
of the microbatch model executes in full refresh mode it _might_ blow away the
97% of work that has been done. This edge case seems to be adapter specific.
* Only pass batches to retry for microbatch model when there was a PartialSuccess
In the previous commit we made it so that retries of microbatch models wouldn't
run in full refresh mode when the microbatch model to retry has batches already
specified from the prior run. This is only problematic when the run being retried
was a full refresh AND all the batches for a given microbatch model failed. In
that case WE DO want to do a full refresh for the given microbatch model. To better
outline the problem, consider the following:
* a microbatch model had a begin of `2020-01-01` and has been running this way for awhile
* the begin config has changed to `2024-01-01` and dbt run --full-refresh gets run
* every batch for an microbatch model fails
* on dbt retry the the relation is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purged
To avoid this, all we have to do is ONLY pass the batch information for partially successful microbatch
models. Note: microbatch models only have a partially successful status IFF they have both
successful and failed batches.
* Fix test_manifest unit tests to know about model 'batches' key
* Add some console output assertions to microbatch functional tests
* add batch_results: None to expected_run_results
* Add changie doc for microbatch retry functionality
* maintain protoc version 5.26.1
* Cleanup extraneous comment in LogModelResult
---------
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
* Test case for `merge_exclude_columns`
* Update expected output for `merge_exclude_columns`
* Skip TestMergeExcludeColumns test
* Enable this test since PostgreSQL 15+ is available in CI now
* Undo modification to expected output
* Remove duplicated constructor for `ResourceTypeSelector`
* Add type annotation for `ResourceTypeSelector`
* Standardize on constructor for `ResourceTypeSelector` where `include_empty_nodes=True`
* Changelog entry
* Adding logic to TestSelector to remove unit tests if they are in excluded_resource_types
* Adding change log
* Respect `--resource-type` and `--exclude-resource-type` CLI flags and associated environment variables
* Test CLI flag for excluding unit tests for the `dbt test` command
* Satisy isort pre-commit hook
* Fix mypy for positional argument "resource_types" in call to "TestSelector"
* Replace `TestSelector` with `ResourceTypeSelector`
* Add co-author
* Update changelog description
* Add functional tests for new feature
* Compare the actual results, not just the count
* Remove test case covered elsewhere
* Test for `DBT_EXCLUDE_RESOURCE_TYPES` environment variable for `dbt test`
* Update per pre-commit hook
* Restore to original form (until we refactor extraneous `ResourceTypeSelector` references later)
---------
Co-authored-by: Matthew Cooper <asimov.1st@gmail.com>
* initial rough-in with CLI flags
* dbt-adapters testing against event-time-ref-filtering
* fix TestList
* Checkpoint
* fix tests
* add event_time_start params to build
* rename configs
* Gate resolve_event_time_filter via micro batch strategy and fix strptime usage
* Add unit test for resolve_event_time_filter
* Additional unit tests for `resolve_event_time_filter` to ensure lookback + batch_size work
* Remove extraneous comments and print statements from resolve_event_time_filter
* Fixup microbatch functional tests to use microbatch strategy
* Gate microbatch functionality behind env_var while in beta
* Add comment about how _is_incremental should be removed
* Improve `event_time_start/end` cli parameters to auto convert to datetime objects
* for testing: dbt-postgres 'microbatch' strategy
* rough in: chunked backfills
* partial failure of microbatch runs
* decouple run result methods
* initial refactor
* rename configs to __dbt_internal
* update compiled_code in context after re-compilation
* finish rename of context vars
* changelog entry
* fix patch_microbatch_end_time
* refactor into MicrobatchBuilder
* fix provider unit tests + add unit tests for MicrobatchBuilder
* add TestMicrobatchJinjaContextVarsAvailable
* unit test offset + truncate timestamp methods
* Remove pairing.md file
* Add tying to microbatch specific functions added in `task/run.py`
* Add doc strings to microbatch.py functions and classes
* Set microbatch node status to `ERROR` if all batches for node failed
* Fire an event for batch exceptions instead of directly printing
* Fix firing of failed microbatch log event
---------
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* Update functional tests to cover this case
* Revert "Update functional tests to cover this case"
This reverts commit 4c78e816f6c370b7a39df774f2fab7328db5312b.
* New functional tests to cover the resource_type config
* Separate data tests from unit tests for `resource_types` config of `dbt list` and `dbt build`
* Changelog entry
* Add functional tests for custom incremental strategies names 'microbatch'
* Point dev-requirement of `dbt-adapters` back to the main branch
The associated branch/PR in `dbt-adapters` that we were previously
pointing to has been merged. Thus we can point back to `main` again.
---------
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* made class changing directory a context manager.
* add change log
* fix conflict
* made base as a context manager
* add assertion
* Remove index.html
* add it test to testDbtRunner
* fix deps args order
* fix test
---------
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
* use full manifest in adapter instead of macro_manifest
* Add test case
* Add changelog entry
* Remove commented code.
---------
Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
* Tests for calling a macro in a pre- or post-hook config in properties.yml
* Late render pre- and post-hooks configs in properties / schema YAML files
* Changelog entry
* Use alias instead of name when adding ephemeral model prefixes
* Adjust TestCustomSchemaWithCustomMacroFromModelName to test ephemeral models
* Add changelog entry for ephemeral model CTE identifier fix
* Reference model.identifier and model.name where appropriate to resolve typing errors
* Move test for ephemeral model with alias to dedicated test in test_compile.py
* update children search
* update search to include children in original selector
* add changie
* remove unused function
* fix wrong function call
* fix depth
* sketch
* Bring back the happy path fixture snapshot file
The commit c783a86 removed the snapshot file from the happy path fixture.
This was done because the snapshot was breaking the tests we were adding,
`test_run_commands`. However this broke `test_ls` in `test_list.py`. In order
to move forward, we need everything to be working. Maybe the idea was to delete
the `test_list.py` file, however that is not noted anywhere, and was not done.
Thus this commit ensures that test is not broken nor or new tests.
* Create conftest for `functional` tests so that happy path fixtures are accessible
* Format `test_commands.py` and update imports to appease pre-commit hooks
* Parametrize `test_run_command` to make it easier to see which command is failing (if any)
* Update the setup for `TestRunCommands.test_run_command` to be more formulaic
* Add test to ensure resource types are selectable
* Fix docstring formatting in TestRunCommands
* Fixup documentation for test_commands.py
---------
Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
* Add breakpoint.
* Move breakpoint.
* Add fix
* Add changelog.
* Avoid sorting for the string case.
* Add unit test.
* Fix test.
* add good unit tests for coverage of sort method.
* add sql format coverage.
* Modify behavior to log a warning and proceed.
* code review comments.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Fix exclusive_primary_alt_value_setting to set warn_error_options correctly
* Add test
* Changie
* Fix unit test
* Replace conversion method
* Refactor normalize_warn_error_options
* Add changelog.
* Avoid sorting for the string case.
* add good unit tests for coverage of sort method.
* add sql format coverage.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Correct `isort` configuration to include dbt-semantic-interfaces as internal
We thought we were already doing this. However, we accidentally missed the last
`s` of `dbt-semantic-interfaces`, so imports from dbt-semantic-interfaces were not
being identified as an internal package by isort. This fixes that.
* Run isort using updated configs to mark `dbt-semantic-interfaces` as included
* Fix `test_can_silence` tests in `test_warn_error_options.py` to ensure silencing
We're fixing an issue wherein `silence` specifications in the `dbt_project.yaml`
weren't being respected. This was odd since we had tests specifically for this.
It turns out the tests were broken. Essentially the warning was instead being raised
as an error due to `include: 'all'`. Then because it was being raised as an error,
the event wasn't going through the logger. We were only asserting in these tests that
the silenced event wasn't going through the logger (which it wasn't) so everything
"appeared" to be working. Unfortunately everything wasn't actually working. This is now
highlighted because `test_warn_error_options::TestWarnErrorOptionsFromProject:test_can_silence`
is now failing with this commit.
* Fix setting `warn_error_options` via `dbt_project.yaml` flags.
Back when I did the work for #10058 (specifically c52d6531) I thought that
the `warn_error_options` would automatically be converted from the yaml
to the `WarnErrorOptions` object as we were building the `ProjectFlags` object,
which holds `warn_error_options`, via `ProjectFlags.from_dict`. And I thought
this was validated by the `test_can_silence` test added in c52d6531. However,
there were two problems:
1. The definition of `warn_error_options` on `PrjectFlags` is a dict, not a
`WarnErrorOptions` object
2. The `test_can_silence` test was broken, and not testing what I thought
The quick fix (this commit) is to ensure `silence` is passed to `WarnErrorOptions`
instantiation in `dbt.cli.flags.convert_config`. The better fix would be to make
the `warn_error_options` of `ProjectFlags` a `WarnErrorOptions` object instead of
a dict. However, to do this we first need to update dbt-common's `WarnErrorOptions`
definition to default `include` to an empty list. Doing so would allow us to get rid
of `convert_config` entirely.
* Add unit test for `ModelRunner.print_result_line`
* Add (and skip) unit test for `ModelRunner.execute`
An attempt at testing `ModelRunner.execute`. We should probably also be
asserting that the model has been executed. However before we could get there,
we're running into runtime errors during `ModelRunner.execute`. Currently the
struggle is ensuring the adapter exists in the global factory when `execute`
goes looking for. The error we're getting looks like the following:
```
def test_execute(self, table_model: ModelNode, manifest: Manifest, model_runner: ModelRunner) -> None:
> model_runner.execute(model=table_model, manifest=manifest)
tests/unit/task/test_run.py:121:
----
core/dbt/task/run.py:259: in execute
context = generate_runtime_model_context(model, self.config, manifest)
core/dbt/context/providers.py:1636: in generate_runtime_model_context
ctx = ModelContext(model, config, manifest, RuntimeProvider(), None)
core/dbt/context/providers.py:834: in __init__
self.adapter = get_adapter(self.config)
venv/lib/python3.10/site-packages/dbt/adapters/factory.py:207: in get_adapter
return FACTORY.lookup_adapter(config.credentials.type)
---`
self = <dbt.adapters.factory.AdapterContainer object at 0x106e73280>, adapter_name = 'postgres'
def lookup_adapter(self, adapter_name: str) -> Adapter:
> return self.adapters[adapter_name]
E KeyError: 'postgres'
venv/lib/python3.10/site-packages/dbt/adapters/factory.py:132: KeyError
```
* Add `postgres_adapter` fixture for use in `TestModelRunner`
Previously we were running into an issue where the during `ModelRunner.execute`
the mock_adapter we were using wouldn't be found in the global adapter
factory. We've gotten past this error by supply a "real" adapter, a
`PostgresAdapter` instance. However we're now running into a new error
in which the materialization macro can't be found. This error looks like
```
model_runner = <dbt.task.run.ModelRunner object at 0x106746650>
def test_execute(
self, table_model: ModelNode, manifest: Manifest, model_runner: ModelRunner
) -> None:
> model_runner.execute(model=table_model, manifest=manifest)
tests/unit/task/test_run.py:129:
----
self = <dbt.task.run.ModelRunner object at 0x106746650>
model = ModelNode(database='dbt', schema='dbt_schema', name='table_model', resource_type=<NodeType.Model: 'model'>, package_na...ected'>, constraints=[], version=None, latest_version=None, deprecation_date=None, defer_relation=None, primary_key=[])
manifest = Manifest(nodes={'seed.pkg.seed': SeedNode(database='dbt', schema='dbt_schema', name='seed', resource_type=<NodeType.Se...s(show=True, node_color=None), patch_path=None, arguments=[], created_at=1718229810.21914, supported_languages=None)}})
def execute(self, model, manifest):
context = generate_runtime_model_context(model, self.config, manifest)
materialization_macro = manifest.find_materialization_macro_by_name(
self.config.project_name, model.get_materialization(), self.adapter.type()
)
if materialization_macro is None:
> raise MissingMaterializationError(
materialization=model.get_materialization(), adapter_type=self.adapter.type()
)
E dbt.adapters.exceptions.compilation.MissingMaterializationError: Compilation Error
E No materialization 'table' was found for adapter postgres! (searched types 'default' and 'postgres')
core/dbt/task/run.py:266: MissingMaterializationError
```
* Add spoofed macro fixture `materialization_table_default` for `test_execute` test
Previously the `TestModelRunner:test_execute` test was running into a runtime error
do to the macro `materialization_table_default` macro not existing in the project. This
commit adds that macro to the project (though it should ideally get loaded via interactions
between the manifest and adapter). Manually adding it resolved our previous issue, but created
a new one. The macro appears to not be properly loaded into the manifest, and thus isn't
discoverable later on when getting the macros for the jinja context. This leads to an error
that looks like the following:
```
model_runner = <dbt.task.run.ModelRunner object at 0x1080a4f70>
def test_execute(
self, table_model: ModelNode, manifest: Manifest, model_runner: ModelRunner
) -> None:
> model_runner.execute(model=table_model, manifest=manifest)
tests/unit/task/test_run.py:129:
----
core/dbt/task/run.py:287: in execute
result = MacroGenerator(
core/dbt/clients/jinja.py:82: in __call__
return self.call_macro(*args, **kwargs)
venv/lib/python3.10/site-packages/dbt_common/clients/jinja.py:294: in call_macro
macro = self.get_macro()
---
self = <dbt.clients.jinja.MacroGenerator object at 0x1080f3130>
def get_macro(self):
name = self.get_name()
template = self.get_template()
# make the module. previously we set both vars and local, but that's
# redundant: They both end up in the same place
# make_module is in jinja2.environment. It returns a TemplateModule
module = template.make_module(vars=self.context, shared=False)
> macro = module.__dict__[get_dbt_macro_name(name)]
E KeyError: 'dbt_macro__materialization_table_default'
venv/lib/python3.10/site-packages/dbt_common/clients/jinja.py:277: KeyError
```
It's becoming apparent that we need to find a better way to either mock or legitimately
load the default and adapter macros. At this point I think I've exausted the time box
I should be using to figure out if testing the `ModelRunner` class is possible currently,
with the result being more work has yet to be done.
* Begin adding the `LogModelResult` event catcher to event manager class fixture
* init push for issue 10198
* add changelog
* add unit tests based on michelle example
* add data_tests, and post_hook unit tests
* pull creating macro_func out of try call
* revert last commit
* pull macro_func definition back out of try
* update code formatting
* Add basic semantic layer fixture nodes to unit test `manifest` fixture
We're doing this in preperation to a for a unit test which will be testing
these nodes (as well as others) and thus we want them in the manifest.
* Add `WhereFilterInteresection` to `QueryParams` of `saved_query` fixture
In the previous commit, 58990aa450, we added
the `saved_query` fixture to the `manifest` fixture. This broke the test
`tests/unit/parser/test_manifest.py::TestPartialParse::test_partial_parse_by_version`.
It broke because the `Manifest.deepcopy` manifest basically dictifies things. When we were
dictifying the `QueryParams` of the `saved_query` before, the `where` key was getting
dropped because it was `None`. We'd then run into a runtime error on instantiation of the
`QueryParams` because although `where` is declared as _optional_, we don't set a default
value for it. And thus, kaboom :(
We should probably provide a default value for `where`. However that is out of scope
for this branch of work.
* Fix `test_select_fqn` to account for newly included semantic model
In 58990aa450 we added a semantic model
to the `manifest` fixture. This broke the test
`tests/unit/graph/test_selector_methods.py::test_select_fqn` because in
the test it selects nodes based on the string `*.*.*_model`. The newly
added semantic model matches this, and thus needed to be added to the
expected results.
* Add unit tests for `_warn_for_unused_resource_config_paths` method
Note: At this point the test when run with for a `unit_test` config
taht should be considered used, fails. This is because it is not being
properly identified as used.
* Include `unit_tests` in `Manifest.get_resouce_fqns`
Because `unit_tests` weren't being included in calls to `Manifest.get_resource.fqns`,
it always appeared to `_warn_for_unused_resource_config_paths` that there were no
unit tests in the manifest. Because of this `_warn_for_unused_resource_config_paths` thought
that _any_ `unit_test` config in `dbt_project.yaml` was unused.
* Rename `maniest` fixture in `test_selector` to `mock_manifest`
We have a globally available `manifest` fixture in our unit tests. In the
coming commits we're going to add tests to the file which use the gloablly
available `manifest` fixture. Prior to this commit, the locally defined
`manifest` fixture was taking precidence. To get around this, the easiest
solution was to rename the locally defined fixture.
I had tried to isolate the locally defined fixture by moving it, and the relevant
tests to a test class like `TestNodeSelector`. However because of _how_ the relevant
tests were parameterized, this proved difficult. Basically for readability we define
a variable which holds a list of all the parameterization variables. By moving to a
test class, the definition of the variables would have had to be defined directly in
the parameterization macro call. Although possible, it made the readability slighty
worse. It might be worth doing anyway in the long run, but instead I used a less heavy
handed alternative (already described)
* Improve type hinting in `tests/unit/utils/manifest.py`
* Ensure `args` get set from global flags for `runtime_config` fixture in unit tests
The `Compiler.compile` method accesses `self.config.args.which`. The `config`
is the `RuntimeConfig` the `Compiler` was instantiated with. Our `runtime_config`
fixture was being instatiated with an empty dict for the `args` property. Thus
a `which` property of the args wasn't being made avaiable, and if `compile` was run
a runtime error would occur. To solve this, we've begun instantiating the args from
the global flags via `get_flags()`. This works because we ensure the `set_test_flags`
fixture is run first which calls `set_from_args`.
* Create a `make_manifest` utility function for use in unit tests and fixture creation
* Refactor `Compiler` and `NodeSelector` tests in `test_selector.py` to use pytesting methodology
* Remove parsing tests that exist in `test_selector.py`
We had some tests in `test_selector.py::GraphTest` that didn't add
anything ontop of what was already being tested else where in the file
except the parsing of models. However, the tests in `test_parser.py::ModelParserTest`
cover everything being tested here (and then some). Thus these tests
in `test_selector.py::GraphTest` are unnecessary and can be deleted.
* Move `test__partial_parse` from `test_selector.py` to `test_manifest.py`
There was a test `test__partial_parse` in `test_selector.py` which tested
the functionality of `is_partial_parsable` of the `ManifestLoader`. This
doesn't really make sense to exist in `test_selector.py` where we are
testing selectors. We test the `ManifestLoader` class in `test_manifest.py`
which seemed like a more appropriate place for the test. Additionally we
renamed the test to `test_is_partial_parsable_by_version` to more accurately
describe what is being tested.
* Make `test_selector`'s manifest fixture name even more specific
* Add type hint to `expected_nodes` in `test_selector.py` tests
In the test `tests/unit/graph/test_selector.py::TestCompiler::test_two_models_simple_ref`
we have a variable `expected_nodes` that we are setting via a list comprehension.
At a glance it isn't immediately obvious what `expected_nodes` actually is. It's a
list, but a list of what? One suggestion was to explicitly write out the list of strings.
However, I worry about the brittleness of doing so. That might be the way we head long
term, but as a compromise for now, I've added a type hint the the variable definition.
* Fire skipped events at debug level
Closes https://github.com/dbt-labs/dbt-core/issues/8774
* add changelog entry
* Update to work with 1.9.*.
* Add tests for --fail-fast not showing skip messages unless --debug.
* Update test that works by itself, but assumes to much to work in integration tests.
---------
Co-authored-by: Scott Gigante <scottgigante@users.noreply.github.com>
* init push arbitrary configs for generic tests pr
* iterative work
* initial test design attempts
* test reformatting
* test rework, have basic structure for 3 of 4 passing, need to figure out how to best represent same key error, failing correctly though
* swap up test formats for new config dict and mixed varitey to run of dbt parse and inspecting the manifest
* modify tests to get passing, then modify the TestBuilder class work from earlier to be more dry
* add changelog
* modify code to match suggested changes around seperate methods and test id fix
* add column_name reference to init for deeper nested _render_values can use the input
* add type annotations
* feedback based on mike review
* Create `runtime_config` fixture and necessary upstream fixtures
* Check for better scoped `ProjectContractError` in test_runtime tests
Previously in `test_unsupported_version_extra_config` and
`test_archive_not_allowed` we were checking for `DbtProjectError`. This
worked because `ProjectContractError` is a subclass of `DbtProjectError`.
However, if we check for `DbtProjectError` in these tests than, some tangential
failure which raises a `DbtProejctError` type error would go undetected. As
we plan on modifying these tests to be pytest in the coming commits, we want to
ensure that the tests are succeeding for the right reason.
* Convert `test_str` of `TestRuntimeConfig` to a pytest test using fixtures
* Convert `test_from_parts` of `TestRuntimeConfig` to a pytest test using fixtures
While converting `test_from_parts` I noticed the comment
> TODO(jeb): Adapters must assert that quoting is populated?
This led me to beleive that `quoting` shouldn't be "fully" realized
in our project fixture unless we're saying that it's gone through
adapter instantiation. Thus I update the `quoting` on our project
fixture to be an empty dict. This change affected `test__str__` in
`test_project.py` which we thus needed to update accordingly.
* Convert runtime version specifier tests to pytest tests and move to test_project
We've done two things in this commit, which arguably _should_ have been done in
two commits. First we moved the version specifier tests from `test_runtime.py::TestRuntimeConfig`
to `test_project.py::TestGetRequiredVersion` this is because what is really being
tested is the `_get_required_version` method. Doing it via `RuntimeConfig.from_parts` method
made actually testing it a lot harder as it requires setting up more of the world and
running with a _full_ project config dict.
The second thing we did was convert it from the old unittest implementation to a pytest
implementation. This saves us from having to create most of the world as we were doing
previously in these tests.
Of note, I did not move the test `test_unsupported_version_range_bad_config`. This test
is a bit different from the rest of the version specifier tests. It was introduced in
[1eb5857811](1eb5857811)
of [#2726](https://github.com/dbt-labs/dbt-core/pull/2726) to resolve [#2638](https://github.com/dbt-labs/dbt-core/issues/2638).
The focus of #2726 was to ensure the version specifier checks were run _before_ the validation
of the `dbt_project.yml`. Thus what this test is actually testing for is order of
operations at parse time. As such, this is really more a _functional_ test than a
unit test. In the next commit we'll get this test moved (and renamed)
* Create a better test for checking that version checks come before project schema validation
* Convert `test_get_metadata` to pytest test
* Refactor `test_archive_not_allowed` to functional test
We do already have tests that ensure "extra" keys aren't allowed in
the dbt_project.yaml. This test is different because it's checking that
a specific key, `archive`, isn't allowed. We do this because at one point
in time `archive` _was_ an allowed key. Specifically, we stopped allowing
`archive` in dbt-core 0.15.0 via commit [f26948dd](f26948dde2).
Given that it's been 5 years and a major version, we could probably remove
this test, but let's keep it around unless we start supporting `archive` again.
* Convert `warn_for_unused_resource_config_paths` tests to use pytest
* Add fixtures for setting and resettign flags for unit tests
* Remove unnecessary `set_from_args` in non `unittest.TestCase` based unit tests
In the previous commit we added a pytest fixture which sets and tears down
the global flags arg via `set_from_args` for every pytest based unit test.
Previously we had added a `set_from_args` in tests or test files to reset
the global flags from if they were modified by a previous test. This is no
longer necessary because of the work done in the previous commit.
Note: We did not modify any tests that use the `unittest.TestCase` class
because they don't use pytest fixtures. Thus those tests need to continue
operating as they currently do until we shift them to pytest unit tests.
* Utilize the new `args_for_flags` fixture for setting of flags in `test_contracts_graph_parsed.py`
* Convert `test_compilation.py` from `TestCase` tests to pytest tests
We did this so in the next commit we can drop the unnecessary `set_from_args`
in the next commit. That will be it's own commit because converting these
tests is a restructuring that doing separately makes things easier to follow.
That is to say, all changes in this commit were just to convert the tests to
pytest, no other changes were made.
* Drop unnecessary `set_from_args` in `test_compilation.py`
* Add return types to all methods in `test_compilation.py`
* Reduce imports from `compilation` in `test_compilation.py`
* Update `test_logging.py` now that we don't need to worry about global flags
* Conditionally import `Generator` type for python 3.8
In python 3.9 `Generator` was moved to `collections.abc` and deprecated
in `typing`. We still support 3.8 and thus need to be conditionally
importing `Generator`. We should remove this in the future when we drop
support for 3.8.
* Add more accurate RSS high water mark measurement for Linux
* Add changelog entry.
* Checks to avoid exception based flow control, per review.
* Fix review nit.
* Add unit test to assert `setup_config_logger` clears the event manager state
* Move `setup_event_logger` tests from `test_functions.py` to `test_logging.py`
* Move `EventCatcher` to `tests.utils` for use in unit and functional tests
* Update fixture mocking global event manager to instead clear it
Previously we had started _mocking_ the global event manager. We did this
because we though that meant anything we did to the global event manager,
including modifying it via things like `setup_event_logger`, would be
automatically cleaned up at the end of any test using the fixture because
the mock would go away. However, this understanding of fixtures and mocking
was incorrect, and the global event manager wouldn't be properly isolated/reset.
Thus we changed to fixture to instead cleanup the global event manager before
any test that uses it and by using `autouse=True` in the fixture definition
we made it so that every unit test uses the fixture.
Note this will no longer be viabled if we every multi-thread our unit testing as
the event manager isn't actually isolated, and thus two tests could both modify
the event manager at the same time.
* Add test for different `write_perf_info` values to `get_full_manifest`
* Add test for different `reset` values to `get_full_manifest`
* Abstract required mocks for `get_full_manifest` tests to reduce duplication
There are a set of required mocks that `get_full_manifest` unit tests need.
Instead of doing these mocks in each test, we've abstracted these mocks into
a reusable function. I did try to do this as a fixture, but for some reaosn
the mocks didn't actually propagate when I did that.
* Add test for different `PARTIAL_PARSE_FILE_DIFF` values to `get_full_manifest`
* Refactor mock fixtures in `test_manifest.py` to make them more widely available
* Convert `set_required_mocks` of `TestGetFullManifest` into a fixture
This wasn't working before, but it does now. Not sure why.
This was done by running `pre-commit run --all`. That this was needed
is a temporary glitch in how our `Tests and Code Checks` github action
works on PRs. Basically we added `isort` to the pre-commit hooks recently, and
this does additional linting/formatting on our imports.
People reasonably have branches which were started prior to `isort` being
part of the pre-commit hooks on main. Thus, unless those branches get caught
up to main, the github action on associated PRs won't run `isort` because
it doesn't exist on those branchs. Once everyone gets their local `main`
branch updated (I suspect this might take a few days) this problem will go
away.
* Add `isort` as a dev-req and pre-commit hook
The tool `isort` reorders imports to be in alphabetical order. I've
added it because our imports in most files are in random order. The lack
of order meant that:
- sometimes the same module would be imported from twice
- figuring out if a a module was already being imported from took
longer
In the next commit I'll actually run isort to order everything. The best
part is that when developing, we don't have to put them in correct order.
Though you can if you want. However, `isort` will take care of re-ordering
things at commit time :)
* Improve isort functionality by setting initial `.isort.cfg`
Specifically we set two config values: `extend_skip_glob` and `known_first_party`.
The `extend_skip_glob` extends the default skipped paths. The defaults can be seen
here https://pycqa.github.io/isort/docs/configuration/options.html#skip. We are skipping
third party stubs because these are more so provided (I believe). We are skipping
`.github` and `scripts` as they feel out of scope and things we can be less strict with.
The `known_first_party` setting makes it so that these imports get grouped separately from
all other imports, which is useful visually of "this comes from us" vs "this comes from
someone/somewhere else".
* Add profile `black` to isort config
I was seeing some odd behavior where running pre-commit, adding the modified
files, and then running pre-commit again would result making more modifications
to some of the same files. This felt odd. You shouldn't have to run pre-commit
more multiple times for it to eventually come to a final "solution". I believe
the problem was because we are using the tool `black` to format things, but weren't
registering the black profile with `isort` this lead to some conflicting formatting
rules, and the two tools had to negotiate a few times before being both satisfied.
Registering the profile `black` with `isort` resolved this problem.
* Reorder, merge-duplicate, and format module imports using `isort`
This was done by running `pre-commit run --all`. I ran it separately from
the commit process itself because I wanted to run it against all files
instead of only changed files.
Of note, this not only reordered and formatted our imports. But we also
had 60 distinct duplicate module import paths across 50 files, which this
took care of. When I say "distinct duplicate module import paths" I mean
when `from x.y.z import` was imported more than once in a single file.
2024-05-03 09:28:11 -07:00
888 changed files with 27388 additions and 17271 deletions
- Move AdapterLogger to adapters folder ([#9151](https://github.com/dbt-labs/dbt-core/issues/9151))
- Rm --dry-run flag from 'dbt deps --add-package', in favor of just 'dbt deps --lock' ([#9100](https://github.com/dbt-labs/dbt-core/issues/9100))
- move event manager setup back to core, remove ref to global EVENT_MANAGER and clean up event manager functions ([#9150](https://github.com/dbt-labs/dbt-core/issues/9150))
- Remove dbt-tests-adapter and dbt-postgres packages from dbt-core ([#9455](https://github.com/dbt-labs/dbt-core/issues/9455))
### Features
- Initial implementation of unit testing ([#8287](https://github.com/dbt-labs/dbt-core/issues/8287))
- Unit test manifest artifacts and selection ([#8295](https://github.com/dbt-labs/dbt-core/issues/8295))
- Support config with tags & meta for unit tests ([#8294](https://github.com/dbt-labs/dbt-core/issues/8294))
- Allow adapters to include package logs in dbt standard logging ([#7859](https://github.com/dbt-labs/dbt-core/issues/7859))
- Enable inline csv fixtures in unit tests ([#8626](https://github.com/dbt-labs/dbt-core/issues/8626))
- Unit tests support --defer and state:modified ([#8517](https://github.com/dbt-labs/dbt-core/issues/8517))
- Support setting export configs hierarchically via saved query and project configs ([#8956](https://github.com/dbt-labs/dbt-core/issues/8956))
- Support source inputs in unit tests ([#8507](https://github.com/dbt-labs/dbt-core/issues/8507))
- Use daff to render diff displayed in stdout when unit test fails ([#8558](https://github.com/dbt-labs/dbt-core/issues/8558))
- Move unit testing to test command ([#8979](https://github.com/dbt-labs/dbt-core/issues/8979))
- Support --empty flag for schema-only dry runs ([#8971](https://github.com/dbt-labs/dbt-core/issues/8971))
- Support unit tests in non-root packages ([#8285](https://github.com/dbt-labs/dbt-core/issues/8285))
- Convert the `tests` config to `data_tests` in both dbt_project.yml and schema files. in schema files. ([#8699](https://github.com/dbt-labs/dbt-core/issues/8699))
- Make fixture files full-fledged parts of the manifest and enable partial parsing ([#9067](https://github.com/dbt-labs/dbt-core/issues/9067))
- Adds support for parsing conversion metric related properties for the semantic layer. ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))
- In build command run unit tests before models ([#9128](https://github.com/dbt-labs/dbt-core/issues/9128))
- Move flags from UserConfig in profiles.yml to flags in dbt_project.yml ([#9183](https://github.com/dbt-labs/dbt-core/issues/9183))
- Added hook support for `dbt source freshness` ([#5609](https://github.com/dbt-labs/dbt-core/issues/5609))
- Align with order of unit test output when `actual` differs from `expected` ([#9370](https://github.com/dbt-labs/dbt-core/issues/9370))
- Added support for external nodes in unit test nodes ([#8944](https://github.com/dbt-labs/dbt-core/issues/8944))
- Enable unit testing versioned models ([#9344](https://github.com/dbt-labs/dbt-core/issues/9344))
- Enable list command for unit tests ([#8508](https://github.com/dbt-labs/dbt-core/issues/8508))
- Integration Test Optimizations ([#9498](https://github.com/dbt-labs/dbt-core/issues/9498))
- Accelerate integration tests with caching. ([#9498](https://github.com/dbt-labs/dbt-core/issues/9498))
- Support meta at the config level for Metric nodes ([#9441](https://github.com/dbt-labs/dbt-core/issues/9441))
- Add cache to SavedQuery config ([#9540](https://github.com/dbt-labs/dbt-core/issues/9540))
### Fixes
- For packages installed with tarball method, fetch metadata to resolve nested dependencies ([#8621](https://github.com/dbt-labs/dbt-core/issues/8621))
- Fix partial parsing not working for semantic model change ([#8859](https://github.com/dbt-labs/dbt-core/issues/8859))
- Handle unknown `type_code` for model contracts ([#8877](https://github.com/dbt-labs/dbt-core/issues/8877), [#8353](https://github.com/dbt-labs/dbt-core/issues/8353))
- Add back contract enforcement for temporary tables on postgres ([#8857](https://github.com/dbt-labs/dbt-core/issues/8857))
- Add version to fqn when version==0 ([#8836](https://github.com/dbt-labs/dbt-core/issues/8836))
- Fix cased comparison in catalog-retrieval function. ([#8939](https://github.com/dbt-labs/dbt-core/issues/8939))
- Catalog queries now assign the correct type to materialized views ([#8864](https://github.com/dbt-labs/dbt-core/issues/8864))
- Fix compilation exception running empty seed file and support new Integer agate data_type ([#8895](https://github.com/dbt-labs/dbt-core/issues/8895))
- Make relation filtering None-tolerant for maximal flexibility across adapters. ([#8974](https://github.com/dbt-labs/dbt-core/issues/8974))
- Update run_results.json from previous versions of dbt to support deferral and rerun from failure ([#9010](https://github.com/dbt-labs/dbt-core/issues/9010))
- Use MANIFEST.in to recursively include all jinja templates; fixes issue where some templates were not included in the distribution ([#9016](https://github.com/dbt-labs/dbt-core/issues/9016))
- Fix git repository with subdirectory for Deps ([#9000](https://github.com/dbt-labs/dbt-core/issues/9000))
- Use seed file from disk for unit testing if rows not specified in YAML config ([#8652](https://github.com/dbt-labs/dbt-core/issues/8652))
- Fix formatting of tarball information in packages-lock.yml ([#9062](https://github.com/dbt-labs/dbt-core/issues/9062))
- deps: Lock git packages to commit SHA during resolution ([#9050](https://github.com/dbt-labs/dbt-core/issues/9050))
- deps: Use PackageRenderer to read package-lock.json ([#9127](https://github.com/dbt-labs/dbt-core/issues/9127))
- Ensure we produce valid jsonschema schemas for manifest, catalog, run-results, and sources ([#8991](https://github.com/dbt-labs/dbt-core/issues/8991))
- Get sources working again in dbt docs generate ([#9119](https://github.com/dbt-labs/dbt-core/issues/9119))
- Fix parsing f-strings in python models ([#6976](https://github.com/dbt-labs/dbt-core/issues/6976))
- Preserve the value of vars and the --full-refresh flags when using retry. ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))
- Support reasonably long unit test names ([#9015](https://github.com/dbt-labs/dbt-core/issues/9015))
- Fix back-compat parsing for model-level 'tests', source table-level 'tests', and 'tests' defined on model versions ([#9411](https://github.com/dbt-labs/dbt-core/issues/9411))
- Fix retry command run from CLI ([#9444](https://github.com/dbt-labs/dbt-core/issues/9444))
- Fix seed and source selection in `dbt docs generate` ([#9161](https://github.com/dbt-labs/dbt-core/issues/9161))
- Add TestGenerateCatalogWithExternalNodes, include empty nodes in node selection during docs generate ([#9456](https://github.com/dbt-labs/dbt-core/issues/9456))
- Fix node type plurals in FoundStats log message ([#9464](https://github.com/dbt-labs/dbt-core/issues/9464))
- Run manifest upgrade preprocessing on any older manifest version, including v11 ([#9487](https://github.com/dbt-labs/dbt-core/issues/9487))
- Update 'compiled_code' context member logic to route based on command ('clone' or not). Reimplement 'sql' context member as wrapper of 'compiled_code'. ([#9502](https://github.com/dbt-labs/dbt-core/issues/9502))
- Fix bug where Semantic Layer filter strings are parsed into lists. ([#9507](https://github.com/dbt-labs/dbt-core/issues/9507))
- Initialize invocation context before test fixtures are built. ([##9489](https://github.com/dbt-labs/dbt-core/issues/#9489))
- When patching versioned models, set constraints after config ([#9364](https://github.com/dbt-labs/dbt-core/issues/9364))
- only include unmodified semantic mdodels in state:modified selection ([#9548](https://github.com/dbt-labs/dbt-core/issues/9548))
- Set query headers when manifest is passed in to dbtRunner ([#9546](https://github.com/dbt-labs/dbt-core/issues/9546))
- Store node_info in node associated logging events ([#9557](https://github.com/dbt-labs/dbt-core/issues/9557))
- Fix Semantic Model Compare node relations ([#9548](https://github.com/dbt-labs/dbt-core/issues/9548))
- Clearer no-op logging in stubbed SavedQueryRunner ([#9533](https://github.com/dbt-labs/dbt-core/issues/9533))
- Added more type annotations. ([#8537](https://github.com/dbt-labs/dbt-core/issues/8537))
- Add unit testing functional tests ([#8512](https://github.com/dbt-labs/dbt-core/issues/8512))
- Remove usage of dbt.include.global_project in dbt/adapters ([#8925](https://github.com/dbt-labs/dbt-core/issues/8925))
- Add a no-op runner for Saved Qeury ([#8893](https://github.com/dbt-labs/dbt-core/issues/8893))
- remove dbt.flags.MP_CONTEXT usage in dbt/adapters ([#8967](https://github.com/dbt-labs/dbt-core/issues/8967))
- Remove usage of dbt.flags.LOG_CACHE_EVENTS in dbt/adapters ([#8969](https://github.com/dbt-labs/dbt-core/issues/8969))
- Move CatalogRelationTypes test case to the shared test suite to be reused by adapter maintainers ([#8952](https://github.com/dbt-labs/dbt-core/issues/8952))
- Treat SystemExit as an interrupt if raised during node execution. ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))
- Fix test_current_timestamp_matches_utc test; allow for MacOS runner system clock variance ([#9057](https://github.com/dbt-labs/dbt-core/issues/9057))
- Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific event types and protos ([#8927](https://github.com/dbt-labs/dbt-core/issues/8927), [#8918](https://github.com/dbt-labs/dbt-core/issues/8918))
- Clean up unused adaptor folders ([#9123](https://github.com/dbt-labs/dbt-core/issues/9123))
- Move column constraints into common/contracts, removing another dependency of adapters on core. ([#9024](https://github.com/dbt-labs/dbt-core/issues/9024))
- Move dbt.semver to dbt.common.semver and update references. ([#9039](https://github.com/dbt-labs/dbt-core/issues/9039))
- Move lowercase utils method to common ([#9180](https://github.com/dbt-labs/dbt-core/issues/9180))
- Remove usages of dbt.clients.jinja in dbt/adapters ([#9205](https://github.com/dbt-labs/dbt-core/issues/9205))
- Remove usage of dbt.contracts in dbt/adapters ([#9208](https://github.com/dbt-labs/dbt-core/issues/9208))
- Remove usage of dbt.contracts.graph.nodes.ResultNode in dbt/adapters ([#9214](https://github.com/dbt-labs/dbt-core/issues/9214))
- remove manifest from adapter.set_relations_cache signature ([#9217](https://github.com/dbt-labs/dbt-core/issues/9217))
- remove manifest from adapter catalog method signatures ([#9218](https://github.com/dbt-labs/dbt-core/issues/9218))
- Move BaseConfig, Metadata and various other contract classes from model_config to common/contracts/config ([#8919](https://github.com/dbt-labs/dbt-core/issues/8919))
- Add MacroResolverProtocol, remove lazy loading of manifest in adapter.execute_macro ([#9244](https://github.com/dbt-labs/dbt-core/issues/9244))
- pass query header context to MacroQueryStringSetter ([#9249](https://github.com/dbt-labs/dbt-core/issues/9249), [#9250](https://github.com/dbt-labs/dbt-core/issues/9250))
- add macro_context_generator on adapter ([#9247](https://github.com/dbt-labs/dbt-core/issues/9247))
- pass mp_context to adapter factory as argument instead of import ([#9025](https://github.com/dbt-labs/dbt-core/issues/9025))
- have dbt-postgres use RelationConfig protocol for materialized views' ([#9292](https://github.com/dbt-labs/dbt-core/issues/9292))
- move system.py to common as dbt-bigquery relies on it to call gcloud ([#9293](https://github.com/dbt-labs/dbt-core/issues/9293))
- Reorganizing event definitions to define core events in dbt/events rather than dbt/common ([#9152](https://github.com/dbt-labs/dbt-core/issues/9152))
- move exceptions used only in dbt/common to dbt/common/exceptions ([#9332](https://github.com/dbt-labs/dbt-core/issues/9332))
- Remove usage of dbt.adapters.factory in dbt/common ([#9334](https://github.com/dbt-labs/dbt-core/issues/9334))
- Accept valid_error_names in WarnErrorOptions constructor, remove global usage of event modules ([#9337](https://github.com/dbt-labs/dbt-core/issues/9337))
- Move result objects to dbt.artifacts ([#9193](https://github.com/dbt-labs/dbt-core/issues/9193))
- dbt Labs OSS standardization of docs and templates. ([#9252](https://github.com/dbt-labs/dbt-core/issues/9252))
- Add dbt-common as a dependency and remove dbt/common ([#9357](https://github.com/dbt-labs/dbt-core/issues/9357))
- move cache exceptions to dbt/adapters ([#9362](https://github.com/dbt-labs/dbt-core/issues/9362))
- Clean up macro contexts. ([#9422](https://github.com/dbt-labs/dbt-core/issues/9422))
- Add the @requires.manifest decorator to the retry command. ([#9426](https://github.com/dbt-labs/dbt-core/issues/9426))
- Move WritableManifest + Documentation to dbt/artifacts ([#9378](https://github.com/dbt-labs/dbt-core/issues/9378), [#9379](https://github.com/dbt-labs/dbt-core/issues/9379))
- Define Macro and Group resources in dbt/artifacts ([#9381](https://github.com/dbt-labs/dbt-core/issues/9381), [#9382](https://github.com/dbt-labs/dbt-core/issues/9382))
- Move `SavedQuery` data definition to `dbt/artifacts` ([#9386](https://github.com/dbt-labs/dbt-core/issues/9386))
- Migrate data parts of `Metric` node to dbt/artifacts ([#9383](https://github.com/dbt-labs/dbt-core/issues/9383))
- Move data portion of `SemanticModel` to dbt/artifacts ([#9387](https://github.com/dbt-labs/dbt-core/issues/9387))
- Move data parts of `Exposure` class to dbt/artifacts ([#9380](https://github.com/dbt-labs/dbt-core/issues/9380))
- Start using `Mergeable` from dbt-common ([#9505](https://github.com/dbt-labs/dbt-core/issues/9505))
- Move manifest nodes to artifacts ([#9388](https://github.com/dbt-labs/dbt-core/issues/9388))
- Move data parts of `SourceDefinition` class to dbt/artifacts ([#9384](https://github.com/dbt-labs/dbt-core/issues/9384))
- Remove uses of Replaceable class ([#7802](https://github.com/dbt-labs/dbt-core/issues/7802))
- Make dbt-core compatible with Python 3.12 ([#9007](https://github.com/dbt-labs/dbt-core/issues/9007))
- Restrict protobuf to major version 4. ([#9566](https://github.com/dbt-labs/dbt-core/issues/9566))
- Remove references to dbt.tracking and dbt.flags from dbt/artifacts ([#9390](https://github.com/dbt-labs/dbt-core/issues/9390))
- Implement primary key inference for model nodes ([#9652](https://github.com/dbt-labs/dbt-core/issues/9652))
- Define UnitTestDefinition resource in dbt/artifacts/resources ([#9667](https://github.com/dbt-labs/dbt-core/issues/9667))
- Use Manifest instead of WritableManifest in PreviousState and _get_deferred_manifest ([#9567](https://github.com/dbt-labs/dbt-core/issues/9567))
### Dependencies
- Bump actions/checkout from 3 to 4 ([#8781](https://github.com/dbt-labs/dbt-core/pull/8781))
- Begin using DSI 0.4.x ([#8892](https://github.com/dbt-labs/dbt-core/pull/8892))
- Update typing-extensions version to >=4.4 ([#9012](https://github.com/dbt-labs/dbt-core/pull/9012))
- Bump ddtrace from 2.1.7 to 2.3.0 ([#9132](https://github.com/dbt-labs/dbt-core/pull/9132))
- Bump freezegun from 0.3.12 to 1.3.0 ([#9197](https://github.com/dbt-labs/dbt-core/pull/9197))
- Bump actions/setup-python from 4 to 5 ([#9267](https://github.com/dbt-labs/dbt-core/pull/9267))
- Bump actions/download-artifact from 3 to 4 ([#9374](https://github.com/dbt-labs/dbt-core/pull/9374))
- remove dbt/adapters and add dependency on dbt-adapters ([#9430](https://github.com/dbt-labs/dbt-core/pull/9430))
- Bump actions/cache from 3 to 4 ([#9471](https://github.com/dbt-labs/dbt-core/pull/9471))
- Bump peter-evans/create-pull-request from 5 to 6 ([#9552](https://github.com/dbt-labs/dbt-core/pull/9552))
- Cap dbt-semantic-interfaces version range to <0.6([#9671](https://github.com/dbt-labs/dbt-core/pull/9671))
- Global config for --target and --profile CLI flags and DBT_TARGET and DBT_PROFILE environment variables. ([#7798](https://github.com/dbt-labs/dbt-core/issues/7798))
- Allow excluding resource types for build, list, and clone commands, and provide env vars ([#9237](https://github.com/dbt-labs/dbt-core/issues/9237))
- SourceDefinition.meta represents source-level and table-level meta properties, instead of only table-level ([#9766](https://github.com/dbt-labs/dbt-core/issues/9766))
- Allow metrics in semantic layer filters. ([#9804](https://github.com/dbt-labs/dbt-core/issues/9804))
### Fixes
- fix lock-file bad indentation ([#9319](https://github.com/dbt-labs/dbt-core/issues/9319))
- Tighten exception handling to avoid worker thread hangs. ([#9583](https://github.com/dbt-labs/dbt-core/issues/9583))
- Do not add duplicate input_measures ([#9360](https://github.com/dbt-labs/dbt-core/issues/9360))
- Throw a ParsingError if a primary key constraint is defined on multiple columns or at both the column and model level. ([#9581](https://github.com/dbt-labs/dbt-core/issues/9581))
- Bug fix: don't parse Jinja in filters for input metrics or measures. ([#9582](https://github.com/dbt-labs/dbt-core/issues/9582))
- Fix traceback parsing for exceptions raised due to csv fixtures moved into or out of fixture/subfolders. ([#9570](https://github.com/dbt-labs/dbt-core/issues/9570))
- Fix partial parsing `KeyError` on deleted schema files ([#8860](https://github.com/dbt-labs/dbt-core/issues/8860))
- Support saved queries in `dbt list` ([#9532](https://github.com/dbt-labs/dbt-core/issues/9532))
- include sources in catalog.json when over 100 relations selected for catalog generation ([#9755](https://github.com/dbt-labs/dbt-core/issues/9755))
- Support overriding macros in packages in unit testing ([#9624](https://github.com/dbt-labs/dbt-core/issues/9624))
- Handle exceptions for failing on-run-* hooks in source freshness ([#9511](https://github.com/dbt-labs/dbt-core/issues/9511))
- Validation of unit test parsing for incremental models ([#9593](https://github.com/dbt-labs/dbt-core/issues/9593))
- Fix use of retry command on command using defer ([#9770](https://github.com/dbt-labs/dbt-core/issues/9770))
- Make `args` variable to be un-modified by `dbt.invoke(args)` ([#8938](https://github.com/dbt-labs/dbt-core/issues/8938), [#9787](https://github.com/dbt-labs/dbt-core/issues/9787))
- Unit test path outputs ([#9608](https://github.com/dbt-labs/dbt-core/issues/9608))
- Fix assorted source freshness edgecases so check is run or actionable information is given ([#9078](https://github.com/dbt-labs/dbt-core/issues/9078))
- "Fix Docker release process to account for both historical and current versions of `dbt-postgres` ([#9827](https://github.com/dbt-labs/dbt-core/issues/9827))
### Docs
- Add analytics for dbt.com ([dbt-docs/#430](https://github.com/dbt-labs/dbt-docs/issues/430))
### Under the Hood
- Remove unused key `wildcard` from MethodName enum ([#9641](https://github.com/dbt-labs/dbt-core/issues/9641))
- Support scrubbing secret vars ([#7247](https://github.com/dbt-labs/dbt-core/issues/7247))
- Add wildcard support to the group selector method ([#9811](https://github.com/dbt-labs/dbt-core/issues/9811))
- source freshness precomputes metadata-based freshness in batch, if possible ([#8705](https://github.com/dbt-labs/dbt-core/issues/8705))
- Better error message when trying to select a disabled model ([#9747](https://github.com/dbt-labs/dbt-core/issues/9747))
- Support SQL in unit testing fixtures ([#9405](https://github.com/dbt-labs/dbt-core/issues/9405))
### Fixes
- fix configuration of turning test warnings into failures with WARN_ERROR_OPTIONS ([#7761](https://github.com/dbt-labs/dbt-core/issues/7761))
- Fix conflict with newer versions of Snowplow tracker ([#8719](https://github.com/dbt-labs/dbt-core/issues/8719))
- Only create the packages-install-path / dbt_packages folder during dbt deps ([#6985](https://github.com/dbt-labs/dbt-core/issues/6985), [#9584](https://github.com/dbt-labs/dbt-core/issues/9584))
- Exclude password-like fields for considering reparse ([#9795](https://github.com/dbt-labs/dbt-core/issues/9795))
- Fixed query comments test ([#9860](https://github.com/dbt-labs/dbt-core/issues/9860))
- Begin warning people about spaces in model names ([#9397](https://github.com/dbt-labs/dbt-core/issues/9397))
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
