* Add docstrings to `contracts/graph/metrics.py` functions to document what they do
Used [dbt-labs/dbt-core#5607](https://github.com/dbt-labs/dbt-core/pull/5607)
for context on what the functions should do.
* Add typing to `reverse_dag_parsing` and update function to work on 1.6+ metrics
* Add typing to `parent_metrics` and `parent_metrics_names`
* Add typing to `base_metric_dependency` and `derived_metric_dependency` and update functions to work on 1.6+ metrics
* Simplify implementations of `basic_metric_dependency` and `derived_metric_dependnecy`
* Add typing to `ResolvedMetricReference` initialization
* Add typing to `derived_metric_dependency_graph`
* Simplify conditional controls in `ResolvedMetricReference` functions
The functions in `ResolvedMetricReference` use `manifest.metric.get(...)`
which will only return either a `Metric` or `None`, never a different
node type. Thus we don't need to check that the returned metric is
a metric.
* Don't recurse on over `depends_on` for non-derived metrics in `reverse_dag_parsing`
The function `reverse_dag_parsing` only cares about derived metrics,
that is metrics that depend on other metrics. Metrics only depend on
other metrics if they are one of the `DERIVED_METRICS` types. Thus
doing a recursive call to `reverse_dag_parsing` for non `DERIVED_METRICS`
types is unnecessary. Previously we were iterating over a metric's
`depends_on` property regardless of whether the metric was a `DERIVED_METRICS`
type. Now we only do this work if the metric is of a `DERIVED_METRICS`
type.
* Simplify `parent_metrics_names` by having it call `parent_metrics`
* Unskip `TestMetricHelperFunctions.test_derived_metric` and update fixture setup
* Add changie doc for metric helper function updates
* Get manifest in `test_derived_metric` from the parse dbt_run invocation
* Remove `Relation` a intiatlization attribute for `ResolvedMetricReference`
* Add return typing to class `__` functions of `ResolvedMetricReference`
* Move from `manifest.metrics.get` to `manifest.expect` in metric helpers
Previously with `manifest.metrics.get` we were just skipping when `None`
was returned. Getting `None` back was expected in that `parent_unique_id`s
that didn't belong to metrics should return `None` when calling
`manifest.metrics.get`, and these are fine to skip. However, there's
an edgecase where a `parent_unique_id` is supposed to be a metric, but
isn't found, thus returning `None`. How likely this edge case could
get hit, I'm not sure, but it's a possible edge case. Using `manifest.metrics.get`
it we can't actually tell if we're in the edge case or not. By moving
to `manifest.expect` we get the error handling built in, and the only
trade off is that we need to change our conditional to skip returned
nodes that aren't metrics.
* update `Number` class to handle integer values (#8306)
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* account for integer vs number on table merges
* add tests for combining number with integer.
* add unit test when nulls are added
* cant none as an Integer
* fix null tests
---------
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Dave Connors <dave.connors@fishtownanalytics.com>
* first draft of adding in table - materialized view swap
* table/view/materialized view can all replace each other
* update renameable relations to a config
* migrate relations macros from `macros/adapters/relations` to `macros/relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar
* move drop macro to drop macro file
* align the behavior of get_drop_sql and drop_relation, adopt existing default from drop_relation
* add explicit ddl for drop statements instead of inheriting the default from dbt-core
* update replace macro dependent macros to align with naming standards
* update type for mashumaro, update related test
* Improve typing of `ContextMember` functions
* Improve typing of `Var` functions
* Improve typing of `ContextMeta.__new__`
* Improve typing `BaseContext` and functions
In addition to just adding parameter typing and return typing to
`BaseContext` functions. We also declared `_context_members_` and
`_context_attrs_` as properites of `BaseContext` this was necessary
because they're being accessed in the classes functions. However,
because they were being indirectly instantiated by the metaclass
`ContextMeta`, the properties weren't actually known to exist. By
adding declaring the properties on the `BaseContext`, we let mypy
know they exist.
* Remove bare `invocations` of `@contextmember` and `@contextproperty`, and add typing to them
Previously `contextmember` and `contextproperty` were 2-in-1 decorators.
This meant they could be invoked either as `@contextmember` or
`@contextmember('some_string')`. This was fine until we wanted to return
typing to the functions. In the instance where the bare decorator was used
(i.e. no `(...)` were present) an object was expected to be returned. However
in the instance where parameters were passed on the invocation, a callable
was expected to be returned. Putting a union of both in the return type
made the invocations complain about each others' return type. To get around this
we've dropped the bare invocation as acceptable. The parenthesis are now always
required, but passing a string in them is optional.
* WIP
* WIP
* get group and enabled added
* changelog
* cleanup
* getting measure lookup working
* missed file
* get project level working
* fix last test
* add groups to config tests
* more group tests
* fix path
* clean up manifest.py
* update error message
* fix test assert
* remove extra check
* resolve conflicts in manaifest
* update manifest
* resolve conflict
* add alias
* Add compiled node properties to run_results.json
* Include compiled-node attributes in run_results.json
* Fix typo
* Bump schema version of run_results
* Fix test assertions
* Update expected run_results to reflect new attributes
* Code review changes
* Fix mypy warnings for ManifestLoader.load() (#8443)
* revert python version for docker images (#8445)
* revert python version for docker images
* add comment to not update python version, update changelog
* Bumping version to 1.7.0b1 and generate changelog
* [CT-3013] Fix parsing of `window_groupings` (#8454)
* Update semantic model parsing tests to check measure non_additive_dimension spec
* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`
* Add changie doc for `window_groupings` parsing fix
* update `Number` class to handle integer values (#8306)
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Improve docker image README (#8212)
* Improve docker image README
- Fix unnecessary/missing newline escapes
- Remove double whitespace between parameters
- 2-space indent for extra lines in image build commands
* Add changelog entry for #8212
* ADAP-814: Refactor prep for MV updates (#8459)
* apply reformatting changes only for #8449
* add logging back to get_create_materialized_view_as_sql
* changie
* swap trigger (#8463)
* update the implementation template (#8466)
* update the implementation template
* add colon
* Split tests into classes (#8474)
* add flaky decorator
* split up tests into classes
* revert update agate for int (#8478)
* updated typing and methods to meet mypy standards (#8485)
* Convert error to conditional warning for unversioned contracted model, fix msg format (#8451)
* first pass, tests need updates
* update proto defn
* fixing tests
* more test fixes
* finish fixing test file
* reformat the message
* formatting messages
* changelog
* add event to unit test
* feedback on message structure
* WIP
* fix up event to take in all fields
* fix test
* Fix ambiguous reference error for duplicate model names across packages with tests (#8488)
* Safely remove external nodes from manifest (#8495)
* [CT-2840] Improved semantic layer protocol satisfaction tests (#8456)
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified
* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols
* Add test asserting Defaults obj satisfies protocol
* Add test asserting SemanticModel with optionals specified satisfies protocol
* Split dimension protocol satisfaction tests into with and without optionals
* Simplify DSI Protocol import strategy in protocol satisfaction tests
* Add test asserting DimensionValidtyParams satisfies protocol
* Add test asserting DimensionTypeParams satisfies protocol
* Split entity protocol satisfaction tests into with and without optionals
* Split measure protocol satisfication tests and add measure aggregation params satisficaition test
* Split metric protocol satisfaction test into optional specified an unspecified
Additionally, create where_filter pytest fixture
* Improve protocol satisfaction tests for MetricTypeParams and sub protocols
Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
* Convert to using mashumaro jsonschema with acceptable performance (#8437)
* Regenerate run_results schema after merging in changes from main.
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Jaime Martínez Rincón <jaime@jamezrin.name>
Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified
* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols
* Add test asserting Defaults obj satisfies protocol
* Add test asserting SemanticModel with optionals specified satisfies protocol
* Split dimension protocol satisfaction tests into with and without optionals
* Simplify DSI Protocol import strategy in protocol satisfaction tests
* Add test asserting DimensionValidtyParams satisfies protocol
* Add test asserting DimensionTypeParams satisfies protocol
* Split entity protocol satisfaction tests into with and without optionals
* Split measure protocol satisfication tests and add measure aggregation params satisficaition test
* Split metric protocol satisfaction test into optional specified an unspecified
Additionally, create where_filter pytest fixture
* Improve protocol satisfaction tests for MetricTypeParams and sub protocols
Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
* first pass, tests need updates
* update proto defn
* fixing tests
* more test fixes
* finish fixing test file
* reformat the message
* formatting messages
* changelog
* add event to unit test
* feedback on message structure
* WIP
* fix up event to take in all fields
* fix test
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Update semantic model parsing tests to check measure non_additive_dimension spec
* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`
* Add changie doc for `window_groupings` parsing fix
* first pass
* WIP
* update issue body
* fix triggering label
* fix docs
* add better run name
* reduce complexity
* update description
* fix PR title
* point at workflow on main
* fix wording
* add label
* Update semantic model parsing test to check `create_metric = true` functionality
* Add `create_metric` boolean property to unparsed measure objects
* Begin creating metrics from measures with `create_metric = True`
* Add test ensuring partial parsing handles metrics generated from measures
* Ensure partial parsing appropriately deletes metrics generated from semantic models
* Add changie doc for addition
* Separate generated metrics from parsed metrics for partial parsing
I was doing a demo earlier today of this branch (minus this commit)
and noticed something odd. When I changes a semantic model, metrics
that should have been technically uneffected would get dropped. Basically
if I made a change to a semantic model which had metrics in the same
file, and then ran parse, those metrics defined in the same file
would get dropped. Then with no other changes, if I ran parse again
they would come back. What was happening was that parsed metrics
and generated metrics were getting tracked the same way on the file
objects for partial parsing. In 0787a7c7b6
we began dropping all metrics tracked in a file objects when changes
to semantic models were detected. Since parsed metrics and generated
metrics were being tracked together on the file object, the parsed
metrics were getting dropped as well. In this commit we begin separating
out the tracking of generated metrics and parsed metrics on the
file object, and now only drop the generated metrics when semantic
models have a detected change.
* Assert in test that semantic model partial parsing doesn't clobber regular metrics
* Replaced the FirstRunResultError and AfterFirstRunResultError events with RunResultError.
* Attempts at reasonable unit tests.
* Restore event manager after unit test.
* Support configurable delimiter for seed files, default to comma (#3990)
* Update Features-20230317-144957.yaml
* Moved "delimiter" to seed config instead of node config
* Update core/dbt/clients/agate_helper.py
Co-authored-by: Cor <jczuurmond@protonmail.com>
* Update test_contracts_graph_parsed.py
* fixed integration tests
* Added functional tests for seed files with a unique delimiter
* Added docstrings
* Added a test for an empty string configured delimiter value
* whitespace
* ran black
* updated changie entry
* Update Features-20230317-144957.yaml
---------
Co-authored-by: Cor <jczuurmond@protonmail.com>
* add param to control maxBytes for single dbt.log file
* nits
* nits
* Update core/dbt/cli/params.py
Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
---------
Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
* Add test ensuring `warn_error_options` is dictified in `invocation_args_dict` of contexts
* Add dictification specific to `warn_error_options` in `args_to_dict`
* Changie doc for serialization changes of warn_error_options
* Add test asserting that a macro with the work materializtion doesn't cause issues
* Let macro names include the word `materialization`
Previously we were checking if a macro included a materialization
based on if the macro name included the word `materialization`. However,
a macro name included the word `materialization` isn't guarnteed to
actually have a materialization, and a macro that doesn't have
`materialization` in the name isn't guaranteed to not have a materialization.
This change is to detect macros with materializations based on the
detected block type of the macro.
* Add changie doc materialization in macro detection
* Add test for checking that `_connection_exception_retry` handles `EOFError`s
* Update `_connection_exception_retry` to handle `EOFError` exceptions
* Add changie docs for `_connection_exception_retry` handling `EOFError` exceptions
- Add --target-path to dbt snapshot command. ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- dbt build selection of tests' descendants ([#7289](https://github.com/dbt-labs/dbt-core/issues/7289))
- fix groupable node partial parsing, raise DbtReferenceError at runtime for safety ([#7437](https://github.com/dbt-labs/dbt-core/issues/7437))
- Fix partial parsing of latest_version changes for downstream references ([#7369](https://github.com/dbt-labs/dbt-core/issues/7369))
- Use "add_node" to update depends_on.nodes ([#7453](https://github.com/dbt-labs/dbt-core/issues/7453))
- Fix var precedence in configs: root vars override package vars ([#6705](https://github.com/dbt-labs/dbt-core/issues/6705))
- Fix inverted `--print/--no-print` flag ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
- Back-compat for previous return type of 'collect_freshness' macro ([#7489](https://github.com/dbt-labs/dbt-core/issues/7489))
- print model version in dbt show if specified ([#7407](https://github.com/dbt-labs/dbt-core/issues/7407))
- enable dbt show for seeds ([#7273](https://github.com/dbt-labs/dbt-core/issues/7273))
- push down limit filtering to adapter ([#7390](https://github.com/dbt-labs/dbt-core/issues/7390))
- Allow missing `profiles.yml` for `dbt deps` and `dbt init` ([#7511](https://github.com/dbt-labs/dbt-core/issues/7511))
-`run_results.json` is now written after every node completes. ([#7302](https://github.com/dbt-labs/dbt-core/issues/7302))
- Do not rewrite manifest.json during 'docs serve' command ([#7553](https://github.com/dbt-labs/dbt-core/issues/7553))
- Pin protobuf to greater than 4.0.0 ([#7565](https://github.com/dbt-labs/dbt-core/issues/7565))
- inject sql header in query for show ([#7413](https://github.com/dbt-labs/dbt-core/issues/7413))
- Pin urllib3 to ~=1.0 ([#7573](https://github.com/dbt-labs/dbt-core/issues/7573))
- Throw error for duplicated versioned and unversioned models ([#7487](https://github.com/dbt-labs/dbt-core/issues/7487))
### Under the Hood
- Update docs link in ContractBreakingChangeError message ([#7366](https://github.com/dbt-labs/dbt-core/issues/7366))
- Reduce memory footprint of cached statement results. ([#7281](https://github.com/dbt-labs/dbt-core/issues/7281))
- Remove noisy parsing events: GenericTestFileParse, MacroFileParse, Note events for static model parsing ([#6671](https://github.com/dbt-labs/dbt-core/issues/6671))
- Update --help text for cache-related parameters ([#7381](https://github.com/dbt-labs/dbt-core/issues/7381))
- Small UX improvements to model versions: Support defining latest_version in unsuffixed file by default. Notify on unpinned ref when a prerelease version is available. ([#7443](https://github.com/dbt-labs/dbt-core/issues/7443))
- Added warnings for model and ref deprecations ([#7433](https://github.com/dbt-labs/dbt-core/issues/7433))
- Update drop_relation macro to allow for configuration of drop statement separately from object name ([#7625](https://github.com/dbt-labs/dbt-core/issues/7625))
- accept publications in dbt.invoke ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Fixes
- Honor `--skip-profile-setup` parameter when inside an existing project ([#7594](https://github.com/dbt-labs/dbt-core/issues/7594))
- Fix: Relative project paths weren't working with deps ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491))
- Exclude password fields from Jinja rendering. ([#7629](https://github.com/dbt-labs/dbt-core/issues/7629))
- Add --target-path to more CLI subcommands ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- Stringify flag paths for Jinja context ([#7495](https://github.com/dbt-labs/dbt-core/issues/7495))
- write run_results.json for run operation ([#7502](https://github.com/dbt-labs/dbt-core/issues/7502))
### Under the Hood
- Add ability to instantiate Flags class from dict ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607))
- Add other relation to reffable nodes ([#7550](https://github.com/dbt-labs/dbt-core/issues/7550))
- Move node patch method to schema parser patch_node_properties and refactor schema parsing ([#7430](https://github.com/dbt-labs/dbt-core/issues/7430))
- Dropped support for Python 3.7 ([#7082](https://github.com/dbt-labs/dbt-core/issues/7082))
- Switch from dbt-metrics to dbt-semantic-interfaces for MetricNode definitions ([#7500](https://github.com/dbt-labs/dbt-core/issues/7500), [#7404](https://github.com/dbt-labs/dbt-core/issues/7404))
### Features
- Add support for materialized views ([#6911](https://github.com/dbt-labs/dbt-core/issues/6911))
- Optimize template rendering for common parse scenarios ([#7449](https://github.com/dbt-labs/dbt-core/issues/7449))
- nodes in packages respect custom generate_alias_name, generate_schema_name, generate_database_name macro overrides defined in packages ([#7444](https://github.com/dbt-labs/dbt-core/issues/7444))
- Enable state for deferral to be separate from state for selectors ([#7300](https://github.com/dbt-labs/dbt-core/issues/7300))
- Validate public models are not materialized as ephemeral ([#7226](https://github.com/dbt-labs/dbt-core/issues/7226))
- Added support for parsing and serializaing semantic models ([#7499](https://github.com/dbt-labs/dbt-core/issues/7499), [#7503](https://github.com/dbt-labs/dbt-core/issues/7503))
### Fixes
- Constraint rendering fixes: wrap check expression in parentheses, foreign key 'references', support expression in all constraint types ([#7417](https://github.com/dbt-labs/dbt-core/issues/7417), [#7480](https://github.com/dbt-labs/dbt-core/issues/7480), [#7416](https://github.com/dbt-labs/dbt-core/issues/7416))
- Fix warning messages for deprecated dbt_project.yml configs ([#7424](https://github.com/dbt-labs/dbt-core/issues/7424))
- Add `%` to adapter suite test cases for `persist_docs` ([#7698](https://github.com/dbt-labs/dbt-core/issues/7698))
- Improve warnings for constraints and materialization types ([#7335](https://github.com/dbt-labs/dbt-core/issues/7335))
- Incorrect paths used for "target" and "state" directories ([#7465](https://github.com/dbt-labs/dbt-core/issues/7465))
- fix StopIteration error when publication for project not found ([#7711](https://github.com/dbt-labs/dbt-core/issues/7711))
- Using version 0 works when resolving single model ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
- Allow semantic model measure exprs to be defined with ints and bools in yaml ([#7865](https://github.com/dbt-labs/dbt-core/issues/7865))
- Update `use_discrete_percentile` and `use_approximate_percentile` to be non optional and default to `False` ([#7866](https://github.com/dbt-labs/dbt-core/issues/7866))
- This change adds new selector methods to the state selector. Namely, state:unmodified and state:old. ([#7564](https://github.com/dbt-labs/dbt-core/issues/7564))
- Support '_'-delimited fqn matching for versioned models and matching on Path.stem for path selection ([#7639](https://github.com/dbt-labs/dbt-core/issues/7639))
- Store time_spline table configuration in semantic manifest ([#7938](https://github.com/dbt-labs/dbt-core/issues/7938))
### Fixes
- Fix CTE insertion position when the model uses WITH RECURSIVE ([#7350](https://github.com/dbt-labs/dbt-core/issues/7350))
- Unified to UTC ([#7664](https://github.com/dbt-labs/dbt-core/issues/7664))
- Remove limitation on use of sqlparse 0.4.4 ([#7515](https://github.com/dbt-labs/dbt-core/issues/7515))
- Move project_root contextvar into events.contextvars ([#7937](https://github.com/dbt-labs/dbt-core/issues/7937))
- Fix typo in ModelNodeArgs ([#7991](https://github.com/dbt-labs/dbt-core/issues/7991))
- Allow on_schema_change = fail for contracted incremental models ([#7975](https://github.com/dbt-labs/dbt-core/issues/7975))
### Docs
- add note before running integration tests ([dbt-docs/#nothing](https://github.com/dbt-labs/dbt-docs/issues/nothing))
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
