* change port bind and add a unittest (#10208)
(cherry picked from commit 0c08d7a19a)
* fix unit tests
* remove ssh-key since schemas repo is public
* fix compare schemas diff
---------
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
* initial push of adding cache functions for materialized views
* update
* add cache_renamed
* remove current attempt of add cache_renamed
* add checks for add, and drop to see if relation exists in dispatch macro
* move calls from distpach macro into the default call/postgres specific versions incase we are not catching the dispatch as a layer of call or in drop case of it being a explicit return call
* split up relation into parts to build up to_relation
* move cache_dropped back to the dispatch original dispatch macro
* move cache_dropped back to the dispatch original dispatch macro
* readd removed space between macros
* move cache_added back to distpach macro
* remove extra curly brace
* revert get_relation call back get cache_renamed
* move of cache_renamed to get_rename_sql
* remove updated cache_changes and remove Relation.MaterializedView from renabmeable (takes us back to previous functionality)
* add changelog
* add simple test case, and doc string
* Fix assorted source freshness edgecases so check is run or actionable information (#9825)
* Ensure BaseRunner handles nodes without `build_path`
Some nodes, like SourceDefinition nodes, don't have a `build_path` property.
This is problematic because we take in nodes with no type checking, and
assume they have properties sometimes, like `build_path`. This was just
the case in BaseRunner's `_handle_generic_exception` and
`_handle_interal_exception` methods. Thus to stop dbt from crashing when
trying to handle an exception related to a node without a `build_path`,
we added an private method to the BaseRunner class for safely trying
to get `build_path`.
* Use keyword arguments when instantiating `Note` events in freshness.py
Previously we were passing arguments during the `Note` event instantiations
in freshness.py as positional arguments. This would cause not the desired
`Note` event to be emitted, but instead get the message
```
[Note] Don't use positional arguments when constructing logging events
```
which was our fault, not the users'. Additionally, we were passing the
level for the event in the `Note` instantiation when we needed to be
passing it to the `fire_event` method.
* Raise error when `loaded_at_field` is `None` and metadata check isn't possible
Previously if a source freshness check didn't have a `loaded_at_field` and
metadata source freshness wasn't supported by the adapter, then we'd log
a warning message and let the source freshness check continue. This was problematic
because the source freshness check couldn't actually continue and the process
would raise an error in the form
```
type object argument after ** must be a mapping, not NoneType
```
because the `freshness` variable was never getting set. This error wasn't particularly
helpful for any person running into it. So instead of letting that error
happen we now deliberately raise an error with helpful information.
* Add test which ensures bad source freshness checks raise appropriate error
This test directly tests that when a source freshness check doesn't have a
`loaded_at_field` and the adapter in use doesn't support metadata checks,
then the appropriate error message gets raised. That is, it directly tests
the change made in a162d53a8. This test indirectly tests the changes in both
7ec2f82a9 and 7b0ff3198 as the appropriate error can only be raised because
we've fixed other upstream issues via those commits.
* Add changelog entry for source freshness edgecase fixes
* Update imports of dbt.artifacts to pre-dbt.artifacts locations
Moving things to dbt.artifacts started to happen AFTER the release of
dbt-core 1.7, thus 1.7 has no concept of dbt.articacts. The changes brought
in via the previous commit, were cherry-picked from main (1.8.0 beta) and
thus reference dbt.artifacts. This caused everything to blow up, because
dbt.artifacts doesn't exist. Here we've updated the dbt.artifacts imports
to their pre-dbt.artifacts paths.
* Add tests to check that saved queries show in `dbt list`
* Update `list` task to support saved queries
This is built off of @jtcohen6 work in d6e7cda on jerco/fix-9532.
I didn't directly cherry pick because there was more work to do as
well as merge conflicts. That is to say @jtcohen6 should be credited
with some of the work.
* Update error message when iterating over nodes during list command errors
This was originally suggested by @jtcohen6 in d6e7cda of jerco/fix-9532.
This commit just makes sure the change gets included because I didn't
cherry-pick that commit into this work.
* Add changie log for saved query list support
* Stop trying to parse deleted schema files (#9722)
* Add test around deleting a YAML file containing semantic models and metrics
It was raised in https://github.com/dbt-labs/dbt-core/issues/8860 that an
error is being raised during partial parsing when files containing
metrics/semantic models are deleted. In further testing it looks like this
error specifically happens when a file containing both semantic models and
metrics is deleted. If the deleted file contains just semantic models or
metrics there seems to be no issue. The next commit should contain the fix.
* Skip deleted schema files when scheduling files during partial parsing
Waaaay back (in 7563b99) deleted schema files started being separated out
from deleted non-schema files. However ever since, when it came to scheduling
files for reparsing, we've only done so for deleted non-schema files. We even
missed this when we refactored the scheduling code in b37e5b5. This change
updates `_schedule_for_parsing` which is used by `schedule_nodes_for_parsing`
to begin skipping deleted schema files in addition to deleted non schema files.
* Update `add_to_pp_files` to ignore `deleted_schema_files`
As noted in the previous commit, we started separating out deleted
schema files from deleted non-schema files a looong time ago. However,
this whole time we've been adding `deleted_schema_files` to the list
of files to be parsed. This change corrects for that.
* Add changie doc for partial parsing KeyError fix
(cherry picked from commit deedeeb9ce)
* Empty commit to trigger github actions
---------
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* Restrict protobuf to 4.* versions (#9630)
Protobuf v5 has breaking changes. Here we are limiting the protobuf
dependency to one major version, 4, so that we don't have to patch
over handling 2 different major versions of protobuf.
(cherry picked from commit e4fe839e45)
---------
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
CVE-2024-22195 identified an issue in Jinja2 versions <= 3.1.2. As such
we've gone and changed our dependency requirement specification to be
3.1.3 or greater (but less than 4).
Note: Preivously we were using the `~=` version specifier. However due
to some issues with the `~=` we've moved to using `>=` in combination
with `<`. This gives us the same range that `~=` gave us, but avoids
a pip resolution issue when multiple packages in an environment use `~=`
for the same dependency.
(cherry picked from commit 7ea4670832)
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
* TestGenerateCatalogWithExternalNodes (#9456)
(cherry picked from commit 06e55bb93e)
* Flip logic in `packages_for_node` to remove error case
By flipping the logic from `not in` to `in` we can drop the exception
and instead default to the model runtime config when the package isn't
found. We're still trying to grok if there will be any fallout from this.
The tests all pass, but that doesn't guarantee nothing bad will happen.
---------
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* Fix full-refresh and vars for retry (#9328)
Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
(cherry picked from commit 1e4286a62d)
* pr feedback
* Update requires.py
* Move minimum DSI version to 0.4.2
We're backporting a feature "conversion metrics" to 1.7. Conversion
metrics don't exist in DSI < 0.4.2 which is problematic if we allow
for those versions. This ensures that those who are on a version of
1.7 that supports conversion metrics will also have the requisit version
of DSI.
* added ConversionTypeParams classes
* updated parser for ConversionTypeParams
* added step to populate input_measure for conversion metrics
* added tests
* added changelog
* Regenerate v11 manifest jsonschema to include conversion metrics definition
* Regenerate v11 manifest test artifact for testing version compatability
---------
Co-authored-by: Will Deng <william.deng14@gmail.com>
* Drop `all_refs=True` from jsonschema-ization build process
Passing `all_refs=True` makes it so that Everything is a ref, even
the top level schema. In jsonschema land, this essentially makes the
produced artifact not a full schema, but a fractal object to be included
in a schema. Thus when `$id` is passed in, jsonschema tools blow up
because `$id` is for identifying a schema, which we explicitly weren't
creating. The alternative was to drop the inclusion of `$id`. Howver, we're
intending to create a schema, and having an `$id` is recommended best
practice. Additionally since we were intending to create a schema,
not a fractal, it seemed best to create to full schema.
* Explicity produce jsonschemas using DRAFT_2020_12 dialect
Previously were were implicitly using the `DRAFT_2020_12` dialect through
mashumaro. It felt wise to begin explicitly specifying this. First, it
is closest in available mashumaro provided dialects to what we produced
pre 1.7. Secondly, if mashumaro changes its default for whatever reason
(say a new dialect is added, and mashumaro moves to that), we don't want
to automatically inherit that.
* Begin including schema dialect specification in produced jsonschema
In jsonschema's documentation they state
> It's not always easy to tell which draft a JSON Schema is using.
> You can use the $schema keyword to declare which version of the JSON Schema specification the schema is written to.
> It's generally good practice to include it, though it is not required.
and
> For brevity, the $schema keyword isn't included in most of the examples in this book, but it should always be used in the real world.
Basically, to know how to parse a schema, it's important to include what
schema dialect is being used for the schema specification. The change in
this commit ensures we include that information.
* Add change documentation for jsonschema schema production fix
* Regenerate dbt jsonschemas with fixed mashumaro jsonschema production process
Specifically we regenerated
* catalog v1
* manifest v11
* run-results v5
* sources v3
using the command `scripts/collect-artifact-schema.py --path schemas`
* During node execution, also treat SystemExit as an interrupt. (#8994)
IDE worker process raises SystemExit in multiple scenarios, including user abort of a command.
(cherry picked from commit 931b2dbe40)
* Add test asserting GraphRunnableTasks attempt to cancel connections on SystemExit (#9101)
* Add test asserting GraphRunnableTasks attempt to cancel connections on SystemExit
* Add test asserting GraphRunnableTasks attempt to cancel connections on KeyboardInterrupt
* Add test asserting GraphRunnableNode doesn't try to cancel connections on generic Exception
---------
Co-authored-by: Ben Mosher <ben.mosher@dbtlabs.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
* Add test asserting `SavedQuery` configs can be set from `dbt_project.yml`
* Allow extraneous properties in Export configs
This brings the Export config object more in line with how other config
objects are specified in the unparsed definition. It allows for specifying
of extra configs, although they won't get propagate to the final config.
* Add `ExportConfig` options to `SavedQueryConfig` options
This allows for specifying `ExportConfig` options at the `SavedQueryConfig` level.
This also therefore allows these options to be specified in the dbt_project.yml
config. The plan in the follow up commit is to merge the `SavedQueryConfig` options
into all configs of `Exports` belonging to the saved query.
There are a couple caveots to call out:
1. We've used `schema` instead of `schema_name` on the `SavedQueryConfig` despite
it being called `schema_name` on the `ExportConfig`. This is because need `schema_name`
to be the name of the property on the `ExportConfig`, but `schema` is the user facing
specification.
2. We didn't add the `ExportConfig` `alias` property to the `SavedQueryConfig` This
is because `alias` will always be specific to a single export, and thus it doesn't
make sense to allow defining it on the `SavedQueryConfig` to then apply to all
`Exports` belonging to the `SavedQuery`
* Begin inheriting configs from saved query config, and transitively from project config
Export configs will now inherit from saved query configs, with a preference
for export config specifications. That is to say an export config will inherity
a config attr from the saved query config only if a value hasn't been supplied
on the export config directly. Additionally because the saved query config has
a similar relationship with the project config, exports configs can inherit
from the project config (again with a preference for export config specifications).
* Correct conditional in export config building for map schema to schema_name
I somehow wrote a really weird, but also valid, conditional statement. Previously
the conditional was
```
if combined.get("schema") is not combined.get("schema_name") is None:
```
which basically checked whether `schema` was a boolean that didn't match
the boolean of whether `schema_name` was None. This would pretty much
always evaluate to True because `schema` should be a string or none, not
a bool, and thus would never match the right hand side. Crazy. It has now
been fixed to do the thing we want to it to do. If `schema` isn't `None`,
and `schema_name` is `None`, then set `schema_name` to have the value of
`schema`.
* Update parameter names in `_get_export_config` to be more verbose
(cherry picked from commit c2f7d75e9e)
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
* changelog
* write test case demonstrating the issue
* update catalog query to reflect materialized views
(cherry picked from commit bb21403c9e)
Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
* Fix issues around new get_catalog_by_relations macro (#8856)
* Fix issues around new get_catalog_by_relations macro
* Add changelog entry
* Fix unit test.
* Additional unit testing
* Fix cased comparison in catalog-retrieval function (#8940)
* Fix cased comparison in catalog-retrieval function.
* Fix cased comparison in catalog-retrieval function.
* add test
* fix test
* first pass with constraint error
* add back column checks for temp tables
* changelog
* Update .changes/unreleased/Fixes-20231024-145504.yaml
(cherry picked from commit 98310b6612)
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* changie doc for DSI 0.3.0 upgrade
* Gracefully handle v10 metric filters
* Fix iteration over metrics in `upgrade_v10_metric_filters`
* Update previous manifest version test fixtures to have more expressive metrics
* Regenerate the test v10 manifest artifact using the more expressive metrics from 904cc1ef
To do this I cherry-picked 904cc1ef onto my local 1.6.latest branch,
had the test regenerate the test v10 manifest artifact, and then over
wrote the test v10 manifest artifact on this branch (cherry-picking it
across the branches didn't work, had to copy paste :grimmace:)
* Regenerate test v11 manifest artifact using the fixture changes in 904cc1ef
* Update `upgrade_v10_metric_filters` to handled disabled metrics
Regenerating the v10 and v11 test manifest artifacts uncovered an
issue wherein we weren't handling disabled metrics that need to
get upgraded. This commit fixes that. Additionally, the
`upgrade_v10_metric_filters` was getting a bit unwieldy, so I broke
extracted the abstracted sub functions.
* Fix `test_backwards_compatible_versions` test
When we regenerated the v10 test manifest artifact, it started having
the `metricflow_time_sine` model, and it didn't previously. This caused
`test_backwards_compatible_versions` to start failing because it was
no longer identified as having modified state for v10. The test has
been altered accordingly
* Bump to dbt-semantic-interfaces 0.3.0b1
* Update import path of `WhereFilterParser` from `dbt-semantic-interfaces`
In 0.3.x of `dbt-semantic-intefaces` the location of the WhereFilterParser
moved to be grouped in with a bunch of new adjacent code. As such,
we needed to correct our import path of it.
* Create basic `SavedQuery` node type based on `SavedQuery` protocol from DSI
* Add ability to add SavedQueries to the manifest
* Define unparsed SavedQuery node
* Begin parsing saved_query objects to manifest
* Skip jinja rendering of `SavedQuery.where` property
* Begin propagating `SavedQueries` on the manifest to the semantic manifest
* Add tests for basic saved query parsing
* Add custom pluralization handling of SavedQuery node type
* Add a config subclass to SavedQuery node
* Move the SavedQuery node to nodes.py
Unfortunately things are a bit too intertwined currently for SavedQuery
to be in it's own file. We need to add the SavedQuery node to the
GraphMemberNode, unfortunately with SavedQuery in it's own file,
importing it would have caused a circular dependency. We'll need
to separately come in and split things up as a cleanup portion of
work.
* Add basic plumbing of saved query configs to projects
* Add basic lookup utility for saved queries, SavedQueryLookup
* Handle disabled SavedQuery nodes in parsing and lookups
* Add SavedQuery nodes to grouping process
Our grouping logic seems to be in a weird spot. It seems liek we're
moving to setting the `group` for a node in the node's `config` however,
all of the logic around grouping is still focused on the top level `group`
property on a nodes. To get group stuff plumbed I've thus added `group`
as a top level property of the `SavedQuery` node, and populated it from
the config group value.
* Plumb through saved query in a lot more places
I don't like making scatter shot commits like this. However, a lot
of this commit was written ~4am, soooo yea. Things were broken, I wanted
things to be unbroken. I mostly searched for `semantic_models` and added
the equivalent necessary `saved_queries`. Some stuff is in support of
writing out the manifest, some stuff helps with node selection, it's a
lot of miscelaneous stuff that I don't fully understand.
* Add `depends_on` to `SavedQuery` nodes and populate from `metrics` property
* Add partial parsing support to SavedQuery nodes
* Add `docs` support for SavedQuery descriptions
* Support selctor methods for SavedQuery nodes
* Add `refs` property to SavedQuery node
We don't actually append anything to `refs` for SavedQuery nodes currently.
I'm not sure if anything needs to be appended to them. Regardless, we
access the `refs` property throughout the codebase while iterating over
nodes. It seems wise to support this attribute as to not accidently blow
something up with it not existing.
* Support `saved_queries` when upgrading from manifests <= v10 (and regenerate v11)
* Add changie doc for saved query node support
* Pin to dbt-semantic-interfaces 0.3.0b1 for saved query work
We're gonna release DSI 0.3.0, and if this PR automatically pulls that
in things will break. But the things that need fixing should be handled
separately from this PR. After releasing DSI 0.3.0 I'm going to create
a branch off/ontop of this one, and open a stacked PR with the associated
changes.
* Bump supported DSI version to 0.3.x
* Switch metric filters and saved query where to use ne WhereFilterIntersection
* Update schema yaml readers to create WhereFilterInterfaces
* Expand metric filters and saved query where property to handle both str and list of strs
* Update tests which were broken by where filter changes
* Regeneate v11 manifest
* Fixup: Update `SavedQueryLookup.perform_lookup` to operate on saved queries
I missed this when I was copy and pasting 🤦
* Add support for getting freshness from DBMS metadata
* Add changelog entry
* Add simple test case
* Change parsing error to warning and add new event type for warning
* Code review simplification of capability dict.
* Revisions to the capability mechanism per review
* Move utility function.
* Reduce try/except scope
* Clean up imports.
* Simplify typing per review
* Unit test fix
* add `store_failures_as` parameter to TestConfig, catch strategy parameter in test materialization
* create test results as views
* updated test expected values for new config option
* break up tests into reusable tests and adapter specific configuration, update test to check for relation type and confirm views update
* move test configuration into base test class
* allow `store_failures_as` to drive whether failures are stored
* update expected test config dicts to include the new default value for store_failures_as
* Add `store_failures_as` config for generic tests
* cover --store-failures on CLI gap
* add generic tests test case for store_failures_as
* update object names for generic test case tests for store_failures_as
* remove unique generic test, it was not testing `store_failures_as`
* pull generic run and assertion into base test class to turn tests into quasi-parameterized tests
* add ephemeral option for store_failures_as, as a way to easily turn off store_failures at the model level
* add compilation error for invalid setting of store_failures_as
---------
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* Explanation of Parsing vs. Compilation vs. Runtime
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Apply suggestions from code review
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
* Fix a couple markdown rendering issues
* Move to the "explain it like im 64" folder
When ELI5 just isnt detailed enough.
* Disambiguate Python references
Disambiguate Python references and delineate SQL models ("Jinja-SQL") from Python models ("dbt-py")
---------
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
* Add semantic model test to `test_contracts_graph_parsed.py`
The tests in `test_contracts_graph_parsed.py` are meant to ensure
that we can go from objects to dictionaries and back without any
changes. We've had a desire to simplify these tests. Most tests in
this file have three to four fixtures, this test only has one. What
a test of this format ensures is that parsing a SemanticModel from
a dictionary doesn't add/drop any keys from the dictionary and that
when going back to the dictionary no keys are dropped. This style of
test will still break whenever the semantic model (or sub objects)
change. However now when that happens, only one fixture will have to
be updated (whereas previously we had to update 3-4 fixtures).
* Begin using hypothesis package for symmetry testing
Hypothesis is a python package for doing property testing. The `@given`
parameterizes a test, with it generating the arguements it has following
`strategies`. The main strategies we use is `builds` this takes in a callable
passes any sub strategies for named arguements, and will try to infer any
other arguments if the callable is typed. I found that even though the
test was run many many times, some of the `SemanticModel` properties
weren't being changed. For instance `dimensions`, `entities`, and `measures`
were always empty lists. Because of this I defined sub strategies for
some attributes of `SemanticModel`s.
* Update unittest readme to have details on test_contracts_graph_parsed methodology
* Include option to generate static index.html
* Added changie
* Using DBT's system load / write file methods for better cross platform
support
* Updated docs tests with dbt.client.systems calls for file reading
* Writing out static_index.html as binary file to prevent line-ending
conversions on Windows. (similar behaviour as index.html)
* Add performance metrics to the CommandCompleted event.
* Add changelog entry.
* Add flag for controling the log level of ResourceReport.
* Update changelog entry to reflect changes
* Remove outdated attributes
* Work around missing resource module on windows
* Fix corner case where flags are not set
* Add new get_catalog_relations macro, allowing dbt to specify which relations in a schema the adapter should return data about
* Implement postgres adapter support for relation filtering on catalog queries
* Code review changes adding feature flag for catalog-by-relation-list support
* Use profile specified in --profile with dbt init (#7450)
* Use profile specified in --profile with dbt init
* Update .changes/unreleased/Fixes-20230424-161642.yaml
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Refactor run() method into functions, replace exit() calls with exceptions
* Update help text for profile option
---------
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* add TestLargeEphemeralCompilation (#8376)
* Fix a couple of issues in the postgres implementation of get_catalog_relations
* Add relation count limit at which to fall back to batch retrieval
* Better feature detection mechanism for adapters.
* Code review changes to get_catalog_relations and adapter feature checking
* Add changelog entry
---------
Co-authored-by: ezraerb <ezraerb@alum.mit.edu>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
* Add `date_spine` macro (and macros it depends on) from dbt-utils to core
The macros added are
- date_spine
- get_intervals_between
- generate_series
- get_powers_of_two
We're adding these to core because they are becoming more prevalently used
with the increase usage in the semantic layer. Basically if you are
using the semantic layer currently, then it is almost a requirement
to use dbt-utils, which is undesireable given the SL is supported directly
in core. The primary focus of this was to just add `date_spine`. However,
because `date_spine` depends on other macros, these other macros were
also moved.
* Add adapter tests for `get_powers_of_two` macro
* Add adapter tests for `generate_series` macro
* Add adapter tests for `get_intervals_between` macro
* Add adapter tests for `date_spine` macro
* Improve test fixture for `date_spine` macro to work with multiple adapters
* Cast to types to date in fixture_date_spine when targeting redshift
* Improve test fixture for `get_intervals_between` macro to work with multiple adapters
* changie doc for adding date_spine macro
* Include 'join_to_timespine` and `fill_nulls_with` in metric fixture
* Support `join_to_timespine` and `fill_nulls_with` properties on measure inputs to metrics
* Assert new `fill_nulls_with` and `join_to_timespine` properties don't break associated DSI protocol
* Add doc for metric null coalescing improvements
* Fix unit test for unparsed metric objects
The `assert_symmetric` function asserts that dictionaries are mostly
equivalent. I say mostly equivalent because it drops keys that are
`None`. The issue is that that `join_to_timespine` gets defaulted
to `False`, so we have to specify it in the `get_ok_dict` so that
they match.
* allow multioption to be quoted
* changelog
* fix test
* remove list format
* fix tests
* fix list object
* review arg change
* fix quotes
* Update .changes/unreleased/Features-20230918-150855.yaml
* add types
* convert list to set in test
* make mypy happy
* mroe mypy happiness
* more mypy happiness
* last mypy change
* add node to test
* Extend use of type annotations in the events module.
* Add return type of None to more __init__ definitions.
* Still more type annotations adding -> None to __init__
* Tweak per review
* Use profile specified in --profile with dbt init
* Update .changes/unreleased/Fixes-20230424-161642.yaml
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Refactor run() method into functions, replace exit() calls with exceptions
* Update help text for profile option
---------
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* move config changes into alter.sql in alignment with other adapters
* move shared relations macros to relations root
* move single models files to models root
* add table to replace
* move create file into relation directory
* implement replace for postgres
* move column specific macros into column directory
* add unit test for can_be_replaced
* update renameable_relations and replaceable_relations to frozensets to set defaults
* fixed tests for new defaults
* Add docstrings to `contracts/graph/metrics.py` functions to document what they do
Used [dbt-labs/dbt-core#5607](https://github.com/dbt-labs/dbt-core/pull/5607)
for context on what the functions should do.
* Add typing to `reverse_dag_parsing` and update function to work on 1.6+ metrics
* Add typing to `parent_metrics` and `parent_metrics_names`
* Add typing to `base_metric_dependency` and `derived_metric_dependency` and update functions to work on 1.6+ metrics
* Simplify implementations of `basic_metric_dependency` and `derived_metric_dependnecy`
* Add typing to `ResolvedMetricReference` initialization
* Add typing to `derived_metric_dependency_graph`
* Simplify conditional controls in `ResolvedMetricReference` functions
The functions in `ResolvedMetricReference` use `manifest.metric.get(...)`
which will only return either a `Metric` or `None`, never a different
node type. Thus we don't need to check that the returned metric is
a metric.
* Don't recurse on over `depends_on` for non-derived metrics in `reverse_dag_parsing`
The function `reverse_dag_parsing` only cares about derived metrics,
that is metrics that depend on other metrics. Metrics only depend on
other metrics if they are one of the `DERIVED_METRICS` types. Thus
doing a recursive call to `reverse_dag_parsing` for non `DERIVED_METRICS`
types is unnecessary. Previously we were iterating over a metric's
`depends_on` property regardless of whether the metric was a `DERIVED_METRICS`
type. Now we only do this work if the metric is of a `DERIVED_METRICS`
type.
* Simplify `parent_metrics_names` by having it call `parent_metrics`
* Unskip `TestMetricHelperFunctions.test_derived_metric` and update fixture setup
* Add changie doc for metric helper function updates
* Get manifest in `test_derived_metric` from the parse dbt_run invocation
* Remove `Relation` a intiatlization attribute for `ResolvedMetricReference`
* Add return typing to class `__` functions of `ResolvedMetricReference`
* Move from `manifest.metrics.get` to `manifest.expect` in metric helpers
Previously with `manifest.metrics.get` we were just skipping when `None`
was returned. Getting `None` back was expected in that `parent_unique_id`s
that didn't belong to metrics should return `None` when calling
`manifest.metrics.get`, and these are fine to skip. However, there's
an edgecase where a `parent_unique_id` is supposed to be a metric, but
isn't found, thus returning `None`. How likely this edge case could
get hit, I'm not sure, but it's a possible edge case. Using `manifest.metrics.get`
it we can't actually tell if we're in the edge case or not. By moving
to `manifest.expect` we get the error handling built in, and the only
trade off is that we need to change our conditional to skip returned
nodes that aren't metrics.
* update `Number` class to handle integer values (#8306)
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* account for integer vs number on table merges
* add tests for combining number with integer.
* add unit test when nulls are added
* cant none as an Integer
* fix null tests
---------
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Dave Connors <dave.connors@fishtownanalytics.com>
* first draft of adding in table - materialized view swap
* table/view/materialized view can all replace each other
* update renameable relations to a config
* migrate relations macros from `macros/adapters/relations` to `macros/relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar
* move drop macro to drop macro file
* align the behavior of get_drop_sql and drop_relation, adopt existing default from drop_relation
* add explicit ddl for drop statements instead of inheriting the default from dbt-core
* update replace macro dependent macros to align with naming standards
* update type for mashumaro, update related test
* Improve typing of `ContextMember` functions
* Improve typing of `Var` functions
* Improve typing of `ContextMeta.__new__`
* Improve typing `BaseContext` and functions
In addition to just adding parameter typing and return typing to
`BaseContext` functions. We also declared `_context_members_` and
`_context_attrs_` as properites of `BaseContext` this was necessary
because they're being accessed in the classes functions. However,
because they were being indirectly instantiated by the metaclass
`ContextMeta`, the properties weren't actually known to exist. By
adding declaring the properties on the `BaseContext`, we let mypy
know they exist.
* Remove bare `invocations` of `@contextmember` and `@contextproperty`, and add typing to them
Previously `contextmember` and `contextproperty` were 2-in-1 decorators.
This meant they could be invoked either as `@contextmember` or
`@contextmember('some_string')`. This was fine until we wanted to return
typing to the functions. In the instance where the bare decorator was used
(i.e. no `(...)` were present) an object was expected to be returned. However
in the instance where parameters were passed on the invocation, a callable
was expected to be returned. Putting a union of both in the return type
made the invocations complain about each others' return type. To get around this
we've dropped the bare invocation as acceptable. The parenthesis are now always
required, but passing a string in them is optional.
* WIP
* WIP
* get group and enabled added
* changelog
* cleanup
* getting measure lookup working
* missed file
* get project level working
* fix last test
* add groups to config tests
* more group tests
* fix path
* clean up manifest.py
* update error message
* fix test assert
* remove extra check
* resolve conflicts in manaifest
* update manifest
* resolve conflict
* add alias
* Add compiled node properties to run_results.json
* Include compiled-node attributes in run_results.json
* Fix typo
* Bump schema version of run_results
* Fix test assertions
* Update expected run_results to reflect new attributes
* Code review changes
* Fix mypy warnings for ManifestLoader.load() (#8443)
* revert python version for docker images (#8445)
* revert python version for docker images
* add comment to not update python version, update changelog
* Bumping version to 1.7.0b1 and generate changelog
* [CT-3013] Fix parsing of `window_groupings` (#8454)
* Update semantic model parsing tests to check measure non_additive_dimension spec
* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`
* Add changie doc for `window_groupings` parsing fix
* update `Number` class to handle integer values (#8306)
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Improve docker image README (#8212)
* Improve docker image README
- Fix unnecessary/missing newline escapes
- Remove double whitespace between parameters
- 2-space indent for extra lines in image build commands
* Add changelog entry for #8212
* ADAP-814: Refactor prep for MV updates (#8459)
* apply reformatting changes only for #8449
* add logging back to get_create_materialized_view_as_sql
* changie
* swap trigger (#8463)
* update the implementation template (#8466)
* update the implementation template
* add colon
* Split tests into classes (#8474)
* add flaky decorator
* split up tests into classes
* revert update agate for int (#8478)
* updated typing and methods to meet mypy standards (#8485)
* Convert error to conditional warning for unversioned contracted model, fix msg format (#8451)
* first pass, tests need updates
* update proto defn
* fixing tests
* more test fixes
* finish fixing test file
* reformat the message
* formatting messages
* changelog
* add event to unit test
* feedback on message structure
* WIP
* fix up event to take in all fields
* fix test
* Fix ambiguous reference error for duplicate model names across packages with tests (#8488)
* Safely remove external nodes from manifest (#8495)
* [CT-2840] Improved semantic layer protocol satisfaction tests (#8456)
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified
* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols
* Add test asserting Defaults obj satisfies protocol
* Add test asserting SemanticModel with optionals specified satisfies protocol
* Split dimension protocol satisfaction tests into with and without optionals
* Simplify DSI Protocol import strategy in protocol satisfaction tests
* Add test asserting DimensionValidtyParams satisfies protocol
* Add test asserting DimensionTypeParams satisfies protocol
* Split entity protocol satisfaction tests into with and without optionals
* Split measure protocol satisfication tests and add measure aggregation params satisficaition test
* Split metric protocol satisfaction test into optional specified an unspecified
Additionally, create where_filter pytest fixture
* Improve protocol satisfaction tests for MetricTypeParams and sub protocols
Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
* Convert to using mashumaro jsonschema with acceptable performance (#8437)
* Regenerate run_results schema after merging in changes from main.
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Jaime Martínez Rincón <jaime@jamezrin.name>
Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified
* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols
* Add test asserting Defaults obj satisfies protocol
* Add test asserting SemanticModel with optionals specified satisfies protocol
* Split dimension protocol satisfaction tests into with and without optionals
* Simplify DSI Protocol import strategy in protocol satisfaction tests
* Add test asserting DimensionValidtyParams satisfies protocol
* Add test asserting DimensionTypeParams satisfies protocol
* Split entity protocol satisfaction tests into with and without optionals
* Split measure protocol satisfication tests and add measure aggregation params satisficaition test
* Split metric protocol satisfaction test into optional specified an unspecified
Additionally, create where_filter pytest fixture
* Improve protocol satisfaction tests for MetricTypeParams and sub protocols
Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
* first pass, tests need updates
* update proto defn
* fixing tests
* more test fixes
* finish fixing test file
* reformat the message
* formatting messages
* changelog
* add event to unit test
* feedback on message structure
* WIP
* fix up event to take in all fields
* fix test
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Update semantic model parsing tests to check measure non_additive_dimension spec
* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`
* Add changie doc for `window_groupings` parsing fix
* first pass
* WIP
* update issue body
* fix triggering label
* fix docs
* add better run name
* reduce complexity
* update description
* fix PR title
* point at workflow on main
* fix wording
* add label
* Update semantic model parsing test to check `create_metric = true` functionality
* Add `create_metric` boolean property to unparsed measure objects
* Begin creating metrics from measures with `create_metric = True`
* Add test ensuring partial parsing handles metrics generated from measures
* Ensure partial parsing appropriately deletes metrics generated from semantic models
* Add changie doc for addition
* Separate generated metrics from parsed metrics for partial parsing
I was doing a demo earlier today of this branch (minus this commit)
and noticed something odd. When I changes a semantic model, metrics
that should have been technically uneffected would get dropped. Basically
if I made a change to a semantic model which had metrics in the same
file, and then ran parse, those metrics defined in the same file
would get dropped. Then with no other changes, if I ran parse again
they would come back. What was happening was that parsed metrics
and generated metrics were getting tracked the same way on the file
objects for partial parsing. In 0787a7c7b67b10a55b0d7727eeb744df831d503c
we began dropping all metrics tracked in a file objects when changes
to semantic models were detected. Since parsed metrics and generated
metrics were being tracked together on the file object, the parsed
metrics were getting dropped as well. In this commit we begin separating
out the tracking of generated metrics and parsed metrics on the
file object, and now only drop the generated metrics when semantic
models have a detected change.
* Assert in test that semantic model partial parsing doesn't clobber regular metrics
* Replaced the FirstRunResultError and AfterFirstRunResultError events with RunResultError.
* Attempts at reasonable unit tests.
* Restore event manager after unit test.
* Support configurable delimiter for seed files, default to comma (#3990)
* Update Features-20230317-144957.yaml
* Moved "delimiter" to seed config instead of node config
* Update core/dbt/clients/agate_helper.py
Co-authored-by: Cor <jczuurmond@protonmail.com>
* Update test_contracts_graph_parsed.py
* fixed integration tests
* Added functional tests for seed files with a unique delimiter
* Added docstrings
* Added a test for an empty string configured delimiter value
* whitespace
* ran black
* updated changie entry
* Update Features-20230317-144957.yaml
---------
Co-authored-by: Cor <jczuurmond@protonmail.com>
* add param to control maxBytes for single dbt.log file
* nits
* nits
* Update core/dbt/cli/params.py
Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
---------
Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
* Add test ensuring `warn_error_options` is dictified in `invocation_args_dict` of contexts
* Add dictification specific to `warn_error_options` in `args_to_dict`
* Changie doc for serialization changes of warn_error_options
* Add test asserting that a macro with the work materializtion doesn't cause issues
* Let macro names include the word `materialization`
Previously we were checking if a macro included a materialization
based on if the macro name included the word `materialization`. However,
a macro name included the word `materialization` isn't guarnteed to
actually have a materialization, and a macro that doesn't have
`materialization` in the name isn't guaranteed to not have a materialization.
This change is to detect macros with materializations based on the
detected block type of the macro.
* Add changie doc materialization in macro detection
* Add test for checking that `_connection_exception_retry` handles `EOFError`s
* Update `_connection_exception_retry` to handle `EOFError` exceptions
* Add changie docs for `_connection_exception_retry` handling `EOFError` exceptions
* applied new integration tests to existing framework
* applied new integration tests to existing framework
* generalized tests for reusability in adapters; fixed drop index issue
* generalized tests for reusability in adapters; fixed drop index issue
* removed unnecessary overrides in tests
* adjusted import to allow for usage in adapters
* adjusted import to allow for usage in adapters
* removed fixture artifact
* generalized the materialized view fixture which will need to be specific to the adapter
* unskipped tests in the test runner package
* corrected test condition
* corrected test condition
* added missing initial build for the relation type swap tests
* add env vars for datadog ci visibility
* modify pytest command for tracing
* fix posargs
* move env vars to job that needs them
* add test repeater to DD
* swap flags
* Bump version support for `dbt-semantic-interfaces` to `~=0.1.0rc1`
* Add tests for asserting WhereFilter satisfies protocol
* Add `call_parameter_sets` to `WhereFilter` class to satisfy protocol
* Changie doc for moving to DSI 0.1.0rc1
* [CT-2822] Fix `NonAdditiveDimension` Implementation (#8089)
* Add test to ensure `NonAdditiveDimension` implementation satisfies protocol
* Fix typo in `NonAdditiveDimension`: `window_grouples` -> `window_groupings`
* Add changie doc for typo fix in NonAdditiveDimension
* Add metrics from metric type params to a metric's depends_on
* Add Lookup utility for finding `SemanticModel`s by measure names
* Add the `SemanticModel` of a `Metric`'s measure property to the `Metric`'s `depends_on`
* Add `SemanticModelConfig` to `SemanticModel`
Some tests were failing due to `Metric`'s referencing `SemanticModels`.
Specifically there was a check to see if a referenced node was disabled,
and because `SemanticModel`'s didn't have a `config` holding the `enabled`
boolean attr, core would blow up.
* Checkpoint on test fixing
* Correct metricflow_time_spine_sql in test fixtures
* Add check for `SemanticModel` nodes in `Linker.link_node`
Now that `Metrics` depend on `SemanticModels` and `SemanticModels`
have their own dependencies on `Models` they need to be checked for
in the `Linker.link_node`. I forget the details but things blow up
without it. Basically it adds the SemanticModels to the dependency
graph.
* Fix artifacts/test_previous_version_state.py tests
* fix access/test_access.py tests
* Fix function metric tests
* Fix functional partial_parsing tests
* Add time dimension to semantic model in exposures fixture
* Bump DSI version to a minimum of 0.1.0dev10
DSI 0.1.0dev10 fixes an incoherence issue in DSI around `agg_time_dimension`
setting. This incoherence was that `measure.agg_time_dimension` was being
required, even though it was no longer supposed to be a required attribute
(it's specificially typed as optional in the protocol). This was causing
a handful of tests to fail because the `semantic_model.defaults.agg_time_dimension`
value wasn't being respected. Pulling in the fix from DSI 0.1.0dev10 fixes
the issue.
Interestingly after bumping the DSI version, the integration tests were
still failing. If I ran the tests individually they passed though. To get
`make integration` to run properly I ended up having to clear my `.tox`
cache, as it seems some outdated state was being persisted.
* Add test specifically for checking the `depends_on` of `Metric` nodes
* Re-enable test asserting calling metric nodes in models
* Migrate `checked_agg_time_dimension` to `checked_agg_time_dimension_for_measure`
DSI 0.1.0dev10 moved `checked_agg_time_dimension` from the `Measure`
protocol to the `SemanticModel` protocol as `checked_agg_time_dimension_for_measure`.
This finishes a change where for a given measure either the `Measure.agg_time_dimension`
or the measure's parent `SemanticModel.defaults.agg_time_dimension` needs to be
set, instead of always require the measure's `Measure.agg_time_dimension`.
* Add changie doc for populating metric
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
The original implementation of validate_sql was called dry_run,
but in the rename the test classes and much of their associated
documentation still retained the old naming.
This is mainly cosmetic, but since these test classes will be
imported into adapter repositories we should fix this now before
the wrong name proliferates.
* Add dry_run method to base adapter with implementation for SQLAdapters
resolves#7839
In the CLI integration, MetricFlow will issue dry run queries as
part of its warehouse-level validation of the semantic manifest,
including all semantic model and metric definitions.
In most cases, issuing an `explain` query is adequate, however,
BigQuery does not support the `explain` keyword and so we cannot
simply pre-pend `explain` to our input queries and expect the
correct behavior across all contexts.
This commit adds a dry_run() method to the BaseAdapter which mirrors
the execute() method in that it simply delegates to the ConnectionManager.
It also adds a working implementation to the SQLConnectionManager and
includes a few test cases for adapter maintainers to try out on their own.
The current implementation should work out of the box with most
of our adapters. BigQuery will require us to implement the dry_run
method on the BigQueryConnectionManager, and community-maintained
adapters can opt in by enabling the test and ensuring their own
implementations work as expected.
Note - we decided to make these concrete methods that throw runtime
exceptions for direct descendants of BaseAdapter in order to avoid
forcing community adapter maintainers to implement a method that does
not currently have any use cases in dbt proper.
* Switch dry_run implementation to be macro-based
The common pattern for engine-specific SQL statement construction
in dbt is to provide a default macro which can then be overridden
on a per-adapter basis by either adapter maintainers or end users.
The advantage of this is users can take advantage of alternative
SQL syntax for performance or other reasons, or even to enable
local usage if an engine relies on a non-standard expression and
the adapter maintainer has not updated the package.
Although there are some risks here they are minimal, and the benefit
of added expressiveness and consistency with other similar constructs
is clear, so we adopt this approach here.
* Improve error message for InvalidConnectionError in test_invalid_dry_run.
* Rename dry_run to validate_sql
The validate_sql name has less chance of colliding with dbt's
command nomenclature, both now and in some future where we have
dry-run operations.
* Rename macro and test files to validate_sql
* Fix changelog entry
* add permissions
* replace db setup
* try with bash instead of just pytest flags
* fix test command
* remove spaces
* remove force-flaky flag
* add starting vlaues
* add mac and windows postgres isntall
* define use bash
* fix typo
* update output report
* tweak last if condition
* clarify failures/successful runs
* print running success and failure tally
* just output pytest instead of capturing it
* set shell to not exit immediately on exit code
* add formatting around results for easier scanning
* more output formatting
* add matrix to unlock parallel runners
* increase to ten batches
* update debug
* add comment
* clean up comments
* Remove `create_metric` as a public facing `SemanticModel.Measure` property
We want to add `create_metric`. The `create_metric` property will be
incredibly useful. However, at this time it is not hooked up, and we don't
have time to hook it up before the code freeze for 1.6.0rc of core. As
it doesn't do anything, we shouldn't allow people to specify it, because
it won't do what one would expect. We plan on making the implementation
of `create_metric` a priority for 1.7 of core
* Changie doc for the removal of create_metric property
* add negative test case
* changie
* missed a comma
* Update changelog entry
* Add a negative number (rather than subtract a positive number)
---------
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* Fix accidental propagation of log messages to root logger.
* Add changelog entry
* Fixed an issue which blocked debug logging to stdout with --log-level debug, unless --debug was also used.
* Use dbt-semantic-interface validations on semantic models and metrics defined in Core.
* Remove empty test, since semantic models don't generate any validation warnings.
* Add changelog entry.
* Temporarily remove requirement that there must be semantic models definied in order to define metrics
* add interface changes section to the PR template
* update entire template
* split up choices for tests and interfaces
* minor formatting change
* add line breaks
* actually put in line breaks
* revert split choices in checklist
* add line breaks to top
* move docs link
* typo
* ct-2551: adds old and unmodified state selection methods
* ct-2551: update check_unmodified_content to simplify
* add unit and integration tests for unmodified and old
* add changelog entry
* ct-2551: reformatting of contingent adapter assignment list
* UnifiedToUTC
* Check proximity of dbt_valid_to and deleted time
* update the message to print if the assertion fails
* add CHANGELOG entries
* test only if naive
* Added comments about naive and aware
* Generalize comparison of datetimes that are "close enough"
---------
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* Fix tests fixtures which were using measures for metric numerator/denominators
In our previous upgrade to DSI dev7, numerators and denominators for
metrics switched from being `MetricInputMeasure`s to `MetricInput`s.
I.e. metric numerators and denominators should references other metrics,
not semantic model measures. However, at that time, we weren't actually
doing anything with numerators and denominators in core, so no issue
got raised. The changes we are about to make though are going to surface
these issues..
* Add tests for ensuring a metric's `input_measures` gets properly populated
* Begin populating `metric.type_params.input_measures`
This isn't my favorite bit of code. Mostly because there are checks for
existence which really should be handled before this point, however a
good point for that to happen doesn't exist currently. For instance,
in an ideal world by the time we get to `_process_metric_node`, if a
metric is of type `RATIO` and the nominator and denominator should be
guaranteed.
* Update test checking that disabled metrics aren't added to the manifest metrics
We updated from the metric `number_of_people` to `average_tenure_minus_people` for
this test because disabling `number_of_people` raised other exceptions at parse
time due to a metric referencing a disabled metric. The metric `average_tenure_minus_people`
is a leaf metric, and so for this test, it is a better candidate.
* Update `test_disabled_metric_ref_model` to have more disabled metrics
There are metrics which depend on the metric `number_of_people`. If
`number_of_people` is disabled without the metrics that depend on it
being disabled, then a different (expected) exception would be raised
than the one this test is testing for. Thus we've disabled those
downstream metrics.
* Add test which checks that metrics depending on disabled metrics raise an exception
* Add changie doc for populating metric input measures
* Add merge incremental strategy
* Expect merge to be a valid strategy for Postgres
---------
Co-authored-by: Anders Swanson <anders.swanson@dbtlabs.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* CT-2711: Add remove_tests() call to delete_schema_source() so that call sites are more uniform with other node deletion call sites. This will enable further code factorization.
* CT-2711: Factor repeated code section (mostly) out of PartialParsing.handle_schema_file_changes()
* CT-2711: Factor a repeated code section out of schedule_nodes_for_parsing()
* Update semantic model parsing test to check measure agg params
* Make `use_discrete_percentile` and `use_approximate_percentile` non optional and default false
This was a mistake in our implementation of the MeasureAggregationParams.
We had defined them as optional and defaulting to `None`. However, as the
protocol states, they cannot be `None`, they must be a boolean value.
Thus now we now ensure them.
* Add changie doc for measure percentile fixes
* Update semantic model parsing test to check different measure expr types
* Allow semantic model measure exprs to be defined with ints and bools in yaml
Sometimes the expr for a measure can defined in yaml with a bool or an int.
However, we were only allowing for strings. There was a work around for this,
which was wrapping your bool or int in double quotes in the yaml, but
this can be fairly annoying for the end user.
* Changie doc for fixing measure expr yaml specification
- Add --target-path to dbt snapshot command. ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- dbt build selection of tests' descendants ([#7289](https://github.com/dbt-labs/dbt-core/issues/7289))
- fix groupable node partial parsing, raise DbtReferenceError at runtime for safety ([#7437](https://github.com/dbt-labs/dbt-core/issues/7437))
- Fix partial parsing of latest_version changes for downstream references ([#7369](https://github.com/dbt-labs/dbt-core/issues/7369))
- Use "add_node" to update depends_on.nodes ([#7453](https://github.com/dbt-labs/dbt-core/issues/7453))
- Fix var precedence in configs: root vars override package vars ([#6705](https://github.com/dbt-labs/dbt-core/issues/6705))
- Fix inverted `--print/--no-print` flag ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
- Back-compat for previous return type of 'collect_freshness' macro ([#7489](https://github.com/dbt-labs/dbt-core/issues/7489))
- print model version in dbt show if specified ([#7407](https://github.com/dbt-labs/dbt-core/issues/7407))
- enable dbt show for seeds ([#7273](https://github.com/dbt-labs/dbt-core/issues/7273))
- push down limit filtering to adapter ([#7390](https://github.com/dbt-labs/dbt-core/issues/7390))
- Allow missing `profiles.yml` for `dbt deps` and `dbt init` ([#7511](https://github.com/dbt-labs/dbt-core/issues/7511))
-`run_results.json` is now written after every node completes. ([#7302](https://github.com/dbt-labs/dbt-core/issues/7302))
- Do not rewrite manifest.json during 'docs serve' command ([#7553](https://github.com/dbt-labs/dbt-core/issues/7553))
- Pin protobuf to greater than 4.0.0 ([#7565](https://github.com/dbt-labs/dbt-core/issues/7565))
- inject sql header in query for show ([#7413](https://github.com/dbt-labs/dbt-core/issues/7413))
- Pin urllib3 to ~=1.0 ([#7573](https://github.com/dbt-labs/dbt-core/issues/7573))
- Throw error for duplicated versioned and unversioned models ([#7487](https://github.com/dbt-labs/dbt-core/issues/7487))
### Under the Hood
- Update docs link in ContractBreakingChangeError message ([#7366](https://github.com/dbt-labs/dbt-core/issues/7366))
- Reduce memory footprint of cached statement results. ([#7281](https://github.com/dbt-labs/dbt-core/issues/7281))
- Remove noisy parsing events: GenericTestFileParse, MacroFileParse, Note events for static model parsing ([#6671](https://github.com/dbt-labs/dbt-core/issues/6671))
- Update --help text for cache-related parameters ([#7381](https://github.com/dbt-labs/dbt-core/issues/7381))
- Small UX improvements to model versions: Support defining latest_version in unsuffixed file by default. Notify on unpinned ref when a prerelease version is available. ([#7443](https://github.com/dbt-labs/dbt-core/issues/7443))
- Added warnings for model and ref deprecations ([#7433](https://github.com/dbt-labs/dbt-core/issues/7433))
- Update drop_relation macro to allow for configuration of drop statement separately from object name ([#7625](https://github.com/dbt-labs/dbt-core/issues/7625))
- accept publications in dbt.invoke ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Fixes
- Honor `--skip-profile-setup` parameter when inside an existing project ([#7594](https://github.com/dbt-labs/dbt-core/issues/7594))
- Fix: Relative project paths weren't working with deps ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491))
- Exclude password fields from Jinja rendering. ([#7629](https://github.com/dbt-labs/dbt-core/issues/7629))
- Add --target-path to more CLI subcommands ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- Stringify flag paths for Jinja context ([#7495](https://github.com/dbt-labs/dbt-core/issues/7495))
- write run_results.json for run operation ([#7502](https://github.com/dbt-labs/dbt-core/issues/7502))
### Under the Hood
- Add ability to instantiate Flags class from dict ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607))
- Add other relation to reffable nodes ([#7550](https://github.com/dbt-labs/dbt-core/issues/7550))
- Move node patch method to schema parser patch_node_properties and refactor schema parsing ([#7430](https://github.com/dbt-labs/dbt-core/issues/7430))
- Dropped support for Python 3.7 ([#7082](https://github.com/dbt-labs/dbt-core/issues/7082))
- Switch from dbt-metrics to dbt-semantic-interfaces for MetricNode definitions ([#7500](https://github.com/dbt-labs/dbt-core/issues/7500), [#7404](https://github.com/dbt-labs/dbt-core/issues/7404))
### Features
- Add support for materialized views ([#6911](https://github.com/dbt-labs/dbt-core/issues/6911))
- Optimize template rendering for common parse scenarios ([#7449](https://github.com/dbt-labs/dbt-core/issues/7449))
- nodes in packages respect custom generate_alias_name, generate_schema_name, generate_database_name macro overrides defined in packages ([#7444](https://github.com/dbt-labs/dbt-core/issues/7444))
- Enable state for deferral to be separate from state for selectors ([#7300](https://github.com/dbt-labs/dbt-core/issues/7300))
- Validate public models are not materialized as ephemeral ([#7226](https://github.com/dbt-labs/dbt-core/issues/7226))
- Added support for parsing and serializaing semantic models ([#7499](https://github.com/dbt-labs/dbt-core/issues/7499), [#7503](https://github.com/dbt-labs/dbt-core/issues/7503))
### Fixes
- Constraint rendering fixes: wrap check expression in parentheses, foreign key 'references', support expression in all constraint types ([#7417](https://github.com/dbt-labs/dbt-core/issues/7417), [#7480](https://github.com/dbt-labs/dbt-core/issues/7480), [#7416](https://github.com/dbt-labs/dbt-core/issues/7416))
- Fix warning messages for deprecated dbt_project.yml configs ([#7424](https://github.com/dbt-labs/dbt-core/issues/7424))
- Add `%` to adapter suite test cases for `persist_docs` ([#7698](https://github.com/dbt-labs/dbt-core/issues/7698))
- Improve warnings for constraints and materialization types ([#7335](https://github.com/dbt-labs/dbt-core/issues/7335))
- Incorrect paths used for "target" and "state" directories ([#7465](https://github.com/dbt-labs/dbt-core/issues/7465))
- fix StopIteration error when publication for project not found ([#7711](https://github.com/dbt-labs/dbt-core/issues/7711))
- Using version 0 works when resolving single model ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
