* Handle unknown `type_code` for model contracts
* Changelog entry
* Fix changelog entry
* Functional test for a `type_code` that is not recognized by psycopg2
* Functional tests for data type mismatches
* add test
* fix test
* first pass with constraint error
* add back column checks for temp tables
* changelog
* Update .changes/unreleased/Fixes-20231024-145504.yaml
* changie doc for DSI 0.3.0 upgrade
* Gracefully handle v10 metric filters
* Fix iteration over metrics in `upgrade_v10_metric_filters`
* Update previous manifest version test fixtures to have more expressive metrics
* Regenerate the test v10 manifest artifact using the more expressive metrics from 904cc1ef
To do this I cherry-picked 904cc1ef onto my local 1.6.latest branch,
had the test regenerate the test v10 manifest artifact, and then over
wrote the test v10 manifest artifact on this branch (cherry-picking it
across the branches didn't work, had to copy paste :grimmace:)
* Regenerate test v11 manifest artifact using the fixture changes in 904cc1ef
* Update `upgrade_v10_metric_filters` to handled disabled metrics
Regenerating the v10 and v11 test manifest artifacts uncovered an
issue wherein we weren't handling disabled metrics that need to
get upgraded. This commit fixes that. Additionally, the
`upgrade_v10_metric_filters` was getting a bit unwieldy, so I broke
extracted the abstracted sub functions.
* Fix `test_backwards_compatible_versions` test
When we regenerated the v10 test manifest artifact, it started having
the `metricflow_time_sine` model, and it didn't previously. This caused
`test_backwards_compatible_versions` to start failing because it was
no longer identified as having modified state for v10. The test has
been altered accordingly
* Bump to dbt-semantic-interfaces 0.3.0b1
* Update import path of `WhereFilterParser` from `dbt-semantic-interfaces`
In 0.3.x of `dbt-semantic-intefaces` the location of the WhereFilterParser
moved to be grouped in with a bunch of new adjacent code. As such,
we needed to correct our import path of it.
* Create basic `SavedQuery` node type based on `SavedQuery` protocol from DSI
* Add ability to add SavedQueries to the manifest
* Define unparsed SavedQuery node
* Begin parsing saved_query objects to manifest
* Skip jinja rendering of `SavedQuery.where` property
* Begin propagating `SavedQueries` on the manifest to the semantic manifest
* Add tests for basic saved query parsing
* Add custom pluralization handling of SavedQuery node type
* Add a config subclass to SavedQuery node
* Move the SavedQuery node to nodes.py
Unfortunately things are a bit too intertwined currently for SavedQuery
to be in it's own file. We need to add the SavedQuery node to the
GraphMemberNode, unfortunately with SavedQuery in it's own file,
importing it would have caused a circular dependency. We'll need
to separately come in and split things up as a cleanup portion of
work.
* Add basic plumbing of saved query configs to projects
* Add basic lookup utility for saved queries, SavedQueryLookup
* Handle disabled SavedQuery nodes in parsing and lookups
* Add SavedQuery nodes to grouping process
Our grouping logic seems to be in a weird spot. It seems liek we're
moving to setting the `group` for a node in the node's `config` however,
all of the logic around grouping is still focused on the top level `group`
property on a nodes. To get group stuff plumbed I've thus added `group`
as a top level property of the `SavedQuery` node, and populated it from
the config group value.
* Plumb through saved query in a lot more places
I don't like making scatter shot commits like this. However, a lot
of this commit was written ~4am, soooo yea. Things were broken, I wanted
things to be unbroken. I mostly searched for `semantic_models` and added
the equivalent necessary `saved_queries`. Some stuff is in support of
writing out the manifest, some stuff helps with node selection, it's a
lot of miscelaneous stuff that I don't fully understand.
* Add `depends_on` to `SavedQuery` nodes and populate from `metrics` property
* Add partial parsing support to SavedQuery nodes
* Add `docs` support for SavedQuery descriptions
* Support selctor methods for SavedQuery nodes
* Add `refs` property to SavedQuery node
We don't actually append anything to `refs` for SavedQuery nodes currently.
I'm not sure if anything needs to be appended to them. Regardless, we
access the `refs` property throughout the codebase while iterating over
nodes. It seems wise to support this attribute as to not accidently blow
something up with it not existing.
* Support `saved_queries` when upgrading from manifests <= v10 (and regenerate v11)
* Add changie doc for saved query node support
* Pin to dbt-semantic-interfaces 0.3.0b1 for saved query work
We're gonna release DSI 0.3.0, and if this PR automatically pulls that
in things will break. But the things that need fixing should be handled
separately from this PR. After releasing DSI 0.3.0 I'm going to create
a branch off/ontop of this one, and open a stacked PR with the associated
changes.
* Bump supported DSI version to 0.3.x
* Switch metric filters and saved query where to use ne WhereFilterIntersection
* Update schema yaml readers to create WhereFilterInterfaces
* Expand metric filters and saved query where property to handle both str and list of strs
* Update tests which were broken by where filter changes
* Regeneate v11 manifest
* Fixup: Update `SavedQueryLookup.perform_lookup` to operate on saved queries
I missed this when I was copy and pasting 🤦
* Add support for getting freshness from DBMS metadata
* Add changelog entry
* Add simple test case
* Change parsing error to warning and add new event type for warning
* Code review simplification of capability dict.
* Revisions to the capability mechanism per review
* Move utility function.
* Reduce try/except scope
* Clean up imports.
* Simplify typing per review
* Unit test fix
* add `store_failures_as` parameter to TestConfig, catch strategy parameter in test materialization
* create test results as views
* updated test expected values for new config option
* break up tests into reusable tests and adapter specific configuration, update test to check for relation type and confirm views update
* move test configuration into base test class
* allow `store_failures_as` to drive whether failures are stored
* update expected test config dicts to include the new default value for store_failures_as
* Add `store_failures_as` config for generic tests
* cover --store-failures on CLI gap
* add generic tests test case for store_failures_as
* update object names for generic test case tests for store_failures_as
* remove unique generic test, it was not testing `store_failures_as`
* pull generic run and assertion into base test class to turn tests into quasi-parameterized tests
* add ephemeral option for store_failures_as, as a way to easily turn off store_failures at the model level
* add compilation error for invalid setting of store_failures_as
---------
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* Explanation of Parsing vs. Compilation vs. Runtime
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md
* Apply suggestions from code review
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
* Fix a couple markdown rendering issues
* Move to the "explain it like im 64" folder
When ELI5 just isnt detailed enough.
* Disambiguate Python references
Disambiguate Python references and delineate SQL models ("Jinja-SQL") from Python models ("dbt-py")
---------
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
* Add semantic model test to `test_contracts_graph_parsed.py`
The tests in `test_contracts_graph_parsed.py` are meant to ensure
that we can go from objects to dictionaries and back without any
changes. We've had a desire to simplify these tests. Most tests in
this file have three to four fixtures, this test only has one. What
a test of this format ensures is that parsing a SemanticModel from
a dictionary doesn't add/drop any keys from the dictionary and that
when going back to the dictionary no keys are dropped. This style of
test will still break whenever the semantic model (or sub objects)
change. However now when that happens, only one fixture will have to
be updated (whereas previously we had to update 3-4 fixtures).
* Begin using hypothesis package for symmetry testing
Hypothesis is a python package for doing property testing. The `@given`
parameterizes a test, with it generating the arguements it has following
`strategies`. The main strategies we use is `builds` this takes in a callable
passes any sub strategies for named arguements, and will try to infer any
other arguments if the callable is typed. I found that even though the
test was run many many times, some of the `SemanticModel` properties
weren't being changed. For instance `dimensions`, `entities`, and `measures`
were always empty lists. Because of this I defined sub strategies for
some attributes of `SemanticModel`s.
* Update unittest readme to have details on test_contracts_graph_parsed methodology
* Include option to generate static index.html
* Added changie
* Using DBT's system load / write file methods for better cross platform
support
* Updated docs tests with dbt.client.systems calls for file reading
* Writing out static_index.html as binary file to prevent line-ending
conversions on Windows. (similar behaviour as index.html)
* Add performance metrics to the CommandCompleted event.
* Add changelog entry.
* Add flag for controling the log level of ResourceReport.
* Update changelog entry to reflect changes
* Remove outdated attributes
* Work around missing resource module on windows
* Fix corner case where flags are not set
* Add new get_catalog_relations macro, allowing dbt to specify which relations in a schema the adapter should return data about
* Implement postgres adapter support for relation filtering on catalog queries
* Code review changes adding feature flag for catalog-by-relation-list support
* Use profile specified in --profile with dbt init (#7450)
* Use profile specified in --profile with dbt init
* Update .changes/unreleased/Fixes-20230424-161642.yaml
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Refactor run() method into functions, replace exit() calls with exceptions
* Update help text for profile option
---------
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* add TestLargeEphemeralCompilation (#8376)
* Fix a couple of issues in the postgres implementation of get_catalog_relations
* Add relation count limit at which to fall back to batch retrieval
* Better feature detection mechanism for adapters.
* Code review changes to get_catalog_relations and adapter feature checking
* Add changelog entry
---------
Co-authored-by: ezraerb <ezraerb@alum.mit.edu>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
* Add `date_spine` macro (and macros it depends on) from dbt-utils to core
The macros added are
- date_spine
- get_intervals_between
- generate_series
- get_powers_of_two
We're adding these to core because they are becoming more prevalently used
with the increase usage in the semantic layer. Basically if you are
using the semantic layer currently, then it is almost a requirement
to use dbt-utils, which is undesireable given the SL is supported directly
in core. The primary focus of this was to just add `date_spine`. However,
because `date_spine` depends on other macros, these other macros were
also moved.
* Add adapter tests for `get_powers_of_two` macro
* Add adapter tests for `generate_series` macro
* Add adapter tests for `get_intervals_between` macro
* Add adapter tests for `date_spine` macro
* Improve test fixture for `date_spine` macro to work with multiple adapters
* Cast to types to date in fixture_date_spine when targeting redshift
* Improve test fixture for `get_intervals_between` macro to work with multiple adapters
* changie doc for adding date_spine macro
* Include 'join_to_timespine` and `fill_nulls_with` in metric fixture
* Support `join_to_timespine` and `fill_nulls_with` properties on measure inputs to metrics
* Assert new `fill_nulls_with` and `join_to_timespine` properties don't break associated DSI protocol
* Add doc for metric null coalescing improvements
* Fix unit test for unparsed metric objects
The `assert_symmetric` function asserts that dictionaries are mostly
equivalent. I say mostly equivalent because it drops keys that are
`None`. The issue is that that `join_to_timespine` gets defaulted
to `False`, so we have to specify it in the `get_ok_dict` so that
they match.
* allow multioption to be quoted
* changelog
* fix test
* remove list format
* fix tests
* fix list object
* review arg change
* fix quotes
* Update .changes/unreleased/Features-20230918-150855.yaml
* add types
* convert list to set in test
* make mypy happy
* mroe mypy happiness
* more mypy happiness
* last mypy change
* add node to test
* Extend use of type annotations in the events module.
* Add return type of None to more __init__ definitions.
* Still more type annotations adding -> None to __init__
* Tweak per review
* Use profile specified in --profile with dbt init
* Update .changes/unreleased/Fixes-20230424-161642.yaml
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Refactor run() method into functions, replace exit() calls with exceptions
* Update help text for profile option
---------
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* move config changes into alter.sql in alignment with other adapters
* move shared relations macros to relations root
* move single models files to models root
* add table to replace
* move create file into relation directory
* implement replace for postgres
* move column specific macros into column directory
* add unit test for can_be_replaced
* update renameable_relations and replaceable_relations to frozensets to set defaults
* fixed tests for new defaults
* Add docstrings to `contracts/graph/metrics.py` functions to document what they do
Used [dbt-labs/dbt-core#5607](https://github.com/dbt-labs/dbt-core/pull/5607)
for context on what the functions should do.
* Add typing to `reverse_dag_parsing` and update function to work on 1.6+ metrics
* Add typing to `parent_metrics` and `parent_metrics_names`
* Add typing to `base_metric_dependency` and `derived_metric_dependency` and update functions to work on 1.6+ metrics
* Simplify implementations of `basic_metric_dependency` and `derived_metric_dependnecy`
* Add typing to `ResolvedMetricReference` initialization
* Add typing to `derived_metric_dependency_graph`
* Simplify conditional controls in `ResolvedMetricReference` functions
The functions in `ResolvedMetricReference` use `manifest.metric.get(...)`
which will only return either a `Metric` or `None`, never a different
node type. Thus we don't need to check that the returned metric is
a metric.
* Don't recurse on over `depends_on` for non-derived metrics in `reverse_dag_parsing`
The function `reverse_dag_parsing` only cares about derived metrics,
that is metrics that depend on other metrics. Metrics only depend on
other metrics if they are one of the `DERIVED_METRICS` types. Thus
doing a recursive call to `reverse_dag_parsing` for non `DERIVED_METRICS`
types is unnecessary. Previously we were iterating over a metric's
`depends_on` property regardless of whether the metric was a `DERIVED_METRICS`
type. Now we only do this work if the metric is of a `DERIVED_METRICS`
type.
* Simplify `parent_metrics_names` by having it call `parent_metrics`
* Unskip `TestMetricHelperFunctions.test_derived_metric` and update fixture setup
* Add changie doc for metric helper function updates
* Get manifest in `test_derived_metric` from the parse dbt_run invocation
* Remove `Relation` a intiatlization attribute for `ResolvedMetricReference`
* Add return typing to class `__` functions of `ResolvedMetricReference`
* Move from `manifest.metrics.get` to `manifest.expect` in metric helpers
Previously with `manifest.metrics.get` we were just skipping when `None`
was returned. Getting `None` back was expected in that `parent_unique_id`s
that didn't belong to metrics should return `None` when calling
`manifest.metrics.get`, and these are fine to skip. However, there's
an edgecase where a `parent_unique_id` is supposed to be a metric, but
isn't found, thus returning `None`. How likely this edge case could
get hit, I'm not sure, but it's a possible edge case. Using `manifest.metrics.get`
it we can't actually tell if we're in the edge case or not. By moving
to `manifest.expect` we get the error handling built in, and the only
trade off is that we need to change our conditional to skip returned
nodes that aren't metrics.
* update `Number` class to handle integer values (#8306)
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* account for integer vs number on table merges
* add tests for combining number with integer.
* add unit test when nulls are added
* cant none as an Integer
* fix null tests
---------
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Dave Connors <dave.connors@fishtownanalytics.com>
* first draft of adding in table - materialized view swap
* table/view/materialized view can all replace each other
* update renameable relations to a config
* migrate relations macros from `macros/adapters/relations` to `macros/relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar
* move drop macro to drop macro file
* align the behavior of get_drop_sql and drop_relation, adopt existing default from drop_relation
* add explicit ddl for drop statements instead of inheriting the default from dbt-core
* update replace macro dependent macros to align with naming standards
* update type for mashumaro, update related test
* Improve typing of `ContextMember` functions
* Improve typing of `Var` functions
* Improve typing of `ContextMeta.__new__`
* Improve typing `BaseContext` and functions
In addition to just adding parameter typing and return typing to
`BaseContext` functions. We also declared `_context_members_` and
`_context_attrs_` as properites of `BaseContext` this was necessary
because they're being accessed in the classes functions. However,
because they were being indirectly instantiated by the metaclass
`ContextMeta`, the properties weren't actually known to exist. By
adding declaring the properties on the `BaseContext`, we let mypy
know they exist.
* Remove bare `invocations` of `@contextmember` and `@contextproperty`, and add typing to them
Previously `contextmember` and `contextproperty` were 2-in-1 decorators.
This meant they could be invoked either as `@contextmember` or
`@contextmember('some_string')`. This was fine until we wanted to return
typing to the functions. In the instance where the bare decorator was used
(i.e. no `(...)` were present) an object was expected to be returned. However
in the instance where parameters were passed on the invocation, a callable
was expected to be returned. Putting a union of both in the return type
made the invocations complain about each others' return type. To get around this
we've dropped the bare invocation as acceptable. The parenthesis are now always
required, but passing a string in them is optional.
* WIP
* WIP
* get group and enabled added
* changelog
* cleanup
* getting measure lookup working
* missed file
* get project level working
* fix last test
* add groups to config tests
* more group tests
* fix path
* clean up manifest.py
* update error message
* fix test assert
* remove extra check
* resolve conflicts in manaifest
* update manifest
* resolve conflict
* add alias
* Add compiled node properties to run_results.json
* Include compiled-node attributes in run_results.json
* Fix typo
* Bump schema version of run_results
* Fix test assertions
* Update expected run_results to reflect new attributes
* Code review changes
* Fix mypy warnings for ManifestLoader.load() (#8443)
* revert python version for docker images (#8445)
* revert python version for docker images
* add comment to not update python version, update changelog
* Bumping version to 1.7.0b1 and generate changelog
* [CT-3013] Fix parsing of `window_groupings` (#8454)
* Update semantic model parsing tests to check measure non_additive_dimension spec
* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`
* Add changie doc for `window_groupings` parsing fix
* update `Number` class to handle integer values (#8306)
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Improve docker image README (#8212)
* Improve docker image README
- Fix unnecessary/missing newline escapes
- Remove double whitespace between parameters
- 2-space indent for extra lines in image build commands
* Add changelog entry for #8212
* ADAP-814: Refactor prep for MV updates (#8459)
* apply reformatting changes only for #8449
* add logging back to get_create_materialized_view_as_sql
* changie
* swap trigger (#8463)
* update the implementation template (#8466)
* update the implementation template
* add colon
* Split tests into classes (#8474)
* add flaky decorator
* split up tests into classes
* revert update agate for int (#8478)
* updated typing and methods to meet mypy standards (#8485)
* Convert error to conditional warning for unversioned contracted model, fix msg format (#8451)
* first pass, tests need updates
* update proto defn
* fixing tests
* more test fixes
* finish fixing test file
* reformat the message
* formatting messages
* changelog
* add event to unit test
* feedback on message structure
* WIP
* fix up event to take in all fields
* fix test
* Fix ambiguous reference error for duplicate model names across packages with tests (#8488)
* Safely remove external nodes from manifest (#8495)
* [CT-2840] Improved semantic layer protocol satisfaction tests (#8456)
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified
* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols
* Add test asserting Defaults obj satisfies protocol
* Add test asserting SemanticModel with optionals specified satisfies protocol
* Split dimension protocol satisfaction tests into with and without optionals
* Simplify DSI Protocol import strategy in protocol satisfaction tests
* Add test asserting DimensionValidtyParams satisfies protocol
* Add test asserting DimensionTypeParams satisfies protocol
* Split entity protocol satisfaction tests into with and without optionals
* Split measure protocol satisfication tests and add measure aggregation params satisficaition test
* Split metric protocol satisfaction test into optional specified an unspecified
Additionally, create where_filter pytest fixture
* Improve protocol satisfaction tests for MetricTypeParams and sub protocols
Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
* Convert to using mashumaro jsonschema with acceptable performance (#8437)
* Regenerate run_results schema after merging in changes from main.
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Jaime Martínez Rincón <jaime@jamezrin.name>
Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified
* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols
* Add test asserting Defaults obj satisfies protocol
* Add test asserting SemanticModel with optionals specified satisfies protocol
* Split dimension protocol satisfaction tests into with and without optionals
* Simplify DSI Protocol import strategy in protocol satisfaction tests
* Add test asserting DimensionValidtyParams satisfies protocol
* Add test asserting DimensionTypeParams satisfies protocol
* Split entity protocol satisfaction tests into with and without optionals
* Split measure protocol satisfication tests and add measure aggregation params satisficaition test
* Split metric protocol satisfaction test into optional specified an unspecified
Additionally, create where_filter pytest fixture
* Improve protocol satisfaction tests for MetricTypeParams and sub protocols
Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
* first pass, tests need updates
* update proto defn
* fixing tests
* more test fixes
* finish fixing test file
* reformat the message
* formatting messages
* changelog
* add event to unit test
* feedback on message structure
* WIP
* fix up event to take in all fields
* fix test
* add show test for json data
* oh changie my changie
* revert unecessary cahnge to fixture
* keep decimal class for precision methods, but return __int__ value
* jerco updates
* update integer type
* update other tests
* Update .changes/unreleased/Fixes-20230803-093502.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Update semantic model parsing tests to check measure non_additive_dimension spec
* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`
* Add changie doc for `window_groupings` parsing fix
* first pass
* WIP
* update issue body
* fix triggering label
* fix docs
* add better run name
* reduce complexity
* update description
* fix PR title
* point at workflow on main
* fix wording
* add label
* Update semantic model parsing test to check `create_metric = true` functionality
* Add `create_metric` boolean property to unparsed measure objects
* Begin creating metrics from measures with `create_metric = True`
* Add test ensuring partial parsing handles metrics generated from measures
* Ensure partial parsing appropriately deletes metrics generated from semantic models
* Add changie doc for addition
* Separate generated metrics from parsed metrics for partial parsing
I was doing a demo earlier today of this branch (minus this commit)
and noticed something odd. When I changes a semantic model, metrics
that should have been technically uneffected would get dropped. Basically
if I made a change to a semantic model which had metrics in the same
file, and then ran parse, those metrics defined in the same file
would get dropped. Then with no other changes, if I ran parse again
they would come back. What was happening was that parsed metrics
and generated metrics were getting tracked the same way on the file
objects for partial parsing. In 0787a7c7b6
we began dropping all metrics tracked in a file objects when changes
to semantic models were detected. Since parsed metrics and generated
metrics were being tracked together on the file object, the parsed
metrics were getting dropped as well. In this commit we begin separating
out the tracking of generated metrics and parsed metrics on the
file object, and now only drop the generated metrics when semantic
models have a detected change.
* Assert in test that semantic model partial parsing doesn't clobber regular metrics
* Replaced the FirstRunResultError and AfterFirstRunResultError events with RunResultError.
* Attempts at reasonable unit tests.
* Restore event manager after unit test.
* Support configurable delimiter for seed files, default to comma (#3990)
* Update Features-20230317-144957.yaml
* Moved "delimiter" to seed config instead of node config
* Update core/dbt/clients/agate_helper.py
Co-authored-by: Cor <jczuurmond@protonmail.com>
* Update test_contracts_graph_parsed.py
* fixed integration tests
* Added functional tests for seed files with a unique delimiter
* Added docstrings
* Added a test for an empty string configured delimiter value
* whitespace
* ran black
* updated changie entry
* Update Features-20230317-144957.yaml
---------
Co-authored-by: Cor <jczuurmond@protonmail.com>
* add param to control maxBytes for single dbt.log file
* nits
* nits
* Update core/dbt/cli/params.py
Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
---------
Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
* Add test ensuring `warn_error_options` is dictified in `invocation_args_dict` of contexts
* Add dictification specific to `warn_error_options` in `args_to_dict`
* Changie doc for serialization changes of warn_error_options
* Add test asserting that a macro with the work materializtion doesn't cause issues
* Let macro names include the word `materialization`
Previously we were checking if a macro included a materialization
based on if the macro name included the word `materialization`. However,
a macro name included the word `materialization` isn't guarnteed to
actually have a materialization, and a macro that doesn't have
`materialization` in the name isn't guaranteed to not have a materialization.
This change is to detect macros with materializations based on the
detected block type of the macro.
* Add changie doc materialization in macro detection
* Add test for checking that `_connection_exception_retry` handles `EOFError`s
* Update `_connection_exception_retry` to handle `EOFError` exceptions
* Add changie docs for `_connection_exception_retry` handling `EOFError` exceptions
* applied new integration tests to existing framework
* applied new integration tests to existing framework
* generalized tests for reusability in adapters; fixed drop index issue
* generalized tests for reusability in adapters; fixed drop index issue
* removed unnecessary overrides in tests
* adjusted import to allow for usage in adapters
* adjusted import to allow for usage in adapters
* removed fixture artifact
* generalized the materialized view fixture which will need to be specific to the adapter
* unskipped tests in the test runner package
* corrected test condition
* corrected test condition
* added missing initial build for the relation type swap tests
* add env vars for datadog ci visibility
* modify pytest command for tracing
* fix posargs
* move env vars to job that needs them
* add test repeater to DD
* swap flags
* Bump version support for `dbt-semantic-interfaces` to `~=0.1.0rc1`
* Add tests for asserting WhereFilter satisfies protocol
* Add `call_parameter_sets` to `WhereFilter` class to satisfy protocol
* Changie doc for moving to DSI 0.1.0rc1
* [CT-2822] Fix `NonAdditiveDimension` Implementation (#8089)
* Add test to ensure `NonAdditiveDimension` implementation satisfies protocol
* Fix typo in `NonAdditiveDimension`: `window_grouples` -> `window_groupings`
* Add changie doc for typo fix in NonAdditiveDimension
* Add metrics from metric type params to a metric's depends_on
* Add Lookup utility for finding `SemanticModel`s by measure names
* Add the `SemanticModel` of a `Metric`'s measure property to the `Metric`'s `depends_on`
* Add `SemanticModelConfig` to `SemanticModel`
Some tests were failing due to `Metric`'s referencing `SemanticModels`.
Specifically there was a check to see if a referenced node was disabled,
and because `SemanticModel`'s didn't have a `config` holding the `enabled`
boolean attr, core would blow up.
* Checkpoint on test fixing
* Correct metricflow_time_spine_sql in test fixtures
* Add check for `SemanticModel` nodes in `Linker.link_node`
Now that `Metrics` depend on `SemanticModels` and `SemanticModels`
have their own dependencies on `Models` they need to be checked for
in the `Linker.link_node`. I forget the details but things blow up
without it. Basically it adds the SemanticModels to the dependency
graph.
* Fix artifacts/test_previous_version_state.py tests
* fix access/test_access.py tests
* Fix function metric tests
* Fix functional partial_parsing tests
* Add time dimension to semantic model in exposures fixture
* Bump DSI version to a minimum of 0.1.0dev10
DSI 0.1.0dev10 fixes an incoherence issue in DSI around `agg_time_dimension`
setting. This incoherence was that `measure.agg_time_dimension` was being
required, even though it was no longer supposed to be a required attribute
(it's specificially typed as optional in the protocol). This was causing
a handful of tests to fail because the `semantic_model.defaults.agg_time_dimension`
value wasn't being respected. Pulling in the fix from DSI 0.1.0dev10 fixes
the issue.
Interestingly after bumping the DSI version, the integration tests were
still failing. If I ran the tests individually they passed though. To get
`make integration` to run properly I ended up having to clear my `.tox`
cache, as it seems some outdated state was being persisted.
* Add test specifically for checking the `depends_on` of `Metric` nodes
* Re-enable test asserting calling metric nodes in models
* Migrate `checked_agg_time_dimension` to `checked_agg_time_dimension_for_measure`
DSI 0.1.0dev10 moved `checked_agg_time_dimension` from the `Measure`
protocol to the `SemanticModel` protocol as `checked_agg_time_dimension_for_measure`.
This finishes a change where for a given measure either the `Measure.agg_time_dimension`
or the measure's parent `SemanticModel.defaults.agg_time_dimension` needs to be
set, instead of always require the measure's `Measure.agg_time_dimension`.
* Add changie doc for populating metric
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
The original implementation of validate_sql was called dry_run,
but in the rename the test classes and much of their associated
documentation still retained the old naming.
This is mainly cosmetic, but since these test classes will be
imported into adapter repositories we should fix this now before
the wrong name proliferates.
* Add dry_run method to base adapter with implementation for SQLAdapters
resolves#7839
In the CLI integration, MetricFlow will issue dry run queries as
part of its warehouse-level validation of the semantic manifest,
including all semantic model and metric definitions.
In most cases, issuing an `explain` query is adequate, however,
BigQuery does not support the `explain` keyword and so we cannot
simply pre-pend `explain` to our input queries and expect the
correct behavior across all contexts.
This commit adds a dry_run() method to the BaseAdapter which mirrors
the execute() method in that it simply delegates to the ConnectionManager.
It also adds a working implementation to the SQLConnectionManager and
includes a few test cases for adapter maintainers to try out on their own.
The current implementation should work out of the box with most
of our adapters. BigQuery will require us to implement the dry_run
method on the BigQueryConnectionManager, and community-maintained
adapters can opt in by enabling the test and ensuring their own
implementations work as expected.
Note - we decided to make these concrete methods that throw runtime
exceptions for direct descendants of BaseAdapter in order to avoid
forcing community adapter maintainers to implement a method that does
not currently have any use cases in dbt proper.
* Switch dry_run implementation to be macro-based
The common pattern for engine-specific SQL statement construction
in dbt is to provide a default macro which can then be overridden
on a per-adapter basis by either adapter maintainers or end users.
The advantage of this is users can take advantage of alternative
SQL syntax for performance or other reasons, or even to enable
local usage if an engine relies on a non-standard expression and
the adapter maintainer has not updated the package.
Although there are some risks here they are minimal, and the benefit
of added expressiveness and consistency with other similar constructs
is clear, so we adopt this approach here.
* Improve error message for InvalidConnectionError in test_invalid_dry_run.
* Rename dry_run to validate_sql
The validate_sql name has less chance of colliding with dbt's
command nomenclature, both now and in some future where we have
dry-run operations.
* Rename macro and test files to validate_sql
* Fix changelog entry
* add permissions
* replace db setup
* try with bash instead of just pytest flags
* fix test command
* remove spaces
* remove force-flaky flag
* add starting vlaues
* add mac and windows postgres isntall
* define use bash
* fix typo
* update output report
* tweak last if condition
* clarify failures/successful runs
* print running success and failure tally
* just output pytest instead of capturing it
* set shell to not exit immediately on exit code
* add formatting around results for easier scanning
* more output formatting
* add matrix to unlock parallel runners
* increase to ten batches
* update debug
* add comment
* clean up comments
* Remove `create_metric` as a public facing `SemanticModel.Measure` property
We want to add `create_metric`. The `create_metric` property will be
incredibly useful. However, at this time it is not hooked up, and we don't
have time to hook it up before the code freeze for 1.6.0rc of core. As
it doesn't do anything, we shouldn't allow people to specify it, because
it won't do what one would expect. We plan on making the implementation
of `create_metric` a priority for 1.7 of core
* Changie doc for the removal of create_metric property
* add negative test case
* changie
* missed a comma
* Update changelog entry
* Add a negative number (rather than subtract a positive number)
---------
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* Fix accidental propagation of log messages to root logger.
* Add changelog entry
* Fixed an issue which blocked debug logging to stdout with --log-level debug, unless --debug was also used.
* Use dbt-semantic-interface validations on semantic models and metrics defined in Core.
* Remove empty test, since semantic models don't generate any validation warnings.
* Add changelog entry.
* Temporarily remove requirement that there must be semantic models definied in order to define metrics
* add interface changes section to the PR template
* update entire template
* split up choices for tests and interfaces
* minor formatting change
* add line breaks
* actually put in line breaks
* revert split choices in checklist
* add line breaks to top
* move docs link
* typo
* ct-2551: adds old and unmodified state selection methods
* ct-2551: update check_unmodified_content to simplify
* add unit and integration tests for unmodified and old
* add changelog entry
* ct-2551: reformatting of contingent adapter assignment list
* UnifiedToUTC
* Check proximity of dbt_valid_to and deleted time
* update the message to print if the assertion fails
* add CHANGELOG entries
* test only if naive
* Added comments about naive and aware
* Generalize comparison of datetimes that are "close enough"
---------
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* Fix tests fixtures which were using measures for metric numerator/denominators
In our previous upgrade to DSI dev7, numerators and denominators for
metrics switched from being `MetricInputMeasure`s to `MetricInput`s.
I.e. metric numerators and denominators should references other metrics,
not semantic model measures. However, at that time, we weren't actually
doing anything with numerators and denominators in core, so no issue
got raised. The changes we are about to make though are going to surface
these issues..
* Add tests for ensuring a metric's `input_measures` gets properly populated
* Begin populating `metric.type_params.input_measures`
This isn't my favorite bit of code. Mostly because there are checks for
existence which really should be handled before this point, however a
good point for that to happen doesn't exist currently. For instance,
in an ideal world by the time we get to `_process_metric_node`, if a
metric is of type `RATIO` and the nominator and denominator should be
guaranteed.
* Update test checking that disabled metrics aren't added to the manifest metrics
We updated from the metric `number_of_people` to `average_tenure_minus_people` for
this test because disabling `number_of_people` raised other exceptions at parse
time due to a metric referencing a disabled metric. The metric `average_tenure_minus_people`
is a leaf metric, and so for this test, it is a better candidate.
* Update `test_disabled_metric_ref_model` to have more disabled metrics
There are metrics which depend on the metric `number_of_people`. If
`number_of_people` is disabled without the metrics that depend on it
being disabled, then a different (expected) exception would be raised
than the one this test is testing for. Thus we've disabled those
downstream metrics.
* Add test which checks that metrics depending on disabled metrics raise an exception
* Add changie doc for populating metric input measures
* Add merge incremental strategy
* Expect merge to be a valid strategy for Postgres
---------
Co-authored-by: Anders Swanson <anders.swanson@dbtlabs.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* CT-2711: Add remove_tests() call to delete_schema_source() so that call sites are more uniform with other node deletion call sites. This will enable further code factorization.
* CT-2711: Factor repeated code section (mostly) out of PartialParsing.handle_schema_file_changes()
* CT-2711: Factor a repeated code section out of schedule_nodes_for_parsing()
* Update semantic model parsing test to check measure agg params
* Make `use_discrete_percentile` and `use_approximate_percentile` non optional and default false
This was a mistake in our implementation of the MeasureAggregationParams.
We had defined them as optional and defaulting to `None`. However, as the
protocol states, they cannot be `None`, they must be a boolean value.
Thus now we now ensure them.
* Add changie doc for measure percentile fixes
* Update semantic model parsing test to check different measure expr types
* Allow semantic model measure exprs to be defined with ints and bools in yaml
Sometimes the expr for a measure can defined in yaml with a bool or an int.
However, we were only allowing for strings. There was a work around for this,
which was wrapping your bool or int in double quotes in the yaml, but
this can be fairly annoying for the end user.
* Changie doc for fixing measure expr yaml specification
* CT-2651: Add Semantic Models to the manifest and various pieces of graph linking code
* CT-2651: Finish integrating semantic models into the partial parsing system
* CT-2651: More semantic model details for partial parsing
* CT-2651: Remove merged references to project_dependencies
* CT-2651: Revise changelog entry
* CT-2651: Disable unit test until partial parsing of semantic models is complete.
* CT-2651: Temporarily disable an apparently-flaky test.
* Add some comments to methods constructing Project/RuntimeConfig
* Save flag that packages dict came from dependencies.yml
* Test for not rendering packages_dict
* Changie
* Ensure packages_yml_dict and dependencies_yml_dict are dictionaries
* Ensure "packages" passed to render_packages is a dict
* Bump DSI dependency version to 0.1.0dev7
* Cleaner DSI type enum importing
Previoulsy we had to use individual import paths for each type enum
that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces
has been updated to allow for importing all the type enums from a
singular path.
* Cleaner DSI protocol importing
Previoulsy we had to use individual import paths for each protocol
that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces
has been updated to allow for importing all the protocols from a
singular path.
* Add semantic protocol satisifcation test for metric type params
* Replace `metric.type_params.measures` with `metric.type_params.input_measures`
In DSI 0.1.0dev7 `measures` on metric type params became `input_measures`.
Additionally `input_measures` should not be user specified but something
we compile at parse time, thus we've removed it from `UnparsedMetricTypeParams`.
Finally, actually populating `input_measures` is somewhat complicated due
to the existance of derived metrics, thus that work is being pushed
off to CT-2707.
* Update metric numerator/denominator to be `MetricInput`s
In DSI 0.1.0dev7 `metric.type_params.numerator` and `metric.type_params.denominator`
switched from being `MetricInputMeasure`s to `MetricInput`s. This
commit reflects that change. Additionally, some helper functions on
metric type params were removed related to the numerator and denominator.
Thus we've removed them respectively in this commit.
* Add protocol satisfaction tests for `MetricInput` and `MetricInputMeasure`
* Add `post_aggregation_reference` to `MetricInput` and fix typo in `MetricInputMeasure`
DSI 0.1.0dev7 added `post_aggregation_reference` to the `MetricInput` protocol,
thus we've added it to our implementation in core. Additionally, we had a typo
in a method name in our implementation of `MetricInputMeasure`, ironically
a similar function to the one we've added for `MetricInput`
* Changie doc for upgraded to DSI 0.1.0dev7
* Fix parsing of metric numerator and denominator in schema_yaml_readers
Previously numerator and denominator of a metric were `MetricInputMeasure`s,
now they're `MetricInput`s. Changing the typing isn't enough though.
We have parsing functions in `schema_yaml_readers` which were specifically
parsing the numerator and denominator as if they were `MetricInputMeasure`s.
Thus we had to updating the schema_yaml_readers to parse them as `MetricInput`s.
During this we had some logic in a parsing function `_get_metric_inputs` which
could be abstracted to newly added functions.
* Upgrade to dbt-semantic-interfaces v0.1.0dev5
This is a fairly simple upgrade. Literally it's just pointing at the
the new versions. The v3 schemas are directly compatible with v5 because
there were no protocol level changes from v3 to v5. All the changers were
updates to tools MetricFlow uses from DSI, not tools that we ourselves
are using in core (yet).
* Add changie doc for DSI version bump
* Update metric filters in testing fixtures
I incorrectly wrote the tests such that they didn't include curly
braces, `{{..}}`, around things like `dimension(..)` for filters.
This updates the tests fixtures to have proper filter specifications
* Skip jinja rendering of `filter` key of metrics
Note that `filter` can show up in multiple places: as a root key
on a metric (`metric.filter`), on a metric input (`metric.type_params.metrics[x].filter`),
denominator (`metric.type_params.denominator.filter`), numerator
(`metric.type_params.numerator.filter`), and a metric input measure
(`metric.type_params.measure.filter` and `metric.type_params.measures[x].filter`).
In this commit we skip all of them :)
* Add changie doc for skipping jinja parsing for metric filters
* Update yaml renderer test for metrics
* Add AdapterRegistered event log message
* Add AdapterRegistered to unit test
* make versioning and logging consistent
* make versioning and logging consistent
* add to_version_string
* remove extra equals
* format fire_event
* Add tests to ensure our semantic layer nodes satisfy the DSI protocols
These tests create runtime checkable versions of the protocols defined in
DSI. Thus we can instantiate instances of our semantic layer nodes and
use `isinstance` to check that they satisfy the protocol. These `runtime_checkable`
versions of the protocols should only exist in testing and should never
be used in the actual package code.
* Update the `Dimension` object of `SemanticModel` node to match DSI protocol
* Make `UnparsedDimension` more strict and update schema readers accordingly
* Update the `Entity` object of `SemanticModel` node to match DSI protocol
* Make `UnparsedEntity` more strict and update schema readers accordingly
* Update the `Measure` object of `SemanticModel` node to match DSI protocol
* Make `UnparsedMeasure` more strict and update schema readers accordingly
* Update the `SemanticModel` node to match DSI protocol
A lot of the additions are helper functions which we don't actually
use in core. This is a known issue. We're in the process of removing
a fair number of them from the DSI protocol spec. However, in the meantime
we need to implement them to satisfy the protocol unfortunately.
* Make `UnparsedSemanticModel` more strict and update schema readers accordingly
* Changie entry for updating SemanticModel node
* Use contextvar to store and get project_root for path selector method
* Changie
* Modify test to check Path selector with project-dir
* Don't set cv_project_root in base task if no config
* Refactor MetricNode definition to satisfy DSI Metric protocol
* Fix tests involving metrics to have updated properties
* Update UnparsedMetricNode to match new metric yaml spec
* Update MetricParser for new unparsed and parsed MetricNodes
* Remove `rename_metric_attr`
We're intentionally breaking the spec. There will be a separate tool provided
for migrating from dbt-metrics to dbt x metricflow. This bit of code was renaming
things like `type` to `calculation_method`. This is problematic because `type` is
on the new spec, while `calculation_method` is not. Additionally, since we're
intentionally breaking the spec, this function, `rename_metric_attr`, shouldn't be
used for any property renaming.
* Fix tests for Metrics (1.6) changes
* Regenerated v10 manifest schema and associated functional test artifact state
* Remove no longer needed tests
* Skip / comment out tests for metrics functionality that we'll be implementing later
* Begin outputting semantic manifest artifact on every run
* Drop metrics during upgrade_manifest_json if manifest is v9 or before
* Update properties of `minimal_parsed_metric_dict` to match new metric spec
* Add changie entry for metric node breaking changes
* Add semantic model nodes to semantic manifest
* Add dbt-semantic-interfaces as a dependency
With the integration with MetricFlow we're taking a dependency on
`dbt-semantic-interfaces` which acts as the source of truth for
protocols which MetricFlow and dbt-core need to agree on. Additionally
we're hard pinning to 0.1.0.dev3 for now. We plan on having a less
restrictive specification when dbt-core 1.6 hits GA.
* Add implementations of DSI Metadata protocol to nodes.py
* CT-2521: Initial work on adding new SemanticModel node
* CT-2521: Second rough draft of SemanticModels
* CT-2521: Update schema v10
* CT-2521: Update unit tests for new SemanticModel collection in manifest
* CT-2521: Add changelog entry
* CT-2521: Final touches on initial implementation of SemanticModel parsing
* Change name of Metadata class to reduce potential for confusion
* Remove "Replaceable" inheritance, per review
* CT-2521: Rename internal variables from semantic_models to semantic_nodes
* CT-2521: Update manifest schema to reflect change
---------
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* changie
* ADAP-387: Stub materialized view as a materialization (#7211)
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* updating to feature main
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-387: reverting db_api implementation (#7322)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* updating to feature main
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* added tests
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* typo
* moved materialized view tests into adapter test framework
* add enum to relation for comparison in jinja
---------
Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
* ADAP-391: Add configuration change option (#7272)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* move up pre-existing drop of backup
* change relation type to view to be consistent
* add base test case
* fix jinja exception message expression, basic test passing
* align names between postgres and default to same convention
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integration tests
* add on_configuration_change to expected_manifest to pass functional tests
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* moved materialized view tests into adapter test framework
* add alter materialized view macro
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* updated default option for on_configuration_change on NodeConfig
* removed explicit calls to enum values
* add test setup for testing fail config option
* updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name
* setup configuration change framework
* skipped tests that are expected to fail without adapter implementation
* cleaned up log checks
---------
Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
* ADAP-388: Stub materialized view as a materialization - postgres (#7244)
* move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-402: Add configuration change option - postgres (#7334)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* added materialized view stubs and test
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* updated postgres to use the new macros structure
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* init copy of pr 387 to begin 391 implementation
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* remove postgres-specific materialization in favor of core default materialization
* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies
* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integretion tests
* add on_configuration_change to expected_manifest to pass functuional tests
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* revert to current approach for materializations
* added tests
* move materialized view logic into the `/materializations` directory in line with `dbt-core`
* moved default macros in `dbt-core` into `dbt-postgres`
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* moved materialized view tests into adapter test framework
* updated materialized view tests to use adapter test framework
* add alter materialized view macro
* add alter materialized view macro
* change class name, and config setup
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* added in `on_configuration_change` setting
* updated default option for on_configuration_change on NodeConfig
* updated default option for on_configuration_change on NodeConfig
* fixed list being passed as string bug
* removed explicit calls to enum values
* removed unneeded test class
* fixed on_configuration_change to be picked up appropriately
* add test setup for testing fail config option
* remove breakpoint, uncomment tests
* update skip scenario to use empty strings
* update skip scenario to avoid using sql at all, remove extra whitespace in some templates
* push up initial addition of indexes for mv macro
* push slight change up
* reverting alt macro and moving the do create_index call to be more in line with other materializations
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402
# Conflicts:
# core/dbt/contracts/graph/model_config.py
# core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/base.py
# tests/functional/materializations/test_materialized_view.py
* merge feature branch into story branch
* merge feature branch into story branch
* added indexes into the workflow
* fix error in jinja that caused print error
* working on test messaging and skipping tests that might not fit quite into current system
* add drop and show macros for indexes
* add drop and show macros for indexes
* add logic to determine the indexes to create or drop
* pulled index updates through the workflow properly
* convert configuration changes to fixtures, implement index changes into tests
* created Model dataclass for readability, added column to swap index columns for testing
* fixed typo
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-395: Implement native materialized view DDL (#7336)
* changie
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* added materialized view stubs and test
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* updated postgres to use the new macros structure
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* init copy of pr 387 to begin 391 implementation
* updating to feature main
* updating to feature main
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* remove postgres-specific materialization in favor of core default materialization
* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies
* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integretion tests
* add on_configuration_change to expected_manifest to pass functuional tests
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* revert to current approach for materializations
* added tests
* move materialized view logic into the `/materializations` directory in line with `dbt-core`
* moved default macros in `dbt-core` into `dbt-postgres`
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* typo
* moved materialized view tests into adapter test framework
* updated materialized view tests to use adapter test framework
* add alter materialized view macro
* add alter materialized view macro
* added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update
* change class name, and config setup
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* added in `on_configuration_change` setting
* updated default option for on_configuration_change on NodeConfig
* updated default option for on_configuration_change on NodeConfig
* fixed list being passed as string bug
* fixed list being passed as string bug
* removed explicit calls to enum values
* removed explicit calls to enum values
* removed unneeded test class
* fixed on_configuration_change to be picked up appropriately
* add test setup for testing fail config option
* remove breakpoint, uncomment tests
* update skip scenario to use empty strings
* update skip scenario to avoid using sql at all, remove extra whitespace in some templates
* push up initial addition of indexes for mv macro
* push slight change up
* reverting alt macro and moving the do create_index call to be more in line with other materializations
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402
# Conflicts:
# core/dbt/contracts/graph/model_config.py
# core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/base.py
# tests/functional/materializations/test_materialized_view.py
* merge feature branch into story branch
* merge feature branch into story branch
* added indexes into the workflow
* fix error in jinja that caused print error
* working on test messaging and skipping tests that might not fit quite into current system
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395
# Conflicts:
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# plugins/postgres/dbt/include/postgres/macros/adapters.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py
# tests/functional/materializations/test_materialized_view.py
* moved postgres implemention into plugin directory
* update index methods to align with the configuration update macro
* added native ddl to postgres macros
* removed extra docstring
* updated references to View, now references MaterializedView
* decomposed materialization into macros
* refactor index create statement parser, add exceptions for unexpected formats
* swapped conditional to check for positive state
* removed skipped test now that materialized view is being used
* return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test
* add check for indexes in particular for apply on configuration scenario
* removed extra argument
* add materialized views to get_relations / list_relations
* typos in index change logic
* moved full refresh check inside the build sql step
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* removing returns from tests to stop logs from printing
* moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures
* fixed overwrite issue, simplified assertion method
* updated import order to standard
* fixed test import paths
* updated naming convention for proper test collection with the test runner
* still trying to make the test runner happy
* rewrite index updates to use a better source in Postgres
* break out a large test suite as a separate run
* update `skip` and `fail` scenarios with more descriptive results
* typo
* removed call to skip status
* reverting `exceptions_jinja.py`
* added FailFastError back, the right way
* removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports
* removed assumed models in method calls, removed odd insert records and replaced with get row count
* fixed index issue, removed some indirection in testing
* made test more readable
* remove the "apply" from the tests and put it on the base as the default
* generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility
* fixed type to be more generic to accommodate adapters with their own relation types
* fixed all the broken index stuff
* updated on_configuration_change to use existing patterns
* updated on_configuration_change to use existing patterns
* reflected update in tests and materialization logic
* reflected update in tests and materialization logic
* reverted the change to create a config object from the option object, using just the option object now
* reverted the change to create a config object from the option object, using just the option object now
* modelled database objects to support monitoring all configuration changes
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* reverted centralized framework, retained a few reusable base classes
* updated names to be more consistent
* readability updates
* added readme specifying that `relation_configs` only supports materialized views for now
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
* --connection-flag
* Standardize the plugin functions used by DebugTask
* Cleanup redundant code and help logic along.
* Add more output tests to add logic coverage and formatting.
* Code review
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Fix names within functional test
* Changelog entry
* Test for implementation of null-safe equals comparison
* Remove duplicated where filter
* Fix null-safe equals comparison
* Fix tests for `concat` and `hash` by using empty strings () instead of `null`
* Remove macro namespace interpolation
* Include null checks in utils test base
* Add tests for the schema test
* Add tests for this macro
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Honor `--skip-profile-setup` parameter when inside an existing project
* Use project name as the profile name
* Use separate file connections for reading and writing
* Raise a custom exception when no adapters are installed
* Test skipping interactive profile setup when inside a dbt project
* Replace `assert_not_called()` since it does not work
* Verbose CLI argument for skipping profile setup
* Use separate file connections for reading and writing
* Check empty list in a Pythonic manner
* CT-2461: Work toward model deprecation
* CT-2461: Remove unneeded conversions
* CT-2461: Fix up unit tests for new fields, correct a couple oversights
* CT-2461: Remaining implementation and tests for model/ref deprecation warnings
* CT-2461: Changelog entry for deprecation warnings
* CT-2461: Refine datetime handling and tests
* CT-2461: Fix up unit test data
* CT-2461: Fix some more unit test data.
* CT-2461: Fix merge issues
* CT-2461: Code review items.
* CT-2461: Improve version -> str conversion
* Allow missing `profiles.yml` for `dbt deps` and `dbt init`
* Some commands allow the `--profiles-dir` to not exist
* Remove fix to verify that CI tests work
* Allow missing `profiles.yml` for `dbt deps` and `dbt init`
* CI is not finding any installed adapters
* Remove functional test for `dbt init`
* Adding perf testing GHA
* Fixing tigger syntax
* Fixing PR creation issue
* Updating testing var
* Remove unneeded branch names
* Fixing branch naming convention
* Standardizing branch name to var
* Consolidating PR jobs
* Updating inputs and making more readable
* Splitting steps up
* Making some updates here to simplify and update
* Remove tab
* Cleaned up testing TODOs before committing
* Fixing spacing
* Fixing spacing issue
* Create publication.py, various Publication classes, Dependency class
* Load dependencies.yml and the corresponding publication file
* Add "public_nodes" and populate ref_lookup
* resolve_ref working
* Add public nodes to parent and child maps
* Bump manifest version and fix tests, use ModelDependsOn
* Split out PublicationArtifact and PublicationConfig, store public_models
separately
* Store dependencies in publication artifact
* change detection of PublicModel for >= python3.10
* Handle removing references for re-processing if publication has changed
* Handle only changed publication artifacts
* Add some logging events
* Remove duplicate nodes from manifest
* refactor relation_from_relation_name
* Remove duplicate writing of manifest.json
* Add public_nodes to flat_graph
* Move some file name constants to core/dbt/constants.py
* Remove "environment" from ProjectDependency. Add
database/schema/identifier to PublicModel. Update TargetNotFound
exception.
* Include external publication dependencies in publication artifact dependencies
* Remove create_from_relation_name, call create_from_node instead
* Change PublicationArtifactChanged message to debug level
* Make write_publication_artifact a function in parser/manifest.py
* Create fixture to create minimal alternate project (just models)
* develop multi project test case
* Latest version should use un-suffixed alias
* Latest version can be in un-suffixed file
* FYI when unpinned ref to model with prerelease version
* [WIP] Nicer error if versioned ref to unversioned model
* Revert "Latest version should use un-suffixed alias"
This reverts commit 3616c52c1eed7588b9e210e1c957dfda598be550.
* Revert "[WIP] Nicer error if versioned ref to unversioned model"
This reverts commit c9ae4af1cfbd6b7bfc5dcbb445556233eb4bd2c0.
* Define real event for UnpinnedRefNewVersionAvailable
* Update pp test for implicit unsuffixed defined_in
* Add changelog entry
* Fix unit test
* marky feedback
* Add test case for UnpinnedRefNewVersionAvailable event
* Adding a new column is not a breaking contract change
* Add changelog entry
* More structured exception
* same_contract: False if non-breaking changes
* PR feedback: rm build_contract_checksum, more comments
* CT-2317: Reset invocation id in preflight for each dbt command.
* CT-2317: Add unit test for invocation_id behavior.
* CT-2317: Add changelog entry.
* CT-2317: Modify freshness test to ignore invocation_id
* CT-2317: Assign invocation_id before tracking initialization.
* CT-2317: Fix unit test failures and a bunch of other stuff
* CT-2317: Remove checks which make outdated assumptions about invocation_id being stable between runs
* CT-2317: Review tweak, more unit test fixes.
* Removed options for `dbt parse`
* Fix misspellings
* Capitalize JSON when appropriate
* Update help text for --write-json/--no-write-json
* Update help text for --config-dir
* Update help text for --resource-types
* Removed decorators for removed dbt parse options
* Remove `--write-manifest` flag from `parse`
* Remove `--parse-only` flag from `compile`
* Update help text for `dbt list --output`
* Standardize on one line per argument
* Factor 3 from 12 Factor CLI Apps
* Update help text for `dbt --version`
* Standardize capitalization of resource types for `dbt build`
* `debug --config-dir` is a boolean flag
* Update help text for `--version-check`
* Specify `-q` as a conventional alias for `--quiet`
* Update help text for `debug --config-dir`
* Update help text for `debug`
* Treat more dense text blobs as binary for `git grep`
* Update help text for `--version-check`
* Update help text for `--defer`
* Update help text for `--indirect-selection`
* Co-locate log colorization with other log settings
* Update help text for `--log-format*`, `--log-level*`, and `--use-colors*`
* Temporarily re-add option for CI tests
* Remove `--parse-only` flag from `show`
* Remove `--write-manifest` flag from `parse` (again)
* Snapshot strategies: newline for subquery
* add changie output
* add test for snapshot ending in comment
* remove import to be flake8 compliant
* add seed import
* add newlines for flake8 compliance
* typo fix
* Fixing up a test, adding a comment or two
* removed un-needed test fixtures
* removed even more un-needed fixtures, collapsed test to single class
* removed errant breakpoint()
* Fix a little typo
---------
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com>
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: (Almost) complete support for model level constraints
* CT-1922: Fix typo affecting correct model constraint parsing.
* CT-1922: Rework base class for model tests for greater simplicity
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: Revise unit tests for new model-level constraints property
* CT-1922: (Almost) complete support for model level constraints
* first pass
* implement in core
* add proto
* WIP
* resolve errors in columns_spec_ddl
* changelog
* update comment
* move logic over to python
* rename and use enum
* update default constraint_support dict
* generate new proto definition after conflicts
* reorganize code and break warnings into each constraint
* fix postgres constraint support
* remove breakpoint
* convert constraint support to constant
* update postgres
* add to export
* add to export
* regen proto types file
* standardize names
* put back mypy error
* more naming + add back comma
* add constraint support to model level constraints
* update event message and method signature
* rename method
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: Revise unit tests for new model-level constraints property
* CT-1922: (Almost) complete support for model level constraints
* CT-1922: Fix typo affecting correct model constraint parsing.
* CT-1922: Improve whitespace handling
* CT-1922: Render raw constraints to constraint list directly
* make method return consistent
* regenerate proto defn
* update evvent test
* add some code cleanup
---------
Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: Revise unit tests for new model-level constraints property
* CT-1922: (Almost) complete support for model level constraints
* CT-1922: Fix typo affecting correct model constraint parsing.
* CT-1922: Minor code review refinements
* CT-1922: Improve whitespace handling
* CT-1922: Render raw constraints to constraint list directly
* CT-1922: Rework base class for model tests for greater simplicity
* CT-1922: Remove debugging properties. Oops.
* CT-1922: Fix type annotation
* improved first line of error
* added basic printing of yaml and sql cols as columns
* added changie log
* used listed dictionary as input to match columns
* swapped order of col headers for printing
* used listed dictionary as input to match columns
* removed merge conflict text from file
* Touch-ups
* Update log introspection in functional tests
* Update format_column macro. Case insensitive test
* PR feedback: just data_type, not formatted
---------
Co-authored-by: Kyle Kent <kyle.kent321@gmail.com>
* remove trial nodes before building subdag
* add changie
* Update graph.py
remove comment
* further optimize by sorting node search by degree
* change degree to product of in and out degree
* Add tests for logging jinja2.Undefined objects
[CT-2259](https://github.com/dbt-labs/dbt-core/issues/7108) identifies
an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined
object is attempted to be logged. This generally happened in the form
of `{{ log(undefined_variable, info=True) }}`. This commit adding this
test exists for two reasons
1. Ensure we don't have a regression in this going forward
2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3
* Add tests for checking `DBT_ENV_SECRET_`s don't break logging
[CT-1783](https://github.com/dbt-labs/dbt-core/issues/6568) describes
a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all
`{{ log("logging stuff", info=True) }}` invocations break. This commit
adds a test for this for two reasons:
1. Ensure we don't regress to this behavior going forward
2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3
* Add tests ensuring failed event serialization is handled correctly
[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113) states
that failed serialization should result in an exception handling path
which will fire another event instead of raising an exception. This is
hard to test perfectly because the exception handling path for
serialization depending on whether pytest is present. If pytest isn't
present, a new event documentation the failed serialization is fired.
If pytest is present, the failed serialization gets raised as an exception.
Thus this added test ensures that the expected exception is raised and
assumes that the correct event will be fired normally.
* Log warning when event serialization fails in `msg_to_dict`
This commit updates the `msg_to_dict` exception handling path to
fire a warning level event instead of raising an exception.
Truthfully, we're not sure if this exception handling path is even
possible to hit. That's because we recently switched from betterproto
to google's protobuf. However, exception path is the subject of
[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113). Though we
don't think it's actually possible to hit it anymore, we still want
to handle the case if it is.
* Update serialization failure note to be a warn level event in `BaseEvent`
[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113) wants
logging messages about event serialization failure to be `WARNING`
level events. This does that.
* Add changie info for changes
* Add test to check exception handling of `msg_to_dict`
* One argument per line
* Tests for multiple `--select` or `--exclude`
* Allow `--select` and `--exclude` multiple times
* Changelog entry
* MultiOption options must be specified with type=tuple or type=ChoiceTuple
* Testing for `--output-keys` and `--resource-type`
* Validate that any new param with `MultiOption` should also have `type=tuple` (or `ChoiceTuple`) and `multiple=True`
* first pass
* adding tests
* changelog
* split up tests due to order importance
* update test
* add back comment
* rename base test classes
* move sql
* fix test name
* move sql
* test changes to match main
* organize and cleanup fixtures
* more cleanup of tests
* add utility function to EventManager for explicitly adding callbacks
Technically these aren't necessary in their current state. We could instead
have people do `<InstantiatedEventManager>.callbacks.extend(...)` directly.
However, it's not hard to imagine a world wherein extra things need to take
place when a callback is added. Thus abstracting to a utility method
now means that as the implementation of how callbacks are actually added
changes, the invocation to do so can stay the same.
* update `setup_event_logger` to optionally take in callbacks add them to the EventManager
* update preflight decorator to check for and pass along callbacks for event logger setup
* Add `callbacks` to `dbtRunner`
On instantiation of `dbtRunner` one can now provide `callbacks`. These
callbacks are for the `EventLogger`. When `invoke` is called on a `dbtRunner`,
the `callbacks` are added to the cli context object. In the preflight
decorator these callbacks are extracted from the cli context and then
passed to the `setup_event_logger`, finally `setup_event_logger` ensures
the callbacks are added to the global `EVENT_MANAGER`.
* add test to check dbtRunner callbacks get properly set
I believe technically this tests qualifies as more of an integration
test, but no other tests like it currently exist (that I could find
via a cursory search). The `tests/unit/test_dbt_runner.py` seemed like
the most intuitive spot. However, if somewhere else makes sense, I'd be
happy to move it.
* add changie documentation for CT-1928
* Convert simple copy.
* Adjust class names for import.
* adjust test namespacing
* Resolve test error.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* ct-2198: clean up some type names and uses
* CT-2198: Unify constraints and constraints_check properties on columns
* Make mypy version consistently 0.981 (#7134)
* CT 1808 diff based partial parsing (#6873)
* model contracts on models materialized as views (#7120)
* first pass
* rename tests
* fix failing test
* changelog
* fix functional test
* Update core/dbt/parser/base.py
* Update core/dbt/parser/schemas.py
* Create method for env var deprecation (#7086)
* update to allow adapters to change model name resolution in py models (#7115)
* update to allow adapters to change model name resolution in py models
* add changie
* fix newline adds
* move quoting into macro
* use single quotes
* add env DBT_PROJECT_DIR support #6078 (#6659)
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
* Add new index.html and changelog yaml files from dbt-docs (#7141)
* Make version configs optional (#7060)
* [CT-1584] New top level commands: interactive compile (#7008)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* CT-2198: Add changelog entry
* CT-2198: Fix tests which broke after merge
* CT-2198: Add explicit validation of constraint types w/ unit test
* CT-2198: Move access property, per code review
* CT-2198: Remove a redundant macro
* CT-1298: Rework constraints to be adapter-generated in Python code
* CT-2198: Clarify function name per review
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Stu Kilgore <stu.kilgore@dbtlabs.com>
Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
Co-authored-by: Leo Schick <67712864+leo-schick@users.noreply.github.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: FishtownBuildBot <77737458+FishtownBuildBot@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* add protobuf message/class for new CommandCompleted event
For [CT-2049](https://github.com/dbt-labs/dbt-core/issues/6878) we
concluded that we wanted a new event type, [CommandCompleted](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1419718606)
with [four (4) values](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1426118283):
which command was run, whether the command succeeded, the timestamp
that the command finished, and how long the command took. This commit
adds the new event proto defition, the auto generated proto_types, and
the instantiatable even type.
* begin emitting CommandCompleted event in the preflight decorator
The [preflight decorator](4186f99b74/core/dbt/cli/requires.py (L19))
runs at the start of every CLI invocation. Thus is a perfect candidate
for emitting the CommandCompleted event. This is noted in the [dicussion
on CT-2049](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1428643539).
* add CommandCompleted event to event unit tests
* Add: changelog entry
* fire CommandCompleted event reguardless of upstream exceptions
Previously, if `--fail-fast` was specified and an issue was run into
or an unhandled issue became an exception, the CommandCompleted event
would not get fired because at this point in the stack we'd be in
exception thrown handling mode. If an exception does reach this point,
we want to still fire the event and also continue to propogate the
exception. Hence the bare `raise` exists to reraise the caught exception
* Update CommandCompleted event to be a `Debug` level event
We don't actually "always" need this event to be logged. Thus we've
updated it to `Debug` level. [Discussion Context](https://github.com/dbt-labs/dbt-core/pull/7180#discussion_r1139281963)
* Init roadmap
* Rework the top paragraph
* Clean-up the whole thing
* Typos and stuff
* Add a missing word
* Fix typo
* Update "when" note
* Next draft
* Propose rename
* Resolve TODOs, still needs a reread
* Being cute
* Another read through
* Fix sentence fragment
---------
Co-authored-by: Florian Eiden <florian.eiden@dbtlabs.com>
* first pass
* WIP
* add notes/stubs on more pieces
* more work
* more cleanup
* cleanup
* add more cleanup and generalization
* update to use reusable workflow
* add TODO
* Add back initialization events
* Fix log_cache_events. Default stdout logger knows less than it used to
* Add back exception handling events
* Revert "Add back exception handling events"
This reverts commit 26f22d91b660f51f0df6a59f9e5cae16b0ee6fe5.
* Add changelog entry
* Fix test by stringifying dict values
* Add generated CLI API docs
---------
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* part 1 of env var for core team
* add logic to use env vars to generate changelog
* modify version bump to add members via env var
* pull in main and tweak
* add token
* changes for testing
* split step
* remove leading slash
* add version check
* more debugging
* try curl
* try more things
* try more things
* chnage auth
* put back token
* update permissions
* add back fishtown pat
* use new pat
* fix typo
* swap token
* comment out list teams
* change url
* debug path
* add continue
* change core case
* more tweaks
* send output to file
* add file view
* make array
* tweak
* remove []
* add quotes
* add tojson
* add quotes to set
* tweak
* fix id
* tweaks
* more
* more
* remove new lines
* more tweaks
* update to generate changelog
* remove debugging bits
* use central version-bump
* use correct author list
* testing with changelog team automation
* add new token to input
* move secret
* remove testing aspects from workflow
* clean up team logic
* explicitly send secret
* move bumpversion comment
* move comments
* point workflow back tp main
* point to branch for testing
* point back to main
* inherit secrets
* first pass at automating latest branches
* checkout repo first
* fetch all history
* reorg
* debugging
* update test id
* swap lines
* incorporate new branch aciton
* tweak vars
* Formatting
* Changelog entry
* Rename to BaseSimpleSeedColumnOverride
* Better error handling
* Update test to include the BOM test
* Cleanup and formating
* Unused import remove
* nit line
* Pr comments
* update regex to match all iterations
* convert to num to match all adapters
* add comments, remove extra .
* clarify with more comments
* Update .bumpversion.cfg
Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
---------
Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
* Add clearer directions for custom test suite vars in Makefile.
* Fix up PR for review
* Fix erroneous whitespace.
* Fix a spelling error.
* Add documentation to discourage makefile edits but provide override tooling.
* Fix quotation marks. Very strange behavior
* Compact code and verify quotations happy inside bash and python.
* Fold comments into Makefile.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Convert test and make it a bit more pytest-onic
* Ax old integration test.
* Run black on test conversion
* I didn't like how pytest was running the fixture so wrapped it into a closure.
* Merge converted test into persist docs.
* Move persist docs tests to the adapter zone. Prep for adapter tests.
* Fix up test names
* Fix name to be less confusing.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Test converted and reformatted for pytest.
* Ax old versions of 052 test
* Nix the 'os' import and black format
* Change names of models to be more PEP like
* cleanup code
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Convert test and make it a bit more pytest-onic
* Ax old integration test.
* Run black on test conversion
* I didn't like how pytest was running the fixture so wrapped it into a closure.
* Merge converted test into persist docs.
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* init commit for column_types test conversion
* init start of test_column_types.py
* pass tes macros into both tests
* remove alt tests, remove old tests, push up working conversion
* rename base class, move to adapter zone so adapters can use
* typo fix
* Code cleanup and adding stderr to capture dbt
* Debug with --log-format json now prints structured logs.
* Add changelog.
* Move logs into miscellaneous and add values to test.
* nix whitespace and fix log levels
* List will now do structured logging when log format set to json.
* Add a quick None check.
* Add a get guard to class check.
* Better null checking
* The boolean doesn't reflect the original logic but a try-catch does.
* Address some code review comments and get us working again.
* Simplify logic now that we have a namespace object for self.config.args.
* Simplify logic for json log format checking.
* Simplify code for allowing our GraphTest cases to pass while also hiding compile stats from dbt ls/list .
* Simplify structured logging types.
* Fix up boolean logic and simplify via De'Morgan.
* Nix unneeded fixture.
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* CT-1786: Port docs tets to pytest
* Add generated CLI API docs
* CT-1786: Comply with the new style requirements
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* add defer_to_manifest in before_run to fix faulty deferred docs generate
* add a changelog
* add declaration of defer_to_manifest to FreshnessTask and GraphRunnableTask
* fix: add defer_to_manifest method to ListTask
* Re-factor list of YAML keys for hooks to late-render
* Add `pre_` and `post_hook` to list of late-rendered hooks
* Check for non-empty set intersection
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
* Test functional synonymy of `*_hook` with `*-hook`
Test that `pre_hook`/`post_hook` are functionally synonymous with `pre-hook`/`post-hook` for model project config
* Undo bugfix to validate the new test fails
* Revert "Undo bugfix to validate the new test fails"
This reverts commit e83a2be2eb.
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
* add meta attribute to nodeinfo for events
* also add meta to dataclass
* add to unit test to ensure meta is added
* adding functional test to check that meta is passed to nodeinfo during logging
* changelog
* remove used imported
* add tests with non-string keys
* renaming test dict keys
* add non-string value
* resolve failing test
* test additional non-string values
* fix flake8
* Stringify meta dict in node_info
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
* convert the test and fix an error due to a dead code seed
* Get rid of old test
* Remove unfortunately added files. Don't use that *
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Update types.proto
* pre-commit passes
* Cleanup tests and tweak EventLevels
* Put node_info back on SQLCommit. Add "level" to fire_event function.
* use event.message() in warn_or_error
* Fix logging test
* Changie
* Fix a couple of unit tests
* import Protocol from typing_extensions for 3.7
* ✨ adding pre-commit install to make dev
* 🎨 updating format of Makefile and CONTRIBUTING.md
* 📝 adding changelog via changie new
* ✨ adding dev_req to Makefile + docs
* 🎨 remove dev_req from docs, dry makefile
* Align names of `.PHONY` targets with their associated rules
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* starting to move jinja exceptions
* convert some exceptions
* add back old functions for backward compatibility
* organize
* more conversions
* more conversions
* add changelog
* split out CacheInconsistency
* more conversions
* convert even more
* convert parsingexceptions
* fix tests
* more conversions
* more conversions
* finish converting exception functions
* convert more tests
* standardize to msg
* remove some TODOs
* fix test param and check the rest
* add comment, move exceptions
* add types
* fix type errors
* fix type for adapter_response
* remove 0.13 version from message
* pass predicated to merge strategy
* postgres delete and insert
* merge with predicates
* update to use arbitrary list of predicates, not dictionaries, merge and delete
* changie
* add functional test to adapter zone
* comma in test config
* add test for incremental predicates delete and insert postgres
* update test structure for inheritance
* handle predicates config for backwards compatibility
* test for predicates keyword
* Add generated CLI API docs
Co-authored-by: Colin <colin.rogers@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* Remove unneeded SQL compilation attributes from SeedNode
* Fix various places that referenced removed attributes
* Cleanup a few Unions
* More formatting in nodes.py
* Mypy passing. Untested.
* Unit tests working
* use "doc" in documentation unique_ids
* update some doc_ids
* Fix some artifact tests. Still need previous version.
* Update manifest/v8.json
* Move relation_names to parsing
* Fix a couple of tests
* Update some artifacts. snapshot_seed has wrong schema.
* Changie
* Tweak NodeType.Documentation
* Put store_failures property in the right place
* Fix setting relation_name
* update changie to require issue or pr, and allow multiple
* remove extraneous data from changelog files.
* allow for multiple PR/issues to be entered
* update contributing guide
* remove issue number from bot changelogs
* update format of PR
* fix dependency changelogs
* remove extra line
* remove extra lines, tweak contributor wording
* Update CONTRIBUTING.md
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Get running with Python 3.11
* More tests passing, mypy still unhappy
* Upgrade to 3.11, and bump mashumaro
* patch importlib.import_module last
* lambda: Policy() default_factory on include and quote policy
* Add changelog entry
* Put a lambda on it
* Fix text formatting for log file
* Handle variant type return from e.log_level()
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Josh Taylor <joshuataylorx@gmail.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
* feat: add a list of default values to the ctx manager
* tests: dbt.get.config default values
* feat: validate the num of args in config.get
* feat: jinja template for dbt.config.get default values
* docs: changie yaml
* fix:typo on error message
Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
* v0 - new dbt deps type: tarball url
in support of
https://github.com/dbt-labs/dbt-core/issues/4205
* flake8 fixes
* adding max size tarball condition
* clean up imports
* typing
* adding sha1 and subdirectory options; improve logging feedback
sha1: allow user to specify sha1 in packages.yaml, will only install if package matches
subdirectory: allow user to specify subdirectory of package in tarfile, if the package is a non standard structure (like with git subdirectory option)
* simple tests added
* flake fixes
* changes to support tests; adding exceptions; fire_event logging
* new logging events
* tarball exceptions added
* build out tests
* removing in memory tarball test
* update type codes to M - Misc
* adding new events to test_events
* fix spacing for flake
* add retry download code - as used in registry calls
* clean
* remove saving tar in memory inside tarfile object
will hit url multiple times instead
* remove duplicative code after refactor
* black updates
* black formatting
* black formatting
* refactor - no more in-memory tarfile - all as file operations now
- remove tarfile passing, always use tempfile instead
- reorganize system.* functions, removing duplicative code
- more notes on current flow and structure - esp need for pattern of 1) unpack 2) scan for package dir 3) copy to destination.
- cleaning
* cleaning and sync to new tarball code
* cleaning and sync to new tarball code
* requested changes from PR
https://github.com/dbt-labs/dbt-core/pull/4689#discussion_r812970847
* reversions from revision 2
removing sha1 check to simplify/mirror hub install pattern
* simplify/mirror hub install pattern
to simplify/mirror hub install pattern
- removing sha1 check
- supply name/version to act as our 'metadata' source
* simplify/mirror hub install pattern
simplify with goal of mirroring hub install pattern
- supporting subfolders like git packages, and sha1 checks are removed
- existing code from RegistryPinnedPackage (install() and download_and_untar()) performs the operations
- RegistryPinnedPackage install() and download_and_untar() are not currently set up as functions that can be used across classes - this should be moved to dbt.deps.base, or to a dbt.deps.common file - need dbt labs feedback on how to proceed (or leave as is)
* remove revisions, no longer doing package check
* slim down to basic tests
more complex features have been removed (sha1, subfolder) so testing is much simpler!
* fix naming to match hubs behavior
remove version from package folder name
* refactor install and download to upstream PinnedPackage class
i'm on the fence if this is right approach, but seems like most sensible after some thought
* Create Features-20221107-105018.yaml
* fix flake, black, mypy errors
* additional flake/black fixes
* Update .changes/unreleased/Features-20221107-105018.yaml
fix username on changelog
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* change to fstring
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* cleaning - remove comment
* remove comment/question for dbt team
* in support of issuecomment 1334055944
https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334055944
* in support of issuecomment 1334118433
https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334118433
* black fixes; remove debug bits
* remove `.format` & add 'tarball' as version
'tarball' as version so that the temp files format nicely:
[tempfile_location]/dbt_utils_2..tar.gz # old
vs
[tempfile_location]/dbt_utils_1.tarball.tar.gz # current
* port os.path refs in `PinnedPackage._install` to pathlib
* lowercase as per PR feedback
* update tests after removing version arg
goes along with 8787ba41af
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* removed Compiled versions of nodes
* Remove compiled fields from dictionary if not compiled
* check compiled is False instead of attribute existence in env_var
processing
* Update artifacts test (CompiledSnapshotNode did not have SnapshotConfig)
* Changie
* more complicated 'compiling' check in env_var
* Update test_exit_codes.py
* CT-1405: Refactor event logging code
* CT-1405: Add changelog entry
* CT-1405: Add code to protect against using closed streams from past tests.
* CT-1405: Restore unit test which was only failing locally
* CT-1405: Document a hack with issue # to resolve it in the future
* CT-1405: Make black happy
* CT-1405: Get rid of confusing factory function and duplicated function
* CT-1405: Remove unused event from types.proto and auto-gen'd file
* Fix the partial parse path
Partial parse should use project root or it does not resolve to correct path.
Eg. `target-path: ../some/dir/target`, if not ran from root, creates an erroneous folder.
* Run pre-commit
* Changie
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
description:This is an implementation ticket intended for use by the maintainers of dbt-core
title:"[<project>] <title>"
labels:["user docs"]
body:
- type:markdown
attributes:
value:This is an implementation ticket intended for use by the maintainers of dbt-core
- type:checkboxes
attributes:
label:Housekeeping
description:>
A couple friendly reminders:
1. Remove the `user docs` label if the scope of this work does not require changes to https://docs.getdbt.com/docs: no end-user interface (e.g. yml spec, CLI, error messages, etc) or functional changes
2. Link any blocking issues in the "Blocked on" field under the "Core devs & maintainers" project.
options:
- label:I am a maintainer of dbt-core
required:true
- type:textarea
attributes:
label:Short description
description:|
Describe the scope of the ticket, a high-level implementation approach and any tradeoffs to consider
validations:
required:true
- type:textarea
attributes:
label:Acceptance criteria
description:|
What is the definition of done for this ticket? Include any relevant edge cases and/or test cases
validations:
required:true
- type:textarea
attributes:
label:Impact to Other Teams
description:|
Will this change impact other teams? Include details of the kinds of changes required (new tests, code changes, related tickets) and _add the relevant `Impact:[team]` label_.
placeholder:|
Example: This change impacts `dbt-redshift` because the tests will need to be modified. The `Impact:[Adapter]` label has been added.
validations:
required:true
- type:textarea
attributes:
label:Will backports be required?
description:|
Will this change need to be backported to previous versions? Add details, possible blockers to backporting and _add the relevant backport labels `backport 1.x.latest`_
placeholder:|
Example: Backport to 1.6.latest, 1.5.latest and 1.4.latest. Since 1.4 isn't using click, the backport may be complicated. The `backport 1.6.latest`, `backport 1.5.latest` and `backport 1.4.latest` labels have been added.
validations:
required:true
- type:textarea
attributes:
label:Context
description:|
Provide the "why", motivation, and alternative approaches considered -- linking to previous refinement issues, spikes, Notion docs as appropriate
- When to use a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) vs the [GITHUB_TOKEN](https://docs.github.com/en/actions/security-guides/automatic-token-authentication) generated for the action?
The `GITHUB_TOKEN` is used by default. In most cases it is sufficient for what you need.
If you expect the workflow to result in a commit to that should retrigger workflows, you will need to use a Personal Access Token for the bot to commit the file. When using the GITHUB_TOKEN, the resulting commit will not trigger another GitHub Actions Workflow run. This is due to limitations set by GitHub. See [the docs](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#using-the-github_token-in-a-workflow) for a more detailed explanation.
For example, we must use a PAT in our workflow to commit a new changelog yaml file for bot PRs. Once the file has been committed to the branch, it should retrigger the check to validate that a changelog exists on the PR. Otherwise, it would stay in a failed state since the check would never retrigger.
@@ -105,7 +105,7 @@ Some triggers of note that we use:
```
# **what?**
# Describe what the action does.
# Describe what the action does.
# **why?**
# Why does this action exist?
@@ -138,7 +138,7 @@ Some triggers of note that we use:
- Print out all variables you will reference as the first step of a job. This allows for easier debugging. The first job should log all inputs. Subsequent jobs should reference outputs of other jobs, if present.
@@ -158,14 +158,14 @@ Some triggers of note that we use:
echo "The build_script_path: ${{ inputs.build_script_path }}"
echo "The s3_bucket_name: ${{ inputs.s3_bucket_name }}"
echo "The package_test_command: ${{ inputs.package_test_command }}"
# collect all the variables that need to be used in subsequent jobs
- Don’t use external actions for things that can easily be accomplished manually.
- Always read through what an external action does before using it! Often an action in the GitHub Actions Marketplace can be replaced with a few lines in bash. This is much more maintainable (and won’t change under us) and clear as to what’s actually happening. It also prevents any
- Pin actions _we don't control_ to tags.
- Pin actions _we don't control_ to tags.
### Connecting to AWS
- Authenticate with the aws managed workflow
```yaml
- name: Configure AWS credentials from Test account
Include the number of the issue addressed by this PR above if applicable.
PRs for code changes without an associated issue*will not be merged*.
See CONTRIBUTING.md for more information.
Add the `user docs` label to this PR if it will need docs changes. An
issue will get opened in docs.getdbt.com upon successful merge of this PR.
-->
### Description
### Problem
<!---
Describe the Pull Request here. Add any references and info to help reviewers
understand your changes. Include any tradeoffs you considered.
Describe the problem this PR is solving. What is the application state
before this PR is merged?
-->
### Solution
<!---
Describe the way this PR solves the above problem. Add as much detail as you
can to help reviewers understand your changes. Include any alternatives and
tradeoffs you considered.
-->
### Checklist
- [ ] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me
- [ ] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ]I have [opened an issue to add/update docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose), or docs changes are not required/relevant for this PR
- [ ]I have run `changie new` to [create a changelog entry](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-a-changelog-entry)
- [ ]This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [ ]This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions
issue_body:"At a minimum, update body to include a link to the page on docs.getdbt.com requiring updates and what part(s) of the page you would like to see updated."
# This workflow will test all test(s) at the input path given number of times to determine if it's flaky or not. You can test with any supported OS/Python combination.
# This is batched in 10 to allow more test iterations faster.
# **why?**
# Testing if a test is flaky and if a previously flaky test has been fixed. This allows easy testing on supported python versions and OS combinations.
# **when?**
# This is triggered manually from dbt-core.
name:Flaky Tester
on:
workflow_dispatch:
inputs:
branch:
description:'Branch to check out'
type:string
required:true
default:'main'
test_path:
description: 'Path to single test to run (ex:tests/functional/retry/test_retry.py::TestRetry::test_fail_fast)'
type:string
required:true
default:'tests/functional/...'
python_version:
description:'Version of Python to Test Against'
type:choice
options:
- '3.8'
- '3.9'
- '3.10'
- '3.11'
os:
description:'OS to run test in'
type:choice
options:
- 'ubuntu-latest'
- 'macos-latest'
- 'windows-latest'
num_runs_per_batch:
description:'Max number of times to run the test per batch. We always run 10 batches.'
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## Previous Releases
For information on prior major and minor releases, see their changelogs:
3. [Setting up an environment](#setting-up-an-environment)
4. [Running `dbt` in development](#running-dbt-core-in-development)
4. [Running dbt-core in development](#running-dbt-core-in-development)
5. [Testing dbt-core](#testing)
6. [Debugging](#debugging)
7. [Adding a changelog entry](#adding-a-changelog-entry)
7. [Adding or modifying a changelog entry](#adding-or-modifying-a-changelog-entry)
8. [Submitting a Pull Request](#submitting-a-pull-request)
## About this document
@@ -56,7 +56,7 @@ There are some tools that will be helpful to you in developing locally. While th
These are the tools used in `dbt-core` development and testing:
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.7, 3.8, 3.9, and 3.10
- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.8, 3.9, 3.10 and 3.11
- [`pytest`](https://docs.pytest.org/en/latest/) to define, discover, and run tests
- [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting
- [`black`](https://github.com/psf/black) for code formatting
@@ -96,12 +96,15 @@ brew install postgresql
### Installation
First make sure that you set up your `virtualenv` as described in [Setting up an environment](#setting-up-an-environment). Also ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-core` (and its dependencies) with:
First make sure that you set up your `virtualenv` as described in [Setting up an environment](#setting-up-an-environment). Also ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-core` (and its dependencies):
When installed in this way, any changes you make to your local copy of the source code will be reflected immediately in your next `dbt` run.
@@ -110,7 +113,7 @@ When installed in this way, any changes you make to your local copy of the sourc
With your virtualenv activated, the `dbt` script should point back to the source code you've cloned on your machine. You can verify this by running `which dbt`. This command should show you a path to an executable in your virtualenv.
Configure your [profile](https://docs.getdbt.com/docs/configure-your-profile) as necessary to connect to your target databases. It may be a good idea to add a new profile pointing to a local Postgres instance, or a specific test sandbox within your data warehouse if appropriate.
Configure your [profile](https://docs.getdbt.com/docs/configure-your-profile) as necessary to connect to your target databases. It may be a good idea to add a new profile pointing to a local Postgres instance, or a specific test sandbox within your data warehouse if appropriate. Make sure to create a profile before running integration tests.
## Testing
@@ -160,7 +163,7 @@ suites.
#### `tox`
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.7, Python 3.8, Python 3.9, and Python 3.10 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration for these tests in located in `tox.ini`.
[`tox`](https://tox.readthedocs.io/en/latest/) takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example, you can run unit tests for Python 3.8, Python 3.9, Python 3.10 and Python 3.11 checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py38`. The configuration for these tests in located in `tox.ini`.
#### `pytest`
@@ -168,12 +171,10 @@ Finally, you can also run a specific test or group of tests using [`pytest`](htt
* unit tests (`test/unit`&`tests/unit`) don’t need to access a database; "pure Python" tests should be written as unit tests
* functional tests (`test/integration`&`tests/functional`) cover anything that interacts with a database, namely adapter
* *everything in* `test/*`*is being steadily migrated to*`tests/*`
* unit tests (`tests/unit`) don’t need to access a database; "pure Python" tests should be written as unit tests
* functional tests (`tests/functional`) cover anything that interacts with a database, namely adapter
## Debugging
@@ -201,13 +201,21 @@ Here are some general rules for adding tests:
* Sometimes flake8 complains about lines that are actually fine, in which case you can put a comment on the line such as: # noqa or # noqa: ANNN, where ANNN is the error code that flake8 issues.
* To collect output for `CProfile`, run dbt with the `-r` option and the name of an output file, i.e. `dbt -r dbt.cprof run`. If you just want to profile parsing, you can do: `dbt -r dbt.cprof parse`. `pip` install `snakeviz` to view the output. Run `snakeviz dbt.cprof` and output will be rendered in a browser window.
## Adding a CHANGELOG Entry
## Adding or modifying a CHANGELOG Entry
We use [changie](https://changie.dev) to generate `CHANGELOG` entries. **Note:** Do not edit the `CHANGELOG.md` directly. Your modifications will be lost.
Follow the steps to [install `changie`](https://changie.dev/guide/installation/) for your system.
Once changie is installed and your PR is created, simply run `changie new` and changie will walk you through the process of creating a changelog entry. Commit the file that's created and your changelog entry is complete!
Once changie is installed and your PR is created for a new feature, simply run the following command and changie will walk you through the process of creating a changelog entry:
```shell
changie new
```
Commit the file that's created and your changelog entry is complete!
If you are contributing to a feature already in progress, you will modify the changie yaml file in dbt/.changes/unreleased/ related to your change. If you need help finding this file, please ask within the discussion for the pull request!
You don't need to worry about which `dbt-core` version your change will go into. Just create the changelog entry with `changie`, and open your PR against the `main` branch. All merged changes will be included in the next minor version of `dbt-core`. The Core maintainers _may_ choose to "backport" specific changes in order to patch older minor versions. In that case, a maintainer will take care of that backport after merging your PR, before releasing the new version of `dbt-core`.
"in get_relations, a None relation was found in the cache!"
)
raiseNoneRelationFoundError()
returnresults
defclear(self):
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
