* Refactor MetricNode definition to satisfy DSI Metric protocol
* Fix tests involving metrics to have updated properties
* Update UnparsedMetricNode to match new metric yaml spec
* Update MetricParser for new unparsed and parsed MetricNodes
* Remove `rename_metric_attr`
We're intentionally breaking the spec. There will be a separate tool provided
for migrating from dbt-metrics to dbt x metricflow. This bit of code was renaming
things like `type` to `calculation_method`. This is problematic because `type` is
on the new spec, while `calculation_method` is not. Additionally, since we're
intentionally breaking the spec, this function, `rename_metric_attr`, shouldn't be
used for any property renaming.
* Fix tests for Metrics (1.6) changes
* Regenerated v10 manifest schema and associated functional test artifact state
* Remove no longer needed tests
* Skip / comment out tests for metrics functionality that we'll be implementing later
* Begin outputting semantic manifest artifact on every run
* Drop metrics during upgrade_manifest_json if manifest is v9 or before
* Update properties of `minimal_parsed_metric_dict` to match new metric spec
* Add changie entry for metric node breaking changes
* Add semantic model nodes to semantic manifest
* Add dbt-semantic-interfaces as a dependency
With the integration with MetricFlow we're taking a dependency on
`dbt-semantic-interfaces` which acts as the source of truth for
protocols which MetricFlow and dbt-core need to agree on. Additionally
we're hard pinning to 0.1.0.dev3 for now. We plan on having a less
restrictive specification when dbt-core 1.6 hits GA.
* Add implementations of DSI Metadata protocol to nodes.py
* CT-2521: Initial work on adding new SemanticModel node
* CT-2521: Second rough draft of SemanticModels
* CT-2521: Update schema v10
* CT-2521: Update unit tests for new SemanticModel collection in manifest
* CT-2521: Add changelog entry
* CT-2521: Final touches on initial implementation of SemanticModel parsing
* Change name of Metadata class to reduce potential for confusion
* Remove "Replaceable" inheritance, per review
* CT-2521: Rename internal variables from semantic_models to semantic_nodes
* CT-2521: Update manifest schema to reflect change
---------
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* changie
* ADAP-387: Stub materialized view as a materialization (#7211)
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* updating to feature main
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-387: reverting db_api implementation (#7322)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* updating to feature main
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* added tests
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* typo
* moved materialized view tests into adapter test framework
* add enum to relation for comparison in jinja
---------
Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
* ADAP-391: Add configuration change option (#7272)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* move up pre-existing drop of backup
* change relation type to view to be consistent
* add base test case
* fix jinja exception message expression, basic test passing
* align names between postgres and default to same convention
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integration tests
* add on_configuration_change to expected_manifest to pass functional tests
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* moved materialized view tests into adapter test framework
* add alter materialized view macro
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* updated default option for on_configuration_change on NodeConfig
* removed explicit calls to enum values
* add test setup for testing fail config option
* updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name
* setup configuration change framework
* skipped tests that are expected to fail without adapter implementation
* cleaned up log checks
---------
Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
* ADAP-388: Stub materialized view as a materialization - postgres (#7244)
* move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-402: Add configuration change option - postgres (#7334)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* added materialized view stubs and test
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* updated postgres to use the new macros structure
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* init copy of pr 387 to begin 391 implementation
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* remove postgres-specific materialization in favor of core default materialization
* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies
* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integretion tests
* add on_configuration_change to expected_manifest to pass functuional tests
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* revert to current approach for materializations
* added tests
* move materialized view logic into the `/materializations` directory in line with `dbt-core`
* moved default macros in `dbt-core` into `dbt-postgres`
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* moved materialized view tests into adapter test framework
* updated materialized view tests to use adapter test framework
* add alter materialized view macro
* add alter materialized view macro
* change class name, and config setup
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* added in `on_configuration_change` setting
* updated default option for on_configuration_change on NodeConfig
* updated default option for on_configuration_change on NodeConfig
* fixed list being passed as string bug
* removed explicit calls to enum values
* removed unneeded test class
* fixed on_configuration_change to be picked up appropriately
* add test setup for testing fail config option
* remove breakpoint, uncomment tests
* update skip scenario to use empty strings
* update skip scenario to avoid using sql at all, remove extra whitespace in some templates
* push up initial addition of indexes for mv macro
* push slight change up
* reverting alt macro and moving the do create_index call to be more in line with other materializations
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402
# Conflicts:
# core/dbt/contracts/graph/model_config.py
# core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/base.py
# tests/functional/materializations/test_materialized_view.py
* merge feature branch into story branch
* merge feature branch into story branch
* added indexes into the workflow
* fix error in jinja that caused print error
* working on test messaging and skipping tests that might not fit quite into current system
* add drop and show macros for indexes
* add drop and show macros for indexes
* add logic to determine the indexes to create or drop
* pulled index updates through the workflow properly
* convert configuration changes to fixtures, implement index changes into tests
* created Model dataclass for readability, added column to swap index columns for testing
* fixed typo
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-395: Implement native materialized view DDL (#7336)
* changie
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* added materialized view stubs and test
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* updated postgres to use the new macros structure
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* init copy of pr 387 to begin 391 implementation
* updating to feature main
* updating to feature main
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* remove postgres-specific materialization in favor of core default materialization
* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies
* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integretion tests
* add on_configuration_change to expected_manifest to pass functuional tests
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* revert to current approach for materializations
* added tests
* move materialized view logic into the `/materializations` directory in line with `dbt-core`
* moved default macros in `dbt-core` into `dbt-postgres`
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* typo
* moved materialized view tests into adapter test framework
* updated materialized view tests to use adapter test framework
* add alter materialized view macro
* add alter materialized view macro
* added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update
* change class name, and config setup
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* added in `on_configuration_change` setting
* updated default option for on_configuration_change on NodeConfig
* updated default option for on_configuration_change on NodeConfig
* fixed list being passed as string bug
* fixed list being passed as string bug
* removed explicit calls to enum values
* removed explicit calls to enum values
* removed unneeded test class
* fixed on_configuration_change to be picked up appropriately
* add test setup for testing fail config option
* remove breakpoint, uncomment tests
* update skip scenario to use empty strings
* update skip scenario to avoid using sql at all, remove extra whitespace in some templates
* push up initial addition of indexes for mv macro
* push slight change up
* reverting alt macro and moving the do create_index call to be more in line with other materializations
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402
# Conflicts:
# core/dbt/contracts/graph/model_config.py
# core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/base.py
# tests/functional/materializations/test_materialized_view.py
* merge feature branch into story branch
* merge feature branch into story branch
* added indexes into the workflow
* fix error in jinja that caused print error
* working on test messaging and skipping tests that might not fit quite into current system
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395
# Conflicts:
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# plugins/postgres/dbt/include/postgres/macros/adapters.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py
# tests/functional/materializations/test_materialized_view.py
* moved postgres implemention into plugin directory
* update index methods to align with the configuration update macro
* added native ddl to postgres macros
* removed extra docstring
* updated references to View, now references MaterializedView
* decomposed materialization into macros
* refactor index create statement parser, add exceptions for unexpected formats
* swapped conditional to check for positive state
* removed skipped test now that materialized view is being used
* return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test
* add check for indexes in particular for apply on configuration scenario
* removed extra argument
* add materialized views to get_relations / list_relations
* typos in index change logic
* moved full refresh check inside the build sql step
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* removing returns from tests to stop logs from printing
* moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures
* fixed overwrite issue, simplified assertion method
* updated import order to standard
* fixed test import paths
* updated naming convention for proper test collection with the test runner
* still trying to make the test runner happy
* rewrite index updates to use a better source in Postgres
* break out a large test suite as a separate run
* update `skip` and `fail` scenarios with more descriptive results
* typo
* removed call to skip status
* reverting `exceptions_jinja.py`
* added FailFastError back, the right way
* removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports
* removed assumed models in method calls, removed odd insert records and replaced with get row count
* fixed index issue, removed some indirection in testing
* made test more readable
* remove the "apply" from the tests and put it on the base as the default
* generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility
* fixed type to be more generic to accommodate adapters with their own relation types
* fixed all the broken index stuff
* updated on_configuration_change to use existing patterns
* updated on_configuration_change to use existing patterns
* reflected update in tests and materialization logic
* reflected update in tests and materialization logic
* reverted the change to create a config object from the option object, using just the option object now
* reverted the change to create a config object from the option object, using just the option object now
* modelled database objects to support monitoring all configuration changes
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* reverted centralized framework, retained a few reusable base classes
* updated names to be more consistent
* readability updates
* added readme specifying that `relation_configs` only supports materialized views for now
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
* CT-2461: Work toward model deprecation
* CT-2461: Remove unneeded conversions
* CT-2461: Fix up unit tests for new fields, correct a couple oversights
* CT-2461: Remaining implementation and tests for model/ref deprecation warnings
* CT-2461: Changelog entry for deprecation warnings
* CT-2461: Refine datetime handling and tests
* CT-2461: Fix up unit test data
* CT-2461: Fix some more unit test data.
* CT-2461: Fix merge issues
* CT-2461: Code review items.
* CT-2461: Improve version -> str conversion
* Create publication.py, various Publication classes, Dependency class
* Load dependencies.yml and the corresponding publication file
* Add "public_nodes" and populate ref_lookup
* resolve_ref working
* Add public nodes to parent and child maps
* Bump manifest version and fix tests, use ModelDependsOn
* Split out PublicationArtifact and PublicationConfig, store public_models
separately
* Store dependencies in publication artifact
* change detection of PublicModel for >= python3.10
* Handle removing references for re-processing if publication has changed
* Handle only changed publication artifacts
* Add some logging events
* Remove duplicate nodes from manifest
* refactor relation_from_relation_name
* Remove duplicate writing of manifest.json
* Add public_nodes to flat_graph
* Move some file name constants to core/dbt/constants.py
* Remove "environment" from ProjectDependency. Add
database/schema/identifier to PublicModel. Update TargetNotFound
exception.
* Include external publication dependencies in publication artifact dependencies
* Remove create_from_relation_name, call create_from_node instead
* Change PublicationArtifactChanged message to debug level
* Make write_publication_artifact a function in parser/manifest.py
* Create fixture to create minimal alternate project (just models)
* develop multi project test case
* Latest version should use un-suffixed alias
* Latest version can be in un-suffixed file
* FYI when unpinned ref to model with prerelease version
* [WIP] Nicer error if versioned ref to unversioned model
* Revert "Latest version should use un-suffixed alias"
This reverts commit 3616c52c1eed7588b9e210e1c957dfda598be550.
* Revert "[WIP] Nicer error if versioned ref to unversioned model"
This reverts commit c9ae4af1cfbd6b7bfc5dcbb445556233eb4bd2c0.
* Define real event for UnpinnedRefNewVersionAvailable
* Update pp test for implicit unsuffixed defined_in
* Add changelog entry
* Fix unit test
* marky feedback
* Add test case for UnpinnedRefNewVersionAvailable event
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: Revise unit tests for new model-level constraints property
* CT-1922: (Almost) complete support for model level constraints
* CT-1922: Fix typo affecting correct model constraint parsing.
* CT-1922: Minor code review refinements
* CT-1922: Improve whitespace handling
* CT-1922: Render raw constraints to constraint list directly
* CT-1922: Rework base class for model tests for greater simplicity
* CT-1922: Remove debugging properties. Oops.
* CT-1922: Fix type annotation
* ct-2198: clean up some type names and uses
* CT-2198: Unify constraints and constraints_check properties on columns
* Make mypy version consistently 0.981 (#7134)
* CT 1808 diff based partial parsing (#6873)
* model contracts on models materialized as views (#7120)
* first pass
* rename tests
* fix failing test
* changelog
* fix functional test
* Update core/dbt/parser/base.py
* Update core/dbt/parser/schemas.py
* Create method for env var deprecation (#7086)
* update to allow adapters to change model name resolution in py models (#7115)
* update to allow adapters to change model name resolution in py models
* add changie
* fix newline adds
* move quoting into macro
* use single quotes
* add env DBT_PROJECT_DIR support #6078 (#6659)
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
* Add new index.html and changelog yaml files from dbt-docs (#7141)
* Make version configs optional (#7060)
* [CT-1584] New top level commands: interactive compile (#7008)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* CT-2198: Add changelog entry
* CT-2198: Fix tests which broke after merge
* CT-2198: Add explicit validation of constraint types w/ unit test
* CT-2198: Move access property, per code review
* CT-2198: Remove a redundant macro
* CT-1298: Rework constraints to be adapter-generated in Python code
* CT-2198: Clarify function name per review
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Stu Kilgore <stu.kilgore@dbtlabs.com>
Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
Co-authored-by: Leo Schick <67712864+leo-schick@users.noreply.github.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: FishtownBuildBot <77737458+FishtownBuildBot@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
