* Update dbt_project.yml jsonschema spec to handle nested config defs
Additionally adds some more cloud configs
* Update schema files jsonschema definition to not have `overrides` for sources
Additionally add some cloud definitions
* Add changie doc
* Update happy_path fixture to include nested config specifations in dbt_project.yml
* First draft of SourceOverrideDeprecation warning.
* Refinements and test
* Back out unneeded change`
* Fix unit test.
* add changie doc
* Bump minimum dbt-protos to 1.0.335
---------
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* Stop dynamically setting ubuntu version for `main.yml` and structured logging actions
These actions are important to run on community PRs. However these workflows
use `on: pull_request` instead of `on: pull_request_target`. That is intentional,
as `on: pull_request` doesn't give access to variables or secrets, and we need
to keep it that way for security purposes. The these actions were trying to access
a variable, which they don't have access to. This was a nicety for us, because
sometimes we'd delay moving to github's `ubuntu-latest`. However, the security
concern is more important, and thus we lose the variable for these workflows.
* Change `runs_on` of `artifact-reviews.yml`
* Stop dynamically setting mac and windows versions in main.yml
* Revert "bump dbt-common (#11640)"
This reverts commit c6b7655b65.
* update freshness model config handling
* lower case all columns when processing unit test results
* add changelog
* swap .columns for .column_names
* use rename instead of select api for normalizing agate table column casing
* Add helper to validate model configs via jsonschema
* Store jsonschemas as module vars instead of reloading everytime
Every time we were calling a jsonschema validation, we were _reloading_
from file the underlying jsonschema. As a one off, this isn't too costly.
However, for large projects it starts to add up. By only loading each json
schema once we can save a lot of time. Calling one of the functions which
loads a jsonschema 10,000 times was costing ~3.7215 seconds. By switching
to this module var paradigm we reduced that to ~0.3743 seconds.
* Begin validating configs from model `.sql` files
It was a bit of a hunt to figure out where to do this. We couldn't do
the validating in `calculate_node_config` because that function is called
4 times per node (which is an issue of itself, but out of scope for this
work). We also couldn't do the validation where `_config_call_dict` is set
because it turns out there are multiple avenues for setting
`_config_call_dict`, which is a fun rabbit hole.
* Ensure .sql configs are validated only once
It turns out that that `update_parsed_node_config` can potentially be
called twice per model. It'll be called from either `ModelParser.render_update`
or `ModelParser.populate`, and it can additionally be called from
`PatchParser.patch_node_config` if there is a .yml definition for the
model. We only want to validate the config once, and we aren't guaranteed
to have a `PatchParser` if there is no patch for the model. Thus, we've
updated `ModelParser.populate` and `ModelParser.render_update` to
request the config validation (which by default doesn't run unless requested).
* Split out the model config specific validation from general jsonschema validation
We're validating model configs from sql files via a subschema of the main
resources jsonschema, different case logic for detecting the different
types of deprecation warnings present. Thus `validate_model_config` cannot
call `jsonschema_validate`. We could have had both logic paths exist in
`jsonschema_validate`, but it would have added another later of if/elses
and bloated the function substantially.
* Handle additional properties of sub config objects
* Give better key path information for .sql config jsonschema issues
* Add tests for validate_model_config
* Add changie doc
* Fix jsonschemas unittests to avoid catching irrelevant issues
* Revert "bump dbt-common (#11640)"
This reverts commit c6b7655b65.
* update freshness model config handling
* lower case all columns when processing unit test results
* add changelog
* swap .columns for .column_names
* Loosen pydantic maximum to <3 (allowing for pydantic 2)
* Add an internal pydantic shim for getting pydantic BaseSettings reguardless of pydantic v1 vs v2
* Add changie doc
In 1.10.0 we began utilizing `jsonschema._keywords`. However, the submodule
`_keywords` wasn't added until jsonschema `4.19.1` which came out September
20th, 2023. Our jsonschema requirement was being set transitively via
dbt-common as `>=4.0,<5`. This mean people doing a _non_ fresh install of
dbt-core `1.10.0` could end up with a broken system if their existing
jsonschema dependency was anywhere in the range `>=4.0,<4.19.1`. By bumping the
minimum jsonschema version we make it such that anyone install dbt-core 1.10.1 will
automatically get there jsonschema updated (assuming they don't have an exclusionary
pin)
* Begin testing that model freshness can't be set as a top level model property
* Remove ability to specify freshness as top level property of models
* Add come comments to calculate_node_config for better readability
* Drop `freshness` as a top level property of models, and let `patch_node_config` handle merging config freshness
Model freshness hasn't been released in a minor release yet, not been documented. Thus
it is safe to remove the top level property of freshness on models. Freshness will instead
be set, and gotten, from the model config. Additionally our way of calculating the
config model freshness only got the top level `+freshness` from dbt_project.yml (ignoring
any path specific definitions). By instead using the built in `calculate_node_config` (which
is eventually called by `patch_node_config`), we get all path specific freshness config handling
and it also handles the precedence of `dbt_project.yml` specification, schema file specification,
and sql file specification.
* add changie doc
* Ensure source node `.freshness` is equal to node's `.config.freshness`
* Default source config freshness to empty spec if no freshenss spec is given
* Update contract tests for source nodes
* Ensure `build_after` is present in model freshness in parsing, otherwise skip freshness definition
* add freshness model config test
* add changelog
---------
Co-authored-by: Colin <colin.rogers@dbtlabs.com>
* Handle explicit setting of null for source freshness config
* Abstract out the creation of the target config
This is useful because it makes that portion of code more re-usable/portable
and makes the work we are about to do easier.
* Fix bug in `merge_source_freshness` where empty freshness was preferenced over `None`
The issue was that during merging of freshnesses, an "empty freshness", one
where all values are `None`, was being preferenced over `None`. This was
problematic because an "empty freshness" indicates that a freshness was not
specified at that level. While `None` means that the freshness was _explicitly_
set to `None`. As such we should preference the thing that was specifically set.
* Properly get dbt_project defined freshness and don't merge with schema defined freshness
Previously we were only getting the "top level" freshness from the
dbt_project.yaml. This was ignoring freshness settings for the direct,
source, and table set in the dbt_project.yaml. Additionally, we were
merging the dbt_project.yaml freshness into the schema freshness. Long
term this merging would be desireably, however before we do that we need
to ensure freshness at diffrent levels within the dbt_project.yml get
properly merged (currently the different levels clobber each other). Fixing
that is a larger issue though. So for the time being, the schema defintion
of freshness will clobber any dbt_project.yml definition of freshness.
* Add changie doc
* Fix whitespace to make code quality happy
* Set the parsed source freshness to an empty FreshnessThreshold if None
This maintains backwards compatibility
* Revert "bump dbt-common (#11640)"
This reverts commit c6b7655b65.
* add file_format as a top level config in CatalogWriteIntegrationConfig
* add changelog
* Clean up changelog on main
* Bumping version to 1.11.0a1
* Code quality cleanup
* add old changelogs
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Add a robust schema definition of singular test to happy path fixture
* Add generic tests to happy path fixture
* Add unit tests to happy path fixture
* Fix data test + unit test happy path fixtures so they're valid
* Fix test_list.py for data test + unit test happy path fixture
* Fixup issues due to imperfect merge
* Drop generic data test definition style that we don't want to support from happy path fixture
* Add data test attributes to a pre-existing data test type
* Fix test_list.py again
* Don't forget to normalize in test_list.py
* Include event name in msg of deprecation warning for all recently added deprecations
* Add behavior flag for gating inclusion of event name in older deprecation messages
* Conditionally append event name to older deprecation events depending on behavior flag
* Add changie doc
* Migrate to `WarnErrorOptionsV2` and begin using `error` and `warn` as primary config keys
* Update tests using `WarnErrorOptions` to use `error` and `warn` terminology
* Begin emitting deprecation warning when include/exclude terminology is used with WarnErrorOptions
* bump minimum of dbt-protos
* Add test for new WarnErrorOptions deprecation
* add changie doc
* Fix test_warn_error_options.py tests
* Fix test_singular_tests.py tests
* Add WOEIncludeExcludeDeprecation to test_events.py serialization test
