Compare commits

...

96 Commits

Author SHA1 Message Date
Github Build Bot
c723cb1f7a Bumping version to 1.7.8 and generate changelog 2024-02-14 20:24:42 +00:00
Gerda Shank
6517c75e7e [Backport 1.7.latest] Store node_info in GenericExceptionOnRun logging event (#9568)
* Add node_info to GenericExceptionOnRun, InternalErrorOnRun &
SQLRunnerException

* Changie

* Formatting
2024-02-14 11:06:36 -05:00
Gerda Shank
f95cf55d33 [Backport 1.7.latest] Set query headers when manifest is passed in to dbtRunner (#9569) 2024-02-14 09:51:56 -05:00
github-actions[bot]
ad37fc9534 Preserve model constraints when versioning models (#9539) (#9541)
(cherry picked from commit 2b6e2e18df)

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
2024-02-12 10:17:18 -05:00
FishtownBuildBot
169972f480 [Automated] Merged prep-release/1.7.7_7745564565 into target 1.7.latest during release process 2024-02-01 12:38:17 -06:00
Github Build Bot
7b96509409 Bumping version to 1.7.7 and generate changelog 2024-02-01 18:00:42 +00:00
Michelle Ark
cfbba81938 TestGenerateCatalogWithExternalNodes (#9456) (#9483)
* TestGenerateCatalogWithExternalNodes (#9456)

(cherry picked from commit 06e55bb93e)

* Flip logic in `packages_for_node` to remove error case

By flipping the logic from `not in` to `in` we can drop the exception
and instead default to the model runtime config when the package isn't
found. We're still trying to grok if there will be any fallout from this.
The tests all pass, but that doesn't guarantee nothing bad will happen.

---------

Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
2024-01-31 13:32:50 -05:00
Michelle Ark
1391363d53 Fix source seed selection docs generate (#9454) (#9493)
(cherry picked from commit 719a50cc91)
2024-01-30 18:23:10 -05:00
github-actions[bot]
0d8e4af2f0 [Backport 1.7.latest] Pin pytest in dev-requirements.txt (#9475)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2024-01-30 16:33:26 +00:00
FishtownBuildBot
e9ee761194 [Automated] Merged prep-release/1.7.6_7658416467 into target 1.7.latest during release process 2024-01-25 12:26:54 -06:00
Github Build Bot
5ec34cd4cc Bumping version to 1.7.6 and generate changelog 2024-01-25 17:48:30 +00:00
github-actions[bot]
bc494bbe80 [Backport 1.7.latest] fix retry as CLI (#9448)
(cherry picked from commit 50b85a0b01)

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2024-01-24 16:29:01 -08:00
github-actions[bot]
3f3dda48f7 Handle unknown type_code for model contracts (#8887) (#9409) 2024-01-19 15:08:10 -06:00
FishtownBuildBot
1f98991376 [Automated] Merged prep-release/1.7.5_7572664948 into target 1.7.latest during release process 2024-01-18 17:42:32 +01:00
Github Build Bot
5f5ddd27ac Bumping version to 1.7.5 and generate changelog 2024-01-18 16:04:29 +00:00
Chenyu Li
6e331833b0 Backport 9328 to 1.7.latest (#9391)
* Fix full-refresh and vars for retry (#9328)

Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
(cherry picked from commit 1e4286a62d)

* pr feedback

* Update requires.py
2024-01-17 13:27:46 -08:00
FishtownBuildBot
d338b3e065 [Automated] Merged prep-release/1.7.4_7213813158 into target 1.7.latest during release process 2023-12-14 15:28:18 -05:00
Github Build Bot
9f0bcf5666 Bumping version to 1.7.4 and generate changelog 2023-12-14 19:50:11 +00:00
Quigley Malcolm
d65179b24e [Backport 1.7.latest] Update parser to support conversion metrics #9173 (#9255)
* Move minimum DSI version to 0.4.2

We're backporting a feature "conversion metrics" to 1.7. Conversion
metrics don't exist in DSI < 0.4.2 which is problematic if we allow
for those versions. This ensures that those who are on a version of
1.7 that supports conversion metrics will also have the requisit version
of DSI.

* added ConversionTypeParams classes

* updated parser for ConversionTypeParams

* added step to populate input_measure for conversion metrics

* added tests

* added changelog

* Regenerate v11 manifest jsonschema to include conversion metrics definition

* Regenerate v11 manifest test artifact for testing version compatability

---------

Co-authored-by: Will Deng <william.deng14@gmail.com>
2023-12-11 10:37:06 -08:00
Quigley Malcolm
0ef8638ab4 [Backport 1.7.latest] Fix ensuring we produce valid jsonschema artifacts for manifest, catalog, sources, and run-results (#9229)
* Drop `all_refs=True` from jsonschema-ization build process

Passing `all_refs=True` makes it so that Everything is a ref, even
the top level schema. In jsonschema land, this essentially makes the
produced artifact not a full schema, but a fractal object to be included
in a schema. Thus when `$id` is passed in, jsonschema tools blow up
because `$id` is for identifying a schema, which we explicitly weren't
creating. The alternative was to drop the inclusion of `$id`. Howver, we're
intending to create a schema, and having an `$id` is recommended best
practice. Additionally since we were intending to create a schema,
not a fractal, it seemed best to create to full schema.

* Explicity produce jsonschemas using DRAFT_2020_12 dialect

Previously were were implicitly using the `DRAFT_2020_12` dialect through
mashumaro. It felt wise to begin explicitly specifying this. First, it
is closest in available mashumaro provided dialects to what we produced
pre 1.7. Secondly, if mashumaro changes its default for whatever reason
(say a new dialect is added, and mashumaro moves to that), we don't want
to automatically inherit that.

* Begin including schema dialect specification in produced jsonschema

In jsonschema's documentation they state
> It's not always easy to tell which draft a JSON Schema is using.
> You can use the $schema keyword to declare which version of the JSON Schema specification the schema is written to.
> It's generally good practice to include it, though it is not required.

and

> For brevity, the $schema keyword isn't included in most of the examples in this book, but it should always be used in the real world.

Basically, to know how to parse a schema, it's important to include what
schema dialect is being used for the schema specification. The change in
this commit ensures we include that information.

* Add change documentation for jsonschema schema production fix

* Regenerate dbt jsonschemas with fixed mashumaro jsonschema production process

Specifically we regenerated
* catalog v1
* manifest v11
* run-results v5
* sources v3
using the command `scripts/collect-artifact-schema.py --path schemas`
2023-12-07 12:28:34 -08:00
FishtownBuildBot
fc2d16fd34 [Automated] Merged prep-release/1.7.3_7034454600 into target 1.7.latest during release process 2023-11-29 10:42:33 -05:00
Github Build Bot
de35686532 Bumping version to 1.7.3 and generate changelog 2023-11-29 15:04:49 +00:00
github-actions[bot]
cf3714d2c1 [Backport 1.7.latest] Fix #9119: Get sources working again in dbt docs generate (#9163)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-11-29 14:45:40 +00:00
Jeremy Cohen
09f5bb3dcf Backport #9147 to 1.7.latest (#9156)
* Fixups for deps lock file (#9147)

* Update git revision with commit SHA

* Use PackageRenderer for lock file

* add unit tests for git and tarball packages

* deepcopy unrendered_packages_data before iteration, fix remaining tests

* Add functional tests

* Add changelog entries

* Assert one more

---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>

* Restore warning on unpinned git packages

---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2023-11-29 08:40:54 +01:00
FishtownBuildBot
f94bf2bba4 [Automated] Merged prep-release/1.7.2_6894976289 into target 1.7.latest during release process 2023-11-16 14:11:56 -05:00
Github Build Bot
ae75cc3a62 Bumping version to 1.7.2 and generate changelog 2023-11-16 18:34:17 +00:00
Emily Rockman
1e45a751a5 roll back agate pin (#9106) 2023-11-16 12:29:54 -06:00
github-actions[bot]
b5885da54e [Backport 1.7.latest] [IDE] also treat SystemExit exception as an interrupt. (#9042)
* During node execution, also treat SystemExit as an interrupt. (#8994)

IDE worker process raises SystemExit in multiple scenarios, including user abort of a command.

(cherry picked from commit 931b2dbe40)

* Add test asserting GraphRunnableTasks attempt to cancel connections on SystemExit (#9101)

* Add test asserting GraphRunnableTasks attempt to cancel connections on SystemExit

* Add test asserting GraphRunnableTasks attempt to cancel connections on KeyboardInterrupt

* Add test asserting GraphRunnableNode doesn't try to cancel connections on generic Exception

---------

Co-authored-by: Ben Mosher <ben.mosher@dbtlabs.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
2023-11-16 08:02:08 -08:00
github-actions[bot]
73ebe53273 Support hierarchical config setting for SavedQueryExport configs (#9065) (#9074)
* Add test asserting `SavedQuery` configs can be set from `dbt_project.yml`

* Allow extraneous properties in Export configs

This brings the Export config object more in line with how other config
objects are specified in the unparsed definition. It allows for specifying
of extra configs, although they won't get propagate to the final config.

* Add `ExportConfig` options to `SavedQueryConfig` options

This allows for specifying `ExportConfig` options at the `SavedQueryConfig` level.
This also therefore allows these options to be specified in the dbt_project.yml
config. The plan in the follow up commit is to merge the `SavedQueryConfig` options
into all configs of `Exports` belonging to the saved query.

There are a couple caveots to call out:
1. We've used `schema` instead of `schema_name` on the `SavedQueryConfig` despite
it being called `schema_name` on the `ExportConfig`. This is because need `schema_name`
to be the name of the property on the `ExportConfig`, but `schema` is the user facing
specification.
2. We didn't add the `ExportConfig` `alias` property to the `SavedQueryConfig` This
is because `alias` will always be specific to a single export, and thus it doesn't
make sense to allow defining it on the `SavedQueryConfig` to then apply to all
`Exports` belonging to the `SavedQuery`

* Begin inheriting configs from saved query config, and transitively from project config

Export configs will now inherit from saved query configs, with a preference
for export config specifications. That is to say an export config will inherity
a config attr from the saved query config only if a value hasn't been supplied
on the export config directly. Additionally because the saved query config has
a similar relationship with the project config, exports configs can inherit
from the project config (again with a preference for export config specifications).

* Correct conditional in export config building for map schema to schema_name

I somehow wrote a really weird, but also valid, conditional statement. Previously
the conditional was
```
if combined.get("schema") is not combined.get("schema_name") is None:
```
which basically checked whether `schema` was a boolean that didn't match
the boolean of whether `schema_name` was None. This would pretty much
always evaluate to True because `schema` should be a string or none, not
a bool, and thus would never match the right hand side. Crazy. It has now
been fixed to do the thing we want to it to do. If `schema` isn't `None`,
and `schema_name` is `None`, then set `schema_name` to have the value of
`schema`.

* Update parameter names in `_get_export_config` to be more verbose

(cherry picked from commit c2f7d75e9e)

Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
2023-11-15 15:34:36 -08:00
github-actions[bot]
2a9c3689a4 test pinning ddtrace (#9090) (#9092)
(cherry picked from commit 3902137dfc)

Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-11-15 16:49:14 -06:00
github-actions[bot]
67d8ce398f [Backport 1.7.latest] Fix dbt deps failing on tarballs (#9075) 2023-11-15 12:07:13 -08:00
FishtownBuildBot
7eb6cdbbfb [Automated] Merged prep-release/1.7.1_6789802369 into target 1.7.latest during release process 2023-11-07 15:52:03 -05:00
Github Build Bot
6ba3dc211c Bumping version to 1.7.1 and generate changelog 2023-11-07 20:15:09 +00:00
github-actions[bot]
ce86238389 Support new agate Integer data_type in adapter code (#9004) (#9027) 2023-11-07 14:56:48 -05:00
github-actions[bot]
7d60d6361e Fix lock for git subduer (#9019) (#9028)
* wip

* add tests

* changelog

* nits

* pr feedback

* nits

(cherry picked from commit 01d481bc8d)

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-11-07 14:30:50 -05:00
github-actions[bot]
bebd6ca0e1 Use MANIFEST.in to identify package data, allows recursive include (#9021) (#9026)
* changelog

* use MANIFEST.in to identify package data

(cherry picked from commit 839c720e91)

Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
2023-11-07 14:05:43 -05:00
Peter Webb
66cb5a0e7c Fix back compat for run_results pre-v5 (#9009) (#9017)
* Fix back compat for run_results pre-v5

* Add type annotations

* Add functional testing

* Add inline annotations

* Add changelog entry.

* Consolidate upgrade_schema_version + upgrade_run_results_json

* Restore accidentally reverted test cases

* Pre-commit fixups

---------

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-11-06 19:26:53 -05:00
FishtownBuildBot
2401600e57 [Automated] Merged prep-release/1.7.0_6734243388 into target 1.7.latest during release process 2023-11-02 11:43:59 -04:00
Github Build Bot
a8fcf77831 Bumping version to 1.7.0 and generate changelog 2023-11-02 14:53:42 +00:00
Peter Webb
0995825793 Make relation filtering None-tolerant for maximal flexibility across adapters. (#8975) (#8978) 2023-11-01 18:14:23 -04:00
Quigley Malcolm
eeaaec4de9 DSI 0.4.0 and Saved Query Exports (#8950) (#8973) 2023-11-01 12:04:34 -07:00
github-actions[bot]
1cbab8079a ADAP-974: Fix issue where materialized views were not showing up in catalog queries (#8945) (#8963)
* changelog
* write test case demonstrating the issue
* update catalog query to reflect materialized views

(cherry picked from commit bb21403c9e)

Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
2023-11-01 14:29:53 -04:00
github-actions[bot]
333d793cf0 add a no-op runner for saved_query (#8937) (#8958)
(cherry picked from commit 211392c4a4)

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-11-01 09:06:07 -07:00
Peter Webb
b6f0eac2cd Backport Catalog Fix to 1.7.latest (#8953)
* Fix issues around new get_catalog_by_relations macro (#8856)

* Fix issues around new get_catalog_by_relations macro

* Add changelog entry

* Fix unit test.

* Additional unit testing

* Fix cased comparison in catalog-retrieval function (#8940)

* Fix cased comparison in catalog-retrieval function.

* Fix cased comparison in catalog-retrieval function.
2023-10-31 16:24:27 -04:00
github-actions[bot]
5e13698b2b [Backport 1.7.latest] Fix #8836: Add version to fqn when version==0 (#8922)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
2023-10-26 13:38:44 -05:00
github-actions[bot]
12c1cbbc64 Contract enforcement on temporary tables (#8889) (#8902)
* add test

* fix test

* first pass with constraint error

* add back column checks for temp tables

* changelog

* Update .changes/unreleased/Fixes-20231024-145504.yaml

(cherry picked from commit 98310b6612)

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-10-26 09:26:57 -05:00
github-actions[bot]
ced70d55c3 Fix partial parsing issue not working for changing semantic model name (#8865) (#8878)
* fix

* test

* changelog

(cherry picked from commit 35f46dac8c)

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-10-25 11:23:48 -07:00
FishtownBuildBot
1baebb423c [Automated] Merged prep-release/1.7.0rc1_6495873283 into target main during release process 2023-10-12 09:50:47 -04:00
Github Build Bot
462df8395e Bumping version to 1.7.0rc1 and generate changelog 2023-10-12 12:53:26 +00:00
Quigley Malcolm
35f214d9db Support dbt-semantic-interfaces 0.3.0 (#8820)
* changie doc for DSI 0.3.0 upgrade

* Gracefully handle v10 metric filters

* Fix iteration over metrics in `upgrade_v10_metric_filters`

* Update previous manifest version test fixtures to have more expressive metrics

* Regenerate the test v10 manifest artifact using the more expressive metrics from 904cc1ef

To do this I cherry-picked 904cc1ef onto my local 1.6.latest branch,
had the test regenerate the test v10 manifest artifact, and then over
wrote the test v10 manifest artifact on this branch (cherry-picking it
across the branches didn't work, had to copy paste :grimmace:)

* Regenerate test v11 manifest artifact using the fixture changes in 904cc1ef

* Update `upgrade_v10_metric_filters` to handled disabled metrics

Regenerating the v10 and v11 test manifest artifacts uncovered an
issue wherein we weren't handling disabled metrics that need to
get upgraded. This commit fixes that. Additionally, the
`upgrade_v10_metric_filters` was getting a bit unwieldy, so I broke
extracted the abstracted sub functions.

* Fix `test_backwards_compatible_versions` test

When we regenerated the v10 test manifest artifact, it started having
the `metricflow_time_sine` model, and it didn't previously. This caused
`test_backwards_compatible_versions` to start failing because it was
no longer identified as having modified state for v10. The test has
been altered accordingly
2023-10-11 16:46:26 -07:00
Quigley Malcolm
af0cbcb6a5 Add SavedQuery nodes (#8798)
* Bump to dbt-semantic-interfaces 0.3.0b1

* Update import path of `WhereFilterParser` from `dbt-semantic-interfaces`

In 0.3.x of `dbt-semantic-intefaces` the location of the WhereFilterParser
moved to be grouped in with a bunch of new adjacent code. As such,
we needed to correct our import path of it.

* Create basic `SavedQuery` node type based on `SavedQuery` protocol from DSI

* Add ability to add SavedQueries to the manifest

* Define unparsed SavedQuery node

* Begin parsing saved_query objects to manifest

* Skip jinja rendering of `SavedQuery.where` property

* Begin propagating `SavedQueries` on the manifest to the semantic manifest

* Add tests for basic saved query parsing

* Add custom pluralization handling of SavedQuery node type

* Add a config subclass to SavedQuery node

* Move the SavedQuery node to nodes.py

Unfortunately things are a bit too intertwined currently for SavedQuery
to be in it's own file. We need to add the SavedQuery node to the
GraphMemberNode, unfortunately with SavedQuery in it's own file,
importing it would have caused a circular dependency. We'll need
to separately come in and split things up as a cleanup portion of
work.

* Add basic plumbing of saved query configs to projects

* Add basic lookup utility for saved queries, SavedQueryLookup

* Handle disabled SavedQuery nodes in parsing and lookups

* Add SavedQuery nodes to grouping process

Our grouping logic seems to be in a weird spot. It seems liek we're
moving to setting the `group` for a node in the node's `config` however,
all of the logic around grouping is still focused on the top level `group`
property on a nodes. To get group stuff plumbed I've thus added `group`
as a top level property of the `SavedQuery` node, and populated it from
the config group value.

* Plumb through saved query in a lot more places

I don't like making scatter shot commits like this. However, a lot
of this commit was written ~4am, soooo yea. Things were broken, I wanted
things to be unbroken. I mostly searched for `semantic_models` and added
the equivalent necessary `saved_queries`. Some stuff is in support of
writing out the manifest, some stuff helps with node selection, it's a
lot of miscelaneous stuff that I don't fully understand.

* Add `depends_on` to `SavedQuery` nodes and populate from `metrics` property

* Add partial parsing support to SavedQuery nodes

* Add `docs` support for SavedQuery descriptions

* Support selctor methods for SavedQuery nodes

* Add `refs` property to SavedQuery node

We don't actually append anything to `refs` for SavedQuery nodes currently.
I'm not sure if anything needs to be appended to them. Regardless, we
access the `refs` property throughout the codebase while iterating over
nodes. It seems wise to support this attribute as to not accidently blow
something up with it not existing.

* Support `saved_queries` when upgrading from manifests <= v10 (and regenerate v11)

* Add changie doc for saved query node support

* Pin to dbt-semantic-interfaces 0.3.0b1 for saved query work

We're gonna release DSI 0.3.0, and if this PR automatically pulls that
in things will break. But the things that need fixing should be handled
separately from this PR. After releasing DSI 0.3.0 I'm going to create
a branch off/ontop of this one, and open a stacked PR with the associated
changes.

* Bump supported DSI version to 0.3.x

* Switch metric filters and saved query where to use ne WhereFilterIntersection

* Update schema yaml readers to create WhereFilterInterfaces

* Expand metric filters and saved query where property to handle both str and list of strs

* Update tests which were broken by where filter changes

* Regeneate v11 manifest

* Fixup: Update `SavedQueryLookup.perform_lookup` to operate on saved queries

I missed this when I was copy and pasting 🤦
2023-10-11 15:54:11 -07:00
Peter Webb
2e35426d11 Add support for getting freshness from DBMS metadata (#8795)
* Add support for getting freshness from DBMS metadata

* Add changelog entry

* Add simple test case

* Change parsing error to warning and add new event type for warning

* Code review simplification of capability dict.

* Revisions to the capability mechanism per review

* Move utility function.

* Reduce try/except scope

* Clean up imports.

* Simplify typing per review

* Unit test fix
2023-10-11 15:55:57 -04:00
Emily Rockman
bf10a29f06 update v10 manifest on main (#8834)
* update manifest

* add changelog
2023-10-11 14:52:01 -05:00
Gerda Shank
a7e2d9bc40 Partial parsing issue when adding groups and updating models at the same time (#8817) 2023-10-11 15:01:49 -04:00
Michelle Ark
a3777496b5 fix changelog (#8833) 2023-10-11 13:22:05 -04:00
dependabot[bot]
edf6aedc51 Bump docker/build-push-action from 4 to 5 (#8783)
* Bump docker/build-push-action from 4 to 5

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4 to 5.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-10-11 09:51:51 -07:00
Jeremy Cohen
53845d0277 Add warning_tag to UnversionedBreakingChange (#8828) 2023-10-11 17:54:16 +02:00
Mike Alfare
3d27483658 ADAP-850: Support test results as a view (#8653)
* add `store_failures_as` parameter to TestConfig, catch strategy parameter in test materialization
* create test results as views
* updated test expected values for new config option
* break up tests into reusable tests and adapter specific configuration, update test to check for relation type and confirm views update
* move test configuration into base test class
* allow `store_failures_as` to drive whether failures are stored
* update expected test config dicts to include the new default value for store_failures_as
* Add `store_failures_as` config for generic tests
* cover --store-failures on CLI gap
* add generic tests test case for store_failures_as
* update object names for generic test case tests for store_failures_as
* remove unique generic test, it was not testing `store_failures_as`
* pull generic run and assertion into base test class to turn tests into quasi-parameterized tests
* add ephemeral option for store_failures_as, as a way to easily turn off store_failures at the model level
* add compilation error for invalid setting of store_failures_as

---------

Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
2023-10-10 17:26:01 -04:00
Emily Rockman
4f9bd0cb38 Fix uncaught exception for group updates (#8792)
* add test

* write test

* fix test

* updating test

* add clean

* cleanup

* more tests, fix comment

* add new test, move fixtures
2023-10-10 15:37:47 -05:00
Kshitij Aranke
3f7f7de179 Fix #8682: Override path-like args in dbt retry (#8803) 2023-10-10 19:31:54 +01:00
Kshitij Aranke
6461f5aacf Fix #8022: Foreign key constraint on incremental model results in Database Error (#8768) 2023-10-10 18:32:51 +01:00
Doug Beatty
339957b42c Explanation of Parsing vs. Compilation vs. Runtime (#8744)
* Explanation of Parsing vs. Compilation vs. Runtime

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Apply suggestions from code review

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

* Fix a couple markdown rendering issues

* Move to the "explain it like im 64" folder

When ELI5 just isnt detailed enough.

* Disambiguate Python references

Disambiguate Python references and delineate SQL models ("Jinja-SQL") from Python models ("dbt-py")

---------

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-10-10 10:51:54 -06:00
Gerda Shank
4391dc1a63 Type aliasing for model contract column data_type (#8592) 2023-10-10 11:43:26 -04:00
Michelle Ark
964e0e4e8a [Fix] respect project root when loading seeds (#8762) 2023-10-10 11:17:41 -04:00
Chenyu Li
549dbf3390 Deps lock by justbldwn (#8408)
*  adding installed_packages.json functionality

*  update test_simple_dependency_deps test

* 📝 adding changelog for deps feature via changie

*  restructure deps command, include lock/add

*  add new deps event types to sample_values

*  fix test_simple_dependency_deps test

* 🐛 attempting to fix cli commands

* 🐛 convert dbt deps to dbt deps install

also leave dbt deps as just a new click group

*  update test_command_mutually_exclusive_option

change deps command to deps install

*  update functional tests from deps > deps install

*  change missing deps to deps install

*  convert adapter tests to deps install from deps

* move back to deps and merge more with main

* fix-unittest

* add hash

* foramt yml and update command structure

* nits

* add new param

* nits

* nits

* nits

* fix_tests

* pr_feedback

* nits

* nits

* move_check

* Update Features-20230125-165933.yaml

---------

Co-authored-by: Justin Baldwin <91483530+justbldwn@users.noreply.github.com>
2023-10-09 21:05:00 -07:00
Quigley Malcolm
70b2e15a25 Add semantic model test to test_contracts_graph_parsed.py (#8654)
* Add semantic model test to `test_contracts_graph_parsed.py`

The tests in `test_contracts_graph_parsed.py` are meant to ensure
that we can go from objects to dictionaries and back without any
changes. We've had a desire to simplify these tests. Most tests in
this file have three to four fixtures, this test only has one. What
a test of this format ensures is that parsing a SemanticModel from
a dictionary doesn't add/drop any keys from the dictionary and that
when going back to the dictionary no keys are dropped. This style of
test will still break whenever the semantic model (or sub objects)
change. However now when that happens, only one fixture will have to
be updated (whereas previously we had to update 3-4 fixtures).

* Begin using hypothesis package for symmetry testing

Hypothesis is a python package for doing property testing. The `@given`
parameterizes a test, with it generating the arguements it has following
`strategies`. The main strategies we use is `builds` this takes in a callable
passes any sub strategies for named arguements, and will try to infer any
other arguments if the callable is typed. I found that even though the
test was run many many times, some of the `SemanticModel` properties
weren't being changed. For instance `dimensions`, `entities`, and `measures`
were always empty lists. Because of this I defined sub strategies for
some attributes of `SemanticModel`s.

* Update unittest readme to have details on test_contracts_graph_parsed methodology
2023-10-09 14:55:26 -07:00
Mark Scannell
bb249d612c Generate static index html documentation (#8615)
* Include option to generate static index.html

* Added changie

* Using DBT's system load / write file methods for better cross platform
support

* Updated docs tests with dbt.client.systems calls for file reading

* Writing out static_index.html as binary file to prevent line-ending
conversions on Windows. (similar behaviour as index.html)
2023-10-06 10:37:13 -07:00
Emily Rockman
17773bdb94 pin types-requests in dev-requirements (#8788)
* pin types-requests

* changelog
2023-10-06 09:53:08 -05:00
Gerda Shank
f30293359c Selectors in docs generate limits catalog generation (#8772) 2023-10-05 16:00:12 -04:00
Emily Rockman
0c85e6149f remove guild as codeowner (#8778) 2023-10-05 10:16:32 -05:00
Emily Rockman
ec57d7af94 add gha for dependabot checks (#8777) 2023-10-05 10:16:20 -05:00
Emily Rockman
df791f729c support doc blocks (#8771) 2023-10-05 08:12:28 -05:00
Emily Rockman
c6ff3abecd remove top level meta attribute (#8766) 2023-10-04 13:23:09 -05:00
Emily Rockman
eac13e3bd3 Add meta to SemanticModels (#8754)
* WIP

* changelog
2023-10-03 13:08:37 -05:00
Emily Rockman
46ee3f3d9c rebuild manifest missed fields (#8755)
* rebuild manifest missed fields

* changelogs
2023-10-02 09:38:50 -05:00
Peter Webb
5e1f0c5fbc Report Resource Usage Statistics When a dbt Command Finishes (#8671)
* Add performance metrics to the CommandCompleted event.

* Add changelog entry.

* Add flag for controling the log level of ResourceReport.

* Update changelog entry to reflect changes

* Remove outdated attributes

* Work around missing resource module on windows

* Fix corner case where flags are not set
2023-09-29 18:37:37 -04:00
Peter Webb
c4f09b160a Add new get_catalog_relations macro, Supporting Changes (#8648)
* Add new get_catalog_relations macro, allowing dbt to specify which relations in a schema the adapter should return data about

* Implement postgres adapter support for relation filtering on catalog queries

* Code review changes adding feature flag for catalog-by-relation-list support

* Use profile specified in --profile with dbt init (#7450)

* Use profile specified in --profile with dbt init

* Update .changes/unreleased/Fixes-20230424-161642.yaml

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Refactor run() method into functions, replace exit() calls with exceptions

* Update help text for profile option

---------

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* add TestLargeEphemeralCompilation (#8376)

* Fix a couple of issues in the postgres implementation of get_catalog_relations

* Add relation count limit at which to fall back to batch retrieval

* Better feature detection mechanism for adapters.

* Code review changes to get_catalog_relations and adapter feature checking

* Add changelog entry

---------

Co-authored-by: ezraerb <ezraerb@alum.mit.edu>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-09-29 16:13:23 -04:00
Jeremy Cohen
48c97e86dd Fix tag: selection for projects with semantic models (#8750)
* Add unit test to repro regression

* Add defensive code for tag: selection

* Add changelog entry
2023-09-29 09:49:32 -07:00
Emily Rockman
416bc845ad fix duplication of docs issues (#8747)
* fix duplication of docs issues

* update conditional to only run on merged PRs
2023-09-29 09:40:34 -05:00
Michelle Ark
408a78985a Fix: avoid double-rendering sql_header in dbt show (#8740) 2023-09-28 19:35:14 +01:00
Emily Rockman
0c965c8115 update changelog kind (#8737) 2023-09-28 08:20:57 -07:00
Philippe Boyd
f65e4b6940 feat: resolve packages with same git repo and unique subdirectory (#8322) 2023-09-27 15:31:37 -04:00
Michelle Ark
a2d4424f92 Inline limit in SQL sent from dbt show (#8641) 2023-09-27 15:56:34 +01:00
Kshitij Aranke
997f839cd6 Disallow cleaning paths outside current working directory (#8469) 2023-09-27 15:47:00 +01:00
Emily Rockman
556fad50df Model contracts: raise warning for numeric types without specified scale (#8721)
* add warning when contracting fields don't have precision

* rename files

* changelog

* move tests out of adapter zone

* Update core/dbt/include/global_project/macros/adapters/columns.sql

Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
2023-09-27 08:00:12 -05:00
dave-connors-3
bb4214b5c2 Dc/8546 semantic models in graph selection (#8589) 2023-09-26 23:23:03 +01:00
Kshitij Aranke
f17c1f3fe7 Fix #6497: Support global flags passed in after subcommands (#8670) 2023-09-26 16:16:36 +01:00
Kshitij Aranke
d4fe9a8ad4 Fix #8509: Support doc blocks in nested semantic model YAML (#8709) 2023-09-26 16:02:21 +01:00
Emily Rockman
2910aa29e4 Automated Repo Cleanup (#8686)
* add workflow

* cleanup and rename workflow

* rename workflow in actions

* add more context
2023-09-26 09:35:29 -05:00
Renan Leme
89cc073ea8 CT-3144 Fix test edges type filter on Graph (#8696)
* CT-3144 Fix test edges filter

* CT-3144 Add changelog

* CT-3144 Remove duplicated line

* CT-3144 Remove duplicated line

* CT-3144 Rename vars

* CT-3144 Update filter to use get_edge_data

* Trigger cla
2023-09-26 09:55:45 -04:00
Quigley Malcolm
aa86fdfe71 Add date spine macros to core (#8616)
* Add `date_spine` macro (and macros it depends on) from dbt-utils to core

The macros added are
- date_spine
- get_intervals_between
- generate_series
- get_powers_of_two

We're adding these to core because they are becoming more prevalently used
with the increase usage in the semantic layer. Basically if you are
using the semantic layer currently, then it is almost a requirement
to use dbt-utils, which is undesireable given the SL is supported directly
in core. The primary focus of this was to just add `date_spine`. However,
because `date_spine` depends on other macros, these other macros were
also moved.

* Add adapter tests for `get_powers_of_two` macro

* Add adapter tests for `generate_series` macro

* Add adapter tests for `get_intervals_between` macro

* Add adapter tests for `date_spine` macro

* Improve test fixture for `date_spine` macro to work with multiple adapters

* Cast to types to date in fixture_date_spine when targeting redshift

* Improve test fixture for `get_intervals_between` macro to work with multiple adapters

* changie doc for adding date_spine macro
2023-09-25 12:20:05 -07:00
Quigley Malcolm
48e9ced781 Support null coalescing properties for metric nodes (#8700)
* Include 'join_to_timespine` and `fill_nulls_with` in metric fixture

* Support `join_to_timespine` and `fill_nulls_with` properties on measure inputs to metrics

* Assert new `fill_nulls_with` and `join_to_timespine` properties don't break associated DSI protocol

* Add doc for metric null coalescing improvements

* Fix unit test for unparsed metric objects

The `assert_symmetric` function asserts that dictionaries are mostly
equivalent. I say mostly equivalent because it drops keys that are
`None`. The issue is that that `join_to_timespine` gets defaulted
to `False`, so we have to specify it in the `get_ok_dict` so that
they match.
2023-09-25 11:02:47 -07:00
Doug Beatty
7b02bd1f02 Lower bound of 8.0.2 for click (#8684)
* Lower bound of `8.0.2` for `click`

* Changelog entry
2023-09-24 13:46:54 -06:00
Emily Rockman
417fc2a735 Support quoted parameter list for MultiOption cli options (#8665)
* allow multioption to be quoted

* changelog

* fix test

* remove list format

* fix tests

* fix list object

* review arg change

* fix quotes

* Update .changes/unreleased/Features-20230918-150855.yaml

* add types

* convert list to set in test

* make mypy happy

* mroe mypy happiness

* more mypy happiness

* last mypy change

* add node to test
2023-09-22 14:34:36 -05:00
Quigley Malcolm
317128f790 Update pull request template to include type annotations as part of checklist (#8687)
* Update pull request template to include type annotations as part of checklist

* Add to checklist item a link to python typing information
2023-09-22 11:30:28 -07:00
Emily Rockman
e3dfb09b10 Support labels for semantic_models, dimensions, measures and entities (#8646)
* first pass

* changelog

* changelog

* Delete .changes/unreleased/Features-20230913-155802.yaml

* Update .changes/unreleased/Features-20230914-074429.yaml
2023-09-22 10:56:57 -05:00
279 changed files with 28958 additions and 8619 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.7.0b2
current_version = 1.7.8
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

View File

@@ -1,70 +0,0 @@
## dbt-core 1.7.0-b1 - August 17, 2023
### Breaking Changes
- Removed the FirstRunResultError and AfterFirstRunResultError event types, using the existing RunResultError in their place. ([#7963](https://github.com/dbt-labs/dbt-core/issues/7963))
### Features
- Enable re-population of metadata vars post-environment change during programmatic invocation ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- Added support to configure a delimiter for a seed file, defaults to comma ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- Allow specification of `create_metric: true` on measures ([#8125](https://github.com/dbt-labs/dbt-core/issues/8125))
### Fixes
- Copy dir during `dbt deps` if symlink fails ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- Fixed double-underline ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Enable converting deprecation warnings to errors ([#8130](https://github.com/dbt-labs/dbt-core/issues/8130))
- Add status to Parse Inline Error ([#8173](https://github.com/dbt-labs/dbt-core/issues/8173))
- Ensure `warn_error_options` get serialized in `invocation_args_dict` ([#7694](https://github.com/dbt-labs/dbt-core/issues/7694))
- Stop detecting materialization macros based on macro name ([#6231](https://github.com/dbt-labs/dbt-core/issues/6231))
- Update `dbt deps` download retry logic to handle `EOFError` exceptions ([#6653](https://github.com/dbt-labs/dbt-core/issues/6653))
- Improve handling of CTE injection with ephemeral models ([#8213](https://github.com/dbt-labs/dbt-core/issues/8213))
- Fix unbound local variable error in `checked_agg_time_dimension_for_measure` ([#8230](https://github.com/dbt-labs/dbt-core/issues/8230))
- Ensure runtime errors are raised for graph runnable tasks (compile, show, run, etc) ([#8166](https://github.com/dbt-labs/dbt-core/issues/8166))
- Fix retry not working with log-file-max-bytes ([#8297](https://github.com/dbt-labs/dbt-core/issues/8297))
- Detect changes to model access, version, or latest_version in state:modified ([#8189](https://github.com/dbt-labs/dbt-core/issues/8189))
- Add connection status into list of statuses for dbt debug ([#8350](https://github.com/dbt-labs/dbt-core/issues/8350))
- fix fqn-selection for external versioned models ([#8374](https://github.com/dbt-labs/dbt-core/issues/8374))
- Fix: DbtInternalError after model that previously ref'd external model is deleted ([#8375](https://github.com/dbt-labs/dbt-core/issues/8375))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
- Remedy performance regression by only writing run_results.json once. ([#8360](https://github.com/dbt-labs/dbt-core/issues/8360))
### Docs
- Corrected spelling of "Partiton" ([dbt-docs/#8100](https://github.com/dbt-labs/dbt-docs/issues/8100))
- Remove static SQL codeblock for metrics ([dbt-docs/#436](https://github.com/dbt-labs/dbt-docs/issues/436))
- fixed comment util.py ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Display contract and column constraints on the model page ([dbt-docs/#433](https://github.com/dbt-labs/dbt-docs/issues/433))
- Display semantic model details in docs ([dbt-docs/#431](https://github.com/dbt-labs/dbt-docs/issues/431))
### Under the Hood
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))
- format exception from dbtPlugin.initialize ([#8152](https://github.com/dbt-labs/dbt-core/issues/8152))
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
- Ref expressions with version can now be processed by the latest version of the high-performance dbt-extractor library. ([#7688](https://github.com/dbt-labs/dbt-core/issues/7688))
- Bump manifest schema version to v11, freeze manifest v10 ([#8333](https://github.com/dbt-labs/dbt-core/issues/8333))
- add tracking for plugin.get_nodes calls ([#8344](https://github.com/dbt-labs/dbt-core/issues/8344))
- add internal flag: --no-partial-parse-file-diff to inform whether to compute a file diff during partial parsing ([#8363](https://github.com/dbt-labs/dbt-core/issues/8363))
- Add return values to a number of functions for mypy ([#8389](https://github.com/dbt-labs/dbt-core/issues/8389))
- Fix mypy warnings for ManifestLoader.load() ([#8401](https://github.com/dbt-labs/dbt-core/issues/8401))
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))
### Dependencies
- Bump mypy from 1.3.0 to 1.4.0 ([#7912](https://github.com/dbt-labs/dbt-core/pull/7912))
- Bump mypy from 1.4.0 to 1.4.1 ([#8219](https://github.com/dbt-labs/dbt-core/pull/8219))
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))
- Support dbt-semantic-interfaces 0.2.0 ([#8250](https://github.com/dbt-labs/dbt-core/pull/8250))
### Contributors
- [@anjutiwari](https://github.com/anjutiwari) ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- [@d-kaneshiro](https://github.com/d-kaneshiro) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@gem7318](https://github.com/gem7318) ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- [@lllong33](https://github.com/lllong33) ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- [@marcodamore](https://github.com/marcodamore) ([#436](https://github.com/dbt-labs/dbt-core/issues/436))
- [@pgoslatara](https://github.com/pgoslatara) ([#8100](https://github.com/dbt-labs/dbt-core/issues/8100))
- [@ramonvermeulen](https://github.com/ramonvermeulen) ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))

View File

@@ -1,29 +0,0 @@
## dbt-core 1.7.0-b2 - September 01, 2023
### Features
- Add node attributes related to compilation to run_results.json ([#7519](https://github.com/dbt-labs/dbt-core/issues/7519))
- Support configuration of semantic models with the addition of enable/disable and group enablement. ([#7968](https://github.com/dbt-labs/dbt-core/issues/7968))
### Fixes
- Add support for swapping materialized views with tables/views and vice versa ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Turn breaking changes to contracted models into warnings for unversioned models ([#8384](https://github.com/dbt-labs/dbt-core/issues/8384), [#8282](https://github.com/dbt-labs/dbt-core/issues/8282))
- Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension` ([#8453](https://github.com/dbt-labs/dbt-core/issues/8453))
- fix ambiguous reference error for tests and versions when model name is duplicated across packages ([#8327](https://github.com/dbt-labs/dbt-core/issues/8327), [#8493](https://github.com/dbt-labs/dbt-core/issues/8493))
- Fix "Internal Error: Expected node <unique-id> not found in manifest" when depends_on set on ModelNodeArgs ([#8506](https://github.com/dbt-labs/dbt-core/issues/8506))
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
### Docs
- Fix newline escapes and improve formatting in docker README ([dbt-docs/#8211](https://github.com/dbt-labs/dbt-docs/issues/8211))
### Under the Hood
- Switch from hologram to mashumaro jsonschema ([#8426](https://github.com/dbt-labs/dbt-core/issues/8426))
- Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>, relation agnostic in /macros/relations ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Update typing to meet mypy standards ([#8396](https://github.com/dbt-labs/dbt-core/issues/8396))
- Mypy errors - adapters/factory.py ([#8387](https://github.com/dbt-labs/dbt-core/issues/8387))
### Contributors
- [@jamezrin](https://github.com/jamezrin) ([#8211](https://github.com/dbt-labs/dbt-core/issues/8211))

157
.changes/1.7.0.md Normal file
View File

@@ -0,0 +1,157 @@
## dbt-core 1.7.0 - November 02, 2023
### Breaking Changes
- Removed the FirstRunResultError and AfterFirstRunResultError event types, using the existing RunResultError in their place. ([#7963](https://github.com/dbt-labs/dbt-core/issues/7963))
### Features
- add log file of installed packages via dbt deps ([#6643](https://github.com/dbt-labs/dbt-core/issues/6643))
- Enable re-population of metadata vars post-environment change during programmatic invocation ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- Added support to configure a delimiter for a seed file, defaults to comma ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- Allow specification of `create_metric: true` on measures ([#8125](https://github.com/dbt-labs/dbt-core/issues/8125))
- Add node attributes related to compilation to run_results.json ([#7519](https://github.com/dbt-labs/dbt-core/issues/7519))
- Add --no-inject-ephemeral-ctes flag for `compile` command, for usage by linting. ([#8480](https://github.com/dbt-labs/dbt-core/issues/8480))
- Support configuration of semantic models with the addition of enable/disable and group enablement. ([#7968](https://github.com/dbt-labs/dbt-core/issues/7968))
- Accept a `dbt-cloud` config in dbt_project.yml ([#8438](https://github.com/dbt-labs/dbt-core/issues/8438))
- Support atomic replace in the global replace macro ([#8539](https://github.com/dbt-labs/dbt-core/issues/8539))
- Use translate_type on data_type in model.columns in templates by default, remove no op `TYPE_LABELS` ([#8007](https://github.com/dbt-labs/dbt-core/issues/8007))
- Add an option to generate static documentation ([#8614](https://github.com/dbt-labs/dbt-core/issues/8614))
- Allow setting "access" as a config in addition to as a property ([#8383](https://github.com/dbt-labs/dbt-core/issues/8383))
- Loosen typing requirement on renameable/replaceable relations to Iterable to allow adapters more flexibility in registering relation types, include docstrings as suggestions ([#8647](https://github.com/dbt-labs/dbt-core/issues/8647))
- Add support for optional label in semantic_models, measures, dimensions and entities. ([#8595](https://github.com/dbt-labs/dbt-core/issues/8595), [#8755](https://github.com/dbt-labs/dbt-core/issues/8755))
- Allow adapters to include package logs in dbt standard logging ([#7859](https://github.com/dbt-labs/dbt-core/issues/7859))
- Support storing test failures as views ([#6914](https://github.com/dbt-labs/dbt-core/issues/6914))
- resolve packages with same git repo and unique subdirectory ([#5374](https://github.com/dbt-labs/dbt-core/issues/5374))
- Add new ResourceReport event to record memory/cpu/io metrics ([#8342](https://github.com/dbt-labs/dbt-core/issues/8342))
- Adding `date_spine` macro (and supporting macros) from dbt-utils to dbt-core ([#8172](https://github.com/dbt-labs/dbt-core/issues/8172))
- Support `fill_nulls_with` and `join_to_timespine` for metric nodes ([#8593](https://github.com/dbt-labs/dbt-core/issues/8593), [#8755](https://github.com/dbt-labs/dbt-core/issues/8755))
- Raise a warning when a contracted model has a numeric field without scale defined ([#8183](https://github.com/dbt-labs/dbt-core/issues/8183))
- Added support for retrieving partial catalog information from a schema ([#8521](https://github.com/dbt-labs/dbt-core/issues/8521))
- Add meta attribute to SemanticModels config ([#8511](https://github.com/dbt-labs/dbt-core/issues/8511))
- Selectors with docs generate limits catalog generation ([#6014](https://github.com/dbt-labs/dbt-core/issues/6014))
- Allow freshness to be determined via DBMS metadata for supported adapters ([#8704](https://github.com/dbt-labs/dbt-core/issues/8704))
- Add support semantic layer SavedQuery node type ([#8594](https://github.com/dbt-labs/dbt-core/issues/8594))
- Add exports to SavedQuery spec ([#8892](https://github.com/dbt-labs/dbt-core/issues/8892))
### Fixes
- Copy dir during `dbt deps` if symlink fails ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- If --profile specified with dbt-init, create the project with the specified profile ([#6154](https://github.com/dbt-labs/dbt-core/issues/6154))
- Fixed double-underline ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Enable converting deprecation warnings to errors ([#8130](https://github.com/dbt-labs/dbt-core/issues/8130))
- Add status to Parse Inline Error ([#8173](https://github.com/dbt-labs/dbt-core/issues/8173))
- Ensure `warn_error_options` get serialized in `invocation_args_dict` ([#7694](https://github.com/dbt-labs/dbt-core/issues/7694))
- Stop detecting materialization macros based on macro name ([#6231](https://github.com/dbt-labs/dbt-core/issues/6231))
- Update `dbt deps` download retry logic to handle `EOFError` exceptions ([#6653](https://github.com/dbt-labs/dbt-core/issues/6653))
- Improve handling of CTE injection with ephemeral models ([#8213](https://github.com/dbt-labs/dbt-core/issues/8213))
- Fix unbound local variable error in `checked_agg_time_dimension_for_measure` ([#8230](https://github.com/dbt-labs/dbt-core/issues/8230))
- Ensure runtime errors are raised for graph runnable tasks (compile, show, run, etc) ([#8166](https://github.com/dbt-labs/dbt-core/issues/8166))
- Fix retry not working with log-file-max-bytes ([#8297](https://github.com/dbt-labs/dbt-core/issues/8297))
- Add explicit support for integers for the show command ([#8153](https://github.com/dbt-labs/dbt-core/issues/8153))
- Detect changes to model access, version, or latest_version in state:modified ([#8189](https://github.com/dbt-labs/dbt-core/issues/8189))
- Add connection status into list of statuses for dbt debug ([#8350](https://github.com/dbt-labs/dbt-core/issues/8350))
- fix fqn-selection for external versioned models ([#8374](https://github.com/dbt-labs/dbt-core/issues/8374))
- Fix: DbtInternalError after model that previously ref'd external model is deleted ([#8375](https://github.com/dbt-labs/dbt-core/issues/8375))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
- Remedy performance regression by only writing run_results.json once. ([#8360](https://github.com/dbt-labs/dbt-core/issues/8360))
- Add support for swapping materialized views with tables/views and vice versa ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Turn breaking changes to contracted models into warnings for unversioned models ([#8384](https://github.com/dbt-labs/dbt-core/issues/8384), [#8282](https://github.com/dbt-labs/dbt-core/issues/8282))
- Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension` ([#8453](https://github.com/dbt-labs/dbt-core/issues/8453))
- fix ambiguous reference error for tests and versions when model name is duplicated across packages ([#8327](https://github.com/dbt-labs/dbt-core/issues/8327), [#8493](https://github.com/dbt-labs/dbt-core/issues/8493))
- Fix "Internal Error: Expected node <unique-id> not found in manifest" when depends_on set on ModelNodeArgs ([#8506](https://github.com/dbt-labs/dbt-core/issues/8506))
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
- Parse the correct schema version from manifest ([#8544](https://github.com/dbt-labs/dbt-core/issues/8544))
- make version comparison insensitive to order ([#8571](https://github.com/dbt-labs/dbt-core/issues/8571))
- Update metric helper functions to work with new semantic layer metrics ([#8134](https://github.com/dbt-labs/dbt-core/issues/8134))
- Disallow cleaning paths outside current working directory ([#8318](https://github.com/dbt-labs/dbt-core/issues/8318))
- Warn when --state == --target ([#8160](https://github.com/dbt-labs/dbt-core/issues/8160))
- update dbt show to include limit in DWH query ([#8496,](https://github.com/dbt-labs/dbt-core/issues/8496,), [#8417](https://github.com/dbt-labs/dbt-core/issues/8417))
- Support quoted parameter list for MultiOption CLI options. ([#8598](https://github.com/dbt-labs/dbt-core/issues/8598))
- Support global flags passed in after subcommands ([#6497](https://github.com/dbt-labs/dbt-core/issues/6497))
- Lower bound of `8.0.2` for `click` ([#8683](https://github.com/dbt-labs/dbt-core/issues/8683))
- Fixes test type edges filter ([#8692](https://github.com/dbt-labs/dbt-core/issues/8692))
- semantic models in graph selection ([#8589](https://github.com/dbt-labs/dbt-core/issues/8589))
- Support doc blocks in nested semantic model YAML ([#8509](https://github.com/dbt-labs/dbt-core/issues/8509))
- avoid double-rendering sql_header in dbt show ([#8739](https://github.com/dbt-labs/dbt-core/issues/8739))
- Fix tag selection for projects with semantic models ([#8749](https://github.com/dbt-labs/dbt-core/issues/8749))
- Foreign key constraint on incremental model results in Database Error ([#8022](https://github.com/dbt-labs/dbt-core/issues/8022))
- Support docs blocks on versioned model column descriptions ([#8540](https://github.com/dbt-labs/dbt-core/issues/8540))
- Enable seeds to be handled from stored manifest data ([#6875](https://github.com/dbt-labs/dbt-core/issues/6875))
- Override path-like args in dbt retry ([#8682](https://github.com/dbt-labs/dbt-core/issues/8682))
- Group updates on unmodified nodes are handled gracefully for state:modified ([#8371](https://github.com/dbt-labs/dbt-core/issues/8371))
- Partial parsing fix for adding groups and updating models at the same time ([#8697](https://github.com/dbt-labs/dbt-core/issues/8697))
- Fix partial parsing not working for semantic model change ([#8859](https://github.com/dbt-labs/dbt-core/issues/8859))
- Rework get_catalog implementation to retain previous adapter interface semantics ([#8846](https://github.com/dbt-labs/dbt-core/issues/8846))
- Add back contract enforcement for temporary tables on postgres ([#8857](https://github.com/dbt-labs/dbt-core/issues/8857))
- Add version to fqn when version==0 ([#8836](https://github.com/dbt-labs/dbt-core/issues/8836))
- Fix cased comparison in catalog-retrieval function. ([#8939](https://github.com/dbt-labs/dbt-core/issues/8939))
- Catalog queries now assign the correct type to materialized views ([#8864](https://github.com/dbt-labs/dbt-core/issues/8864))
- Make relation filtering None-tolerant for maximal flexibility across adapters. ([#8974](https://github.com/dbt-labs/dbt-core/issues/8974))
### Docs
- Corrected spelling of "Partiton" ([dbt-docs/#8100](https://github.com/dbt-labs/dbt-docs/issues/8100))
- Remove static SQL codeblock for metrics ([dbt-docs/#436](https://github.com/dbt-labs/dbt-docs/issues/436))
- fixed comment util.py ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Fix newline escapes and improve formatting in docker README ([dbt-docs/#8211](https://github.com/dbt-labs/dbt-docs/issues/8211))
- Display contract and column constraints on the model page ([dbt-docs/#433](https://github.com/dbt-labs/dbt-docs/issues/433))
- Display semantic model details in docs ([dbt-docs/#431](https://github.com/dbt-labs/dbt-docs/issues/431))
### Under the Hood
- Switch from hologram to mashumaro jsonschema ([#8426](https://github.com/dbt-labs/dbt-core/issues/8426))
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))
- format exception from dbtPlugin.initialize ([#8152](https://github.com/dbt-labs/dbt-core/issues/8152))
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
- Ref expressions with version can now be processed by the latest version of the high-performance dbt-extractor library. ([#7688](https://github.com/dbt-labs/dbt-core/issues/7688))
- Bump manifest schema version to v11, freeze manifest v10 ([#8333](https://github.com/dbt-labs/dbt-core/issues/8333))
- add tracking for plugin.get_nodes calls ([#8344](https://github.com/dbt-labs/dbt-core/issues/8344))
- add internal flag: --no-partial-parse-file-diff to inform whether to compute a file diff during partial parsing ([#8363](https://github.com/dbt-labs/dbt-core/issues/8363))
- Add return values to a number of functions for mypy ([#8389](https://github.com/dbt-labs/dbt-core/issues/8389))
- Fix mypy warnings for ManifestLoader.load() ([#8401](https://github.com/dbt-labs/dbt-core/issues/8401))
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))
- Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>, relation agnostic in /macros/relations ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Update typing to meet mypy standards ([#8396](https://github.com/dbt-labs/dbt-core/issues/8396))
- Mypy errors - adapters/factory.py ([#8387](https://github.com/dbt-labs/dbt-core/issues/8387))
- Added more type annotations. ([#8537](https://github.com/dbt-labs/dbt-core/issues/8537))
- Audit potential circular dependencies ([#8349](https://github.com/dbt-labs/dbt-core/issues/8349))
- Add functional test for advanced ref override ([#8566](https://github.com/dbt-labs/dbt-core/issues/8566))
- Add typing to __init__ in base.py ([#8398](https://github.com/dbt-labs/dbt-core/issues/8398))
- Fix untyped functions in task/runnable.py (mypy warning) ([#8402](https://github.com/dbt-labs/dbt-core/issues/8402))
- add a test for ephemeral cte injection ([#8225](https://github.com/dbt-labs/dbt-core/issues/8225))
- Fix test_numeric_values to look for more specific strings ([#8470](https://github.com/dbt-labs/dbt-core/issues/8470))
- Pin types-requests<2.31.0 in `dev-requirements.txt` ([#8789](https://github.com/dbt-labs/dbt-core/issues/8789))
- Add warning_tag to UnversionedBreakingChange ([#8827](https://github.com/dbt-labs/dbt-core/issues/8827))
- Update v10 manifest schema to match 1.6 for testing schema compatibility ([#8835](https://github.com/dbt-labs/dbt-core/issues/8835))
- Add a no-op runner for Saved Qeury ([#8893](https://github.com/dbt-labs/dbt-core/issues/8893))
### Dependencies
- Bump mypy from 1.3.0 to 1.4.0 ([#7912](https://github.com/dbt-labs/dbt-core/pull/7912))
- Bump mypy from 1.4.0 to 1.4.1 ([#8219](https://github.com/dbt-labs/dbt-core/pull/8219))
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))
- Support dbt-semantic-interfaces 0.2.0 ([#8250](https://github.com/dbt-labs/dbt-core/pull/8250))
- Bump docker/build-push-action from 4 to 5 ([#8783](https://github.com/dbt-labs/dbt-core/pull/8783))
- Upgrade dbt-semantic-interfaces dep to 0.3.0 ([#8819](https://github.com/dbt-labs/dbt-core/pull/8819))
- Begin using DSI 0.4.x ([#8892](https://github.com/dbt-labs/dbt-core/pull/8892))
### Contributors
- [@anjutiwari](https://github.com/anjutiwari) ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- [@benmosher](https://github.com/benmosher) ([#8480](https://github.com/dbt-labs/dbt-core/issues/8480))
- [@d-kaneshiro](https://github.com/d-kaneshiro) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#8153](https://github.com/dbt-labs/dbt-core/issues/8153), [#8589](https://github.com/dbt-labs/dbt-core/issues/8589))
- [@dylan-murray](https://github.com/dylan-murray) ([#8683](https://github.com/dbt-labs/dbt-core/issues/8683))
- [@ezraerb](https://github.com/ezraerb) ([#6154](https://github.com/dbt-labs/dbt-core/issues/6154))
- [@gem7318](https://github.com/gem7318) ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- [@jamezrin](https://github.com/jamezrin) ([#8211](https://github.com/dbt-labs/dbt-core/issues/8211))
- [@jusbaldw](https://github.com/jusbaldw) ([#6643](https://github.com/dbt-labs/dbt-core/issues/6643))
- [@lllong33](https://github.com/lllong33) ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- [@marcodamore](https://github.com/marcodamore) ([#436](https://github.com/dbt-labs/dbt-core/issues/436))
- [@mescanne](https://github.com/mescanne) ([#8614](https://github.com/dbt-labs/dbt-core/issues/8614))
- [@pgoslatara](https://github.com/pgoslatara) ([#8100](https://github.com/dbt-labs/dbt-core/issues/8100))
- [@philippeboyd](https://github.com/philippeboyd) ([#5374](https://github.com/dbt-labs/dbt-core/issues/5374))
- [@ramonvermeulen](https://github.com/ramonvermeulen) ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- [@renanleme](https://github.com/renanleme) ([#8692](https://github.com/dbt-labs/dbt-core/issues/8692))

View File

@@ -1,7 +0,0 @@
kind: Breaking Changes
body: Removed the FirstRunResultError and AfterFirstRunResultError event types, using
the existing RunResultError in their place.
time: 2023-07-25T17:13:59.441682-04:00
custom:
Author: peterallenwebb
Issue: "7963"

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Bump mypy from 1.3.0 to 1.4.0"
time: 2023-06-21T00:57:52.00000Z
custom:
Author: dependabot[bot]
PR: 7912

View File

@@ -1,6 +0,0 @@
kind: "Dependencies"
body: "Bump mypy from 1.4.0 to 1.4.1"
time: 2023-07-26T20:17:40.00000Z
custom:
Author: dependabot[bot]
PR: 8219

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Update pin for click<9
time: 2023-07-27T14:57:03.180458-05:00
custom:
Author: emmyoop
PR: "8232"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Add upper bound to sqlparse pin of <0.5
time: 2023-07-27T14:57:26.40416-05:00
custom:
Author: emmyoop
PR: "8236"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Support dbt-semantic-interfaces 0.2.0
time: 2023-07-28T13:52:27.207241-07:00
custom:
Author: QMalcolm
PR: "8250"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Corrected spelling of "Partiton"
time: 2023-07-15T20:09:07.057361092+02:00
custom:
Author: pgoslatara
Issue: "8100"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Remove static SQL codeblock for metrics
time: 2023-07-18T19:24:22.155323+02:00
custom:
Author: marcodamore
Issue: "436"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: fixed comment util.py
time: 2023-07-27T17:09:00.089237+09:00
custom:
Author: d-kaneshiro
Issue: None

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Fix newline escapes and improve formatting in docker README
time: 2023-07-28T19:34:38.351042747+02:00
custom:
Author: jamezrin
Issue: "8211"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Display contract and column constraints on the model page
time: 2023-08-04T13:18:15.627005-05:00
custom:
Author: emmyoop
Issue: "433"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Display semantic model details in docs
time: 2023-08-07T15:25:48.711627-05:00
custom:
Author: emmyoop
Issue: "431"

View File

@@ -1,7 +0,0 @@
kind: Features
body: Enable re-population of metadata vars post-environment change during programmatic
invocation
time: 2023-07-02T12:28:13.416305-04:00
custom:
Author: gem7318
Issue: "8010"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Added support to configure a delimiter for a seed file, defaults to comma
time: 2023-07-14T20:24:45.513847165+02:00
custom:
Author: ramonvermeulen
Issue: "3990"

View File

@@ -1,6 +0,0 @@
kind: Features
body: 'Allow specification of `create_metric: true` on measures'
time: 2023-08-03T15:18:24.351003-07:00
custom:
Author: QMalcolm
Issue: "8125"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add node attributes related to compilation to run_results.json
time: 2023-08-21T10:33:57.200883-04:00
custom:
Author: peterallenwebb
Issue: "7519"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support configuration of semantic models with the addition of enable/disable and group enablement.
time: 2023-08-28T09:21:00.551633-05:00
custom:
Author: emmyoop
Issue: "7968"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Copy dir during `dbt deps` if symlink fails
time: 2023-04-24T21:07:34.336797+05:30
custom:
Author: anjutiwari
Issue: "7428 8223"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fixed double-underline
time: 2023-06-25T14:27:31.231253719+08:00
custom:
Author: lllong33
Issue: "5301"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Copy target_schema from config into snapshot node
time: 2023-07-17T16:06:52.957724-04:00
custom:
Author: gshank
Issue: "6745"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Enable converting deprecation warnings to errors
time: 2023-07-18T12:55:18.03914-04:00
custom:
Author: michelleark
Issue: "8130"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Add status to Parse Inline Error
time: 2023-07-20T12:27:23.085084-07:00
custom:
Author: ChenyuLInx
Issue: "8173"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Ensure `warn_error_options` get serialized in `invocation_args_dict`
time: 2023-07-20T16:15:13.761813-07:00
custom:
Author: QMalcolm
Issue: "7694"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Stop detecting materialization macros based on macro name
time: 2023-07-20T17:01:12.496238-07:00
custom:
Author: QMalcolm
Issue: "6231"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Update `dbt deps` download retry logic to handle `EOFError` exceptions
time: 2023-07-20T17:24:22.969951-07:00
custom:
Author: QMalcolm
Issue: "6653"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Improve handling of CTE injection with ephemeral models
time: 2023-07-26T10:44:48.888451-04:00
custom:
Author: gshank
Issue: "8213"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix unbound local variable error in `checked_agg_time_dimension_for_measure`
time: 2023-07-27T12:58:30.673803-07:00
custom:
Author: QMalcolm
Issue: "8230"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Ensure runtime errors are raised for graph runnable tasks (compile, show, run,
etc)
time: 2023-07-28T11:56:20.863718-04:00
custom:
Author: michelleark
Issue: "8166"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix retry not working with log-file-max-bytes
time: 2023-08-02T14:15:56.306027-07:00
custom:
Author: ChenyuLInx
Issue: "8297"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Detect changes to model access, version, or latest_version in state:modified
time: 2023-08-06T22:23:19.166334-04:00
custom:
Author: michelleark
Issue: "8189"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Add connection status into list of statuses for dbt debug
time: 2023-08-10T18:48:59.221344+01:00
custom:
Author: aranke
Issue: "8350"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: fix fqn-selection for external versioned models
time: 2023-08-11T20:41:44.725144-04:00
custom:
Author: michelleark
Issue: "8374"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: 'Fix: DbtInternalError after model that previously ref''d external model is
deleted'
time: 2023-08-11T21:20:08.145554-04:00
custom:
Author: michelleark
Issue: "8375"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix using list command with path selector and project-dir
time: 2023-08-14T14:57:02.02816-04:00
custom:
Author: gshank
Issue: "8385"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Remedy performance regression by only writing run_results.json once.
time: 2023-08-15T10:44:44.836991-04:00
custom:
Author: peterallenwebb
Issue: "8360"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Use python version 3.10.7 in Docker image.
time: 2023-08-17T13:09:15.936349-05:00
custom:
Author: McKnight-42
Issue: "8444"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Add support for swapping materialized views with tables/views and vice versa
time: 2023-08-17T18:57:39.01958-04:00
custom:
Author: mikealfare
Issue: "8449"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension`
time: 2023-08-18T09:53:48.154848-07:00
custom:
Author: QMalcolm
Issue: "8453"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Turn breaking changes to contracted models into warnings for unversioned models
time: 2023-08-18T10:38:02.251286-05:00
custom:
Author: emmyoop
Issue: 8384 8282

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: fix ambiguous reference error for tests and versions when model name is duplicated across
packages
time: 2023-08-24T16:10:24.437362-04:00
custom:
Author: michelleark
Issue: "8327 8493"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: 'Fix "Internal Error: Expected node <unique-id> not found in manifest" when
depends_on set on ModelNodeArgs'
time: 2023-08-28T12:58:58.061228-04:00
custom:
Author: michelleark
Issue: "8506"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix snapshot success message
time: 2023-08-30T15:08:03.429373-04:00
custom:
Author: gshank
Issue: "7583"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Switch from hologram to mashumaro jsonschema
time: 2023-07-18T14:54:28.41453-04:00
custom:
Author: gshank
Issue: "8426"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Refactor flaky test pp_versioned_models
time: 2023-07-19T12:46:11.972481-04:00
custom:
Author: gshank
Issue: "7781"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: format exception from dbtPlugin.initialize
time: 2023-07-19T16:33:34.586377-04:00
custom:
Author: michelleark
Issue: "8152"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: A way to control maxBytes for a single dbt.log file
time: 2023-07-24T15:06:54.263822-07:00
custom:
Author: ChenyuLInx
Issue: "8199"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: Ref expressions with version can now be processed by the latest version of the
high-performance dbt-extractor library.
time: 2023-07-25T10:26:09.902878-04:00
custom:
Author: peterallenwebb
Issue: "7688"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Bump manifest schema version to v11, freeze manifest v10
time: 2023-08-07T16:45:09.712744-04:00
custom:
Author: gshank
Issue: "8333"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: add tracking for plugin.get_nodes calls
time: 2023-08-09T09:48:34.819445-04:00
custom:
Author: michelleark
Issue: "8344"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: 'add internal flag: --no-partial-parse-file-diff to inform whether to compute
a file diff during partial parsing'
time: 2023-08-11T10:09:02.832241-04:00
custom:
Author: michelleark
Issue: "8363"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Add return values to a number of functions for mypy
time: 2023-08-15T17:03:07.895252-04:00
custom:
Author: gshank
Issue: "8389"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Fix mypy warnings for ManifestLoader.load()
time: 2023-08-17T13:45:48.937252-04:00
custom:
Author: gshank
Issue: "8401"

View File

@@ -1,7 +0,0 @@
kind: Under the Hood
body: 'Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>,
relation agnostic in /macros/relations'
time: 2023-08-21T13:48:01.474731-04:00
custom:
Author: mikealfare
Issue: "8449"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Update typing to meet mypy standards
time: 2023-08-23T19:42:37.130694-04:00
custom:
Author: mikealfare
Issue: "8396"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Mypy errors - adapters/factory.py
time: 2023-08-30T14:02:31.519929-04:00
custom:
Author: gshank
Issue: "8387"

8
.changes/1.7.1.md Normal file
View File

@@ -0,0 +1,8 @@
## dbt-core 1.7.1 - November 07, 2023
### Fixes
- Fix compilation exception running empty seed file and support new Integer agate data_type ([#8895](https://github.com/dbt-labs/dbt-core/issues/8895))
- Update run_results.json from previous versions of dbt to support deferral and rerun from failure ([#9010](https://github.com/dbt-labs/dbt-core/issues/9010))
- Use MANIFEST.in to recursively include all jinja templates; fixes issue where some templates were not included in the distribution ([#9016](https://github.com/dbt-labs/dbt-core/issues/9016))
- Fix git repository with subdirectory for Deps ([#9000](https://github.com/dbt-labs/dbt-core/issues/9000))

16
.changes/1.7.2.md Normal file
View File

@@ -0,0 +1,16 @@
## dbt-core 1.7.2 - November 16, 2023
### Features
- Support setting export configs hierarchically via saved query and project configs ([#8956](https://github.com/dbt-labs/dbt-core/issues/8956))
### Fixes
- Fix formatting of tarball information in packages-lock.yml ([#9062](https://github.com/dbt-labs/dbt-core/issues/9062))
### Under the Hood
- Treat SystemExit as an interrupt if raised during node execution. ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))
### Contributors
- [@benmosher](https://github.com/benmosher) ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))

7
.changes/1.7.3.md Normal file
View File

@@ -0,0 +1,7 @@
## dbt-core 1.7.3 - November 29, 2023
### Fixes
- deps: Lock git packages to commit SHA during resolution ([#9050](https://github.com/dbt-labs/dbt-core/issues/9050))
- deps: Use PackageRenderer to read package-lock.json ([#9127](https://github.com/dbt-labs/dbt-core/issues/9127))
- Get sources working again in dbt docs generate ([#9119](https://github.com/dbt-labs/dbt-core/issues/9119))

12
.changes/1.7.4.md Normal file
View File

@@ -0,0 +1,12 @@
## dbt-core 1.7.4 - December 14, 2023
### Features
- Adds support for parsing conversion metric related properties for the semantic layer. ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))
### Fixes
- Ensure we produce valid jsonschema schemas for manifest, catalog, run-results, and sources ([#8991](https://github.com/dbt-labs/dbt-core/issues/8991))
### Contributors
- [@WilliamDee](https://github.com/WilliamDee) ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))

8
.changes/1.7.5.md Normal file
View File

@@ -0,0 +1,8 @@
## dbt-core 1.7.5 - January 18, 2024
### Fixes
- Preserve the value of vars and the --full-refresh flags when using retry. ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))
### Contributors
- [@peterallenwebb,](https://github.com/peterallenwebb,) ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))

6
.changes/1.7.6.md Normal file
View File

@@ -0,0 +1,6 @@
## dbt-core 1.7.6 - January 25, 2024
### Fixes
- Handle unknown `type_code` for model contracts ([#8877](https://github.com/dbt-labs/dbt-core/issues/8877), [#8353](https://github.com/dbt-labs/dbt-core/issues/8353))
- Fix retry command run from CLI ([#9444](https://github.com/dbt-labs/dbt-core/issues/9444))

6
.changes/1.7.7.md Normal file
View File

@@ -0,0 +1,6 @@
## dbt-core 1.7.7 - February 01, 2024
### Fixes
- Fix seed and source selection in `dbt docs generate` ([#9161](https://github.com/dbt-labs/dbt-core/issues/9161))
- Add TestGenerateCatalogWithExternalNodes, include empty nodes in node selection during docs generate ([#9456](https://github.com/dbt-labs/dbt-core/issues/9456))

6
.changes/1.7.8.md Normal file
View File

@@ -0,0 +1,6 @@
## dbt-core 1.7.8 - February 14, 2024
### Fixes
- When patching versioned models, set constraints after config ([#9364](https://github.com/dbt-labs/dbt-core/issues/9364))
- Store node_info in node associated logging events ([#9557](https://github.com/dbt-labs/dbt-core/issues/9557))

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add --no-inject-ephemeral-ctes flag for `compile` command, for usage by linting.
time: 2023-08-23T14:04:07.617476-04:00
custom:
Author: benmosher
Issue: "8480"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Accept a `dbt-cloud` config in dbt_project.yml
time: 2023-08-30T21:28:28.976746-05:00
custom:
Author: emmyoop
Issue: "8438"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support atomic replace in the global replace macro
time: 2023-08-31T20:48:04.098933-04:00
custom:
Author: mikealfare
Issue: "8539"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Allow setting "access" as a config in addition to as a property
time: 2023-09-12T15:30:13.859595-04:00
custom:
Author: gshank
Issue: "8383"

View File

@@ -1,8 +0,0 @@
kind: Features
body: Loosen typing requirement on renameable/replaceable relations to Iterable to
allow adapters more flexibility in registering relation types, include docstrings
as suggestions
time: 2023-09-13T18:27:07.974612-04:00
custom:
Author: mikealfare
Issue: "8647"

View File

@@ -1,6 +0,0 @@
kind: Features
body: 'Allow adapters to include package logs in dbt standard logging '
time: 2023-09-15T12:37:33.862862-07:00
custom:
Author: colin-rogers-dbt
Issue: "7859"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: If --profile specified with dbt-init, create the project with the specified
profile
time: 2023-04-24T16:16:42.994547-04:00
custom:
Author: ezraerb
Issue: "6154"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Add explicit support for integers for the show command
time: 2023-08-03T09:35:02.163968-05:00
custom:
Author: dave-connors-3
Issue: "8153"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Update metric helper functions to work with new semantic layer metrics
time: 2023-09-06T12:02:12.156534-07:00
custom:
Author: QMalcolm
Issue: "8134"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: make version comparison insensitive to order
time: 2023-09-06T14:22:13.114549-04:00
custom:
Author: michelleark
Issue: "8571"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Parse the correct schema version from manifest
time: 2023-09-06T16:24:27.849069+01:00
custom:
Author: aranke
Issue: "8544"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Warn when --state == --target
time: 2023-09-12T22:53:29.869746+01:00
custom:
Author: aranke
Issue: "8160"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Added more type annotations.
time: 2023-08-31T16:44:35.737954-04:00
custom:
Author: peterallenwebb
Issue: "8537"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Add functional test for advanced ref override
time: 2023-09-06T11:02:49.253486-04:00
custom:
Author: michelleark
Issue: "8566"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Fix untyped functions in task/runnable.py (mypy warning)
time: 2023-09-06T13:24:35.448782-04:00
custom:
Author: michelleark
Issue: "8402"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Audit potential circular dependencies
time: 2023-09-06T14:16:16.58457+01:00
custom:
Author: aranke
Issue: "8349"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Add typing to __init__ in base.py
time: 2023-09-06T16:49:01.150713+01:00
custom:
Author: aranke
Issue: "8398"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: add a test for ephemeral cte injection
time: 2023-09-12T23:06:19.938207+01:00
custom:
Author: michelleark
Issue: "8376"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Fix test_numeric_values to look for more specific strings
time: 2023-09-13T14:16:51.453247-04:00
custom:
Author: gshank
Issue: "8470"

19
.github/CODEOWNERS vendored
View File

@@ -13,23 +13,6 @@
# the core team as a whole will be assigned
* @dbt-labs/core-team
### OSS Tooling Guild
/.github/ @dbt-labs/guild-oss-tooling
.bumpversion.cfg @dbt-labs/guild-oss-tooling
.changie.yaml @dbt-labs/guild-oss-tooling
pre-commit-config.yaml @dbt-labs/guild-oss-tooling
pytest.ini @dbt-labs/guild-oss-tooling
tox.ini @dbt-labs/guild-oss-tooling
pyproject.toml @dbt-labs/guild-oss-tooling
requirements.txt @dbt-labs/guild-oss-tooling
dev_requirements.txt @dbt-labs/guild-oss-tooling
/core/setup.py @dbt-labs/guild-oss-tooling
/core/MANIFEST.in @dbt-labs/guild-oss-tooling
### ADAPTERS
# Adapter interface ("base" + "sql" adapter defaults, cache)
@@ -40,7 +23,7 @@ dev_requirements.txt @dbt-labs/guild-oss-tooling
# Postgres plugin
/plugins/ @dbt-labs/core-adapters
/plugins/postgres/setup.py @dbt-labs/core-adapters @dbt-labs/guild-oss-tooling
/plugins/postgres/setup.py @dbt-labs/core-adapters
# Functional tests for adapter plugins
/tests/adapter @dbt-labs/core-adapters

View File

@@ -28,3 +28,10 @@ updates:
schedule:
interval: "weekly"
rebase-strategy: "disabled"
# github dependencies
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
rebase-strategy: "disabled"

View File

@@ -30,3 +30,4 @@ resolves #
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [ ] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions

View File

@@ -27,7 +27,14 @@ permissions:
jobs:
open_issues:
if: contains( github.event.pull_request.labels.*.name, 'user docs') && github.event.pull_request.merged == true
# we only want to run this when the PR has been merged or the label in the labeled event is `user docs`. Otherwise it runs the
# risk of duplicaton of issues being created due to merge and label both triggering this workflow to run and neither having
# generating the comment before the other runs. This lives here instead of the shared workflow because this is where we
# decide if it should run or not.
if: |
(github.event.pull_request.merged == true) &&
((github.event.action == 'closed' && contains( github.event.pull_request.labels.*.name, 'user docs')) ||
(github.event.action == 'labeled' && github.event.label.name == 'user docs'))
uses: dbt-labs/actions/.github/workflows/open-issue-in-repo.yml@main
with:
issue_repository: "dbt-labs/docs.getdbt.com"

View File

@@ -83,7 +83,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push MAJOR.MINOR.PATCH tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v5
with:
file: docker/Dockerfile
push: True
@@ -94,7 +94,7 @@ jobs:
ghcr.io/dbt-labs/${{ github.event.inputs.package }}:${{ github.event.inputs.version_number }}
- name: Build and push MINOR.latest tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v5
if: ${{ needs.get_version_meta.outputs.minor_latest == 'True' }}
with:
file: docker/Dockerfile
@@ -106,7 +106,7 @@ jobs:
ghcr.io/dbt-labs/${{ github.event.inputs.package }}:${{ needs.get_version_meta.outputs.major }}.${{ needs.get_version_meta.outputs.minor }}.latest
- name: Build and push latest tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v5
if: ${{ needs.get_version_meta.outputs.latest == 'True' }}
with:
file: docker/Dockerfile

View File

@@ -0,0 +1,30 @@
# **what?**
# Cleanup branches left over from automation and testing. Also cleanup
# draft releases from release testing.
# **why?**
# The automations are leaving behind branches and releases that clutter
# the repository. Sometimes we need them to debug processes so we don't
# want them immediately deleted. Running on Saturday to avoid running
# at the same time as an actual release to prevent breaking a release
# mid-release.
# **when?**
# Mainly on a schedule of 12:00 Saturday.
# Manual trigger can also run on demand
name: Repository Cleanup
on:
schedule:
- cron: '0 12 * * SAT' # At 12:00 on Saturday - details in `why` above
workflow_dispatch: # for manual triggering
permissions:
contents: write
jobs:
cleanup-repo:
uses: dbt-labs/actions/.github/workflows/repository-cleanup.yml@main
secrets: inherit

View File

@@ -5,38 +5,86 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.7.0-b2 - September 01, 2023
### Features
- Add node attributes related to compilation to run_results.json ([#7519](https://github.com/dbt-labs/dbt-core/issues/7519))
- Support configuration of semantic models with the addition of enable/disable and group enablement. ([#7968](https://github.com/dbt-labs/dbt-core/issues/7968))
## dbt-core 1.7.8 - February 14, 2024
### Fixes
- Add support for swapping materialized views with tables/views and vice versa ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Turn breaking changes to contracted models into warnings for unversioned models ([#8384](https://github.com/dbt-labs/dbt-core/issues/8384), [#8282](https://github.com/dbt-labs/dbt-core/issues/8282))
- Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension` ([#8453](https://github.com/dbt-labs/dbt-core/issues/8453))
- fix ambiguous reference error for tests and versions when model name is duplicated across packages ([#8327](https://github.com/dbt-labs/dbt-core/issues/8327), [#8493](https://github.com/dbt-labs/dbt-core/issues/8493))
- Fix "Internal Error: Expected node <unique-id> not found in manifest" when depends_on set on ModelNodeArgs ([#8506](https://github.com/dbt-labs/dbt-core/issues/8506))
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
- When patching versioned models, set constraints after config ([#9364](https://github.com/dbt-labs/dbt-core/issues/9364))
- Store node_info in node associated logging events ([#9557](https://github.com/dbt-labs/dbt-core/issues/9557))
### Docs
- Fix newline escapes and improve formatting in docker README ([dbt-docs/#8211](https://github.com/dbt-labs/dbt-docs/issues/8211))
## dbt-core 1.7.7 - February 01, 2024
### Fixes
- Fix seed and source selection in `dbt docs generate` ([#9161](https://github.com/dbt-labs/dbt-core/issues/9161))
- Add TestGenerateCatalogWithExternalNodes, include empty nodes in node selection during docs generate ([#9456](https://github.com/dbt-labs/dbt-core/issues/9456))
## dbt-core 1.7.6 - January 25, 2024
### Fixes
- Handle unknown `type_code` for model contracts ([#8877](https://github.com/dbt-labs/dbt-core/issues/8877), [#8353](https://github.com/dbt-labs/dbt-core/issues/8353))
- Fix retry command run from CLI ([#9444](https://github.com/dbt-labs/dbt-core/issues/9444))
## dbt-core 1.7.5 - January 18, 2024
### Fixes
- Preserve the value of vars and the --full-refresh flags when using retry. ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))
### Contributors
- [@peterallenwebb,](https://github.com/peterallenwebb,) ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))
## dbt-core 1.7.4 - December 14, 2023
### Features
- Adds support for parsing conversion metric related properties for the semantic layer. ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))
### Fixes
- Ensure we produce valid jsonschema schemas for manifest, catalog, run-results, and sources ([#8991](https://github.com/dbt-labs/dbt-core/issues/8991))
### Contributors
- [@WilliamDee](https://github.com/WilliamDee) ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))
## dbt-core 1.7.3 - November 29, 2023
### Fixes
- deps: Lock git packages to commit SHA during resolution ([#9050](https://github.com/dbt-labs/dbt-core/issues/9050))
- deps: Use PackageRenderer to read package-lock.json ([#9127](https://github.com/dbt-labs/dbt-core/issues/9127))
- Get sources working again in dbt docs generate ([#9119](https://github.com/dbt-labs/dbt-core/issues/9119))
## dbt-core 1.7.2 - November 16, 2023
### Features
- Support setting export configs hierarchically via saved query and project configs ([#8956](https://github.com/dbt-labs/dbt-core/issues/8956))
### Fixes
- Fix formatting of tarball information in packages-lock.yml ([#9062](https://github.com/dbt-labs/dbt-core/issues/9062))
### Under the Hood
- Switch from hologram to mashumaro jsonschema ([#8426](https://github.com/dbt-labs/dbt-core/issues/8426))
- Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>, relation agnostic in /macros/relations ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Update typing to meet mypy standards ([#8396](https://github.com/dbt-labs/dbt-core/issues/8396))
- Mypy errors - adapters/factory.py ([#8387](https://github.com/dbt-labs/dbt-core/issues/8387))
- Treat SystemExit as an interrupt if raised during node execution. ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))
### Contributors
- [@jamezrin](https://github.com/jamezrin) ([#8211](https://github.com/dbt-labs/dbt-core/issues/8211))
- [@benmosher](https://github.com/benmosher) ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))
## dbt-core 1.7.1 - November 07, 2023
## dbt-core 1.7.0-b1 - August 17, 2023
### Fixes
- Fix compilation exception running empty seed file and support new Integer agate data_type ([#8895](https://github.com/dbt-labs/dbt-core/issues/8895))
- Update run_results.json from previous versions of dbt to support deferral and rerun from failure ([#9010](https://github.com/dbt-labs/dbt-core/issues/9010))
- Use MANIFEST.in to recursively include all jinja templates; fixes issue where some templates were not included in the distribution ([#9016](https://github.com/dbt-labs/dbt-core/issues/9016))
- Fix git repository with subdirectory for Deps ([#9000](https://github.com/dbt-labs/dbt-core/issues/9000))
## dbt-core 1.7.0 - November 02, 2023
### Breaking Changes
@@ -44,13 +92,38 @@
### Features
- add log file of installed packages via dbt deps ([#6643](https://github.com/dbt-labs/dbt-core/issues/6643))
- Enable re-population of metadata vars post-environment change during programmatic invocation ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- Added support to configure a delimiter for a seed file, defaults to comma ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- Allow specification of `create_metric: true` on measures ([#8125](https://github.com/dbt-labs/dbt-core/issues/8125))
- Add node attributes related to compilation to run_results.json ([#7519](https://github.com/dbt-labs/dbt-core/issues/7519))
- Add --no-inject-ephemeral-ctes flag for `compile` command, for usage by linting. ([#8480](https://github.com/dbt-labs/dbt-core/issues/8480))
- Support configuration of semantic models with the addition of enable/disable and group enablement. ([#7968](https://github.com/dbt-labs/dbt-core/issues/7968))
- Accept a `dbt-cloud` config in dbt_project.yml ([#8438](https://github.com/dbt-labs/dbt-core/issues/8438))
- Support atomic replace in the global replace macro ([#8539](https://github.com/dbt-labs/dbt-core/issues/8539))
- Use translate_type on data_type in model.columns in templates by default, remove no op `TYPE_LABELS` ([#8007](https://github.com/dbt-labs/dbt-core/issues/8007))
- Add an option to generate static documentation ([#8614](https://github.com/dbt-labs/dbt-core/issues/8614))
- Allow setting "access" as a config in addition to as a property ([#8383](https://github.com/dbt-labs/dbt-core/issues/8383))
- Loosen typing requirement on renameable/replaceable relations to Iterable to allow adapters more flexibility in registering relation types, include docstrings as suggestions ([#8647](https://github.com/dbt-labs/dbt-core/issues/8647))
- Add support for optional label in semantic_models, measures, dimensions and entities. ([#8595](https://github.com/dbt-labs/dbt-core/issues/8595), [#8755](https://github.com/dbt-labs/dbt-core/issues/8755))
- Allow adapters to include package logs in dbt standard logging ([#7859](https://github.com/dbt-labs/dbt-core/issues/7859))
- Support storing test failures as views ([#6914](https://github.com/dbt-labs/dbt-core/issues/6914))
- resolve packages with same git repo and unique subdirectory ([#5374](https://github.com/dbt-labs/dbt-core/issues/5374))
- Add new ResourceReport event to record memory/cpu/io metrics ([#8342](https://github.com/dbt-labs/dbt-core/issues/8342))
- Adding `date_spine` macro (and supporting macros) from dbt-utils to dbt-core ([#8172](https://github.com/dbt-labs/dbt-core/issues/8172))
- Support `fill_nulls_with` and `join_to_timespine` for metric nodes ([#8593](https://github.com/dbt-labs/dbt-core/issues/8593), [#8755](https://github.com/dbt-labs/dbt-core/issues/8755))
- Raise a warning when a contracted model has a numeric field without scale defined ([#8183](https://github.com/dbt-labs/dbt-core/issues/8183))
- Added support for retrieving partial catalog information from a schema ([#8521](https://github.com/dbt-labs/dbt-core/issues/8521))
- Add meta attribute to SemanticModels config ([#8511](https://github.com/dbt-labs/dbt-core/issues/8511))
- Selectors with docs generate limits catalog generation ([#6014](https://github.com/dbt-labs/dbt-core/issues/6014))
- Allow freshness to be determined via DBMS metadata for supported adapters ([#8704](https://github.com/dbt-labs/dbt-core/issues/8704))
- Add support semantic layer SavedQuery node type ([#8594](https://github.com/dbt-labs/dbt-core/issues/8594))
- Add exports to SavedQuery spec ([#8892](https://github.com/dbt-labs/dbt-core/issues/8892))
### Fixes
- Copy dir during `dbt deps` if symlink fails ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- If --profile specified with dbt-init, create the project with the specified profile ([#6154](https://github.com/dbt-labs/dbt-core/issues/6154))
- Fixed double-underline ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Enable converting deprecation warnings to errors ([#8130](https://github.com/dbt-labs/dbt-core/issues/8130))
@@ -62,23 +135,59 @@
- Fix unbound local variable error in `checked_agg_time_dimension_for_measure` ([#8230](https://github.com/dbt-labs/dbt-core/issues/8230))
- Ensure runtime errors are raised for graph runnable tasks (compile, show, run, etc) ([#8166](https://github.com/dbt-labs/dbt-core/issues/8166))
- Fix retry not working with log-file-max-bytes ([#8297](https://github.com/dbt-labs/dbt-core/issues/8297))
- Add explicit support for integers for the show command ([#8153](https://github.com/dbt-labs/dbt-core/issues/8153))
- Detect changes to model access, version, or latest_version in state:modified ([#8189](https://github.com/dbt-labs/dbt-core/issues/8189))
- Add connection status into list of statuses for dbt debug ([#8350](https://github.com/dbt-labs/dbt-core/issues/8350))
- fix fqn-selection for external versioned models ([#8374](https://github.com/dbt-labs/dbt-core/issues/8374))
- Fix: DbtInternalError after model that previously ref'd external model is deleted ([#8375](https://github.com/dbt-labs/dbt-core/issues/8375))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
- Remedy performance regression by only writing run_results.json once. ([#8360](https://github.com/dbt-labs/dbt-core/issues/8360))
- Add support for swapping materialized views with tables/views and vice versa ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Turn breaking changes to contracted models into warnings for unversioned models ([#8384](https://github.com/dbt-labs/dbt-core/issues/8384), [#8282](https://github.com/dbt-labs/dbt-core/issues/8282))
- Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension` ([#8453](https://github.com/dbt-labs/dbt-core/issues/8453))
- fix ambiguous reference error for tests and versions when model name is duplicated across packages ([#8327](https://github.com/dbt-labs/dbt-core/issues/8327), [#8493](https://github.com/dbt-labs/dbt-core/issues/8493))
- Fix "Internal Error: Expected node <unique-id> not found in manifest" when depends_on set on ModelNodeArgs ([#8506](https://github.com/dbt-labs/dbt-core/issues/8506))
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
- Parse the correct schema version from manifest ([#8544](https://github.com/dbt-labs/dbt-core/issues/8544))
- make version comparison insensitive to order ([#8571](https://github.com/dbt-labs/dbt-core/issues/8571))
- Update metric helper functions to work with new semantic layer metrics ([#8134](https://github.com/dbt-labs/dbt-core/issues/8134))
- Disallow cleaning paths outside current working directory ([#8318](https://github.com/dbt-labs/dbt-core/issues/8318))
- Warn when --state == --target ([#8160](https://github.com/dbt-labs/dbt-core/issues/8160))
- update dbt show to include limit in DWH query ([#8496,](https://github.com/dbt-labs/dbt-core/issues/8496,), [#8417](https://github.com/dbt-labs/dbt-core/issues/8417))
- Support quoted parameter list for MultiOption CLI options. ([#8598](https://github.com/dbt-labs/dbt-core/issues/8598))
- Support global flags passed in after subcommands ([#6497](https://github.com/dbt-labs/dbt-core/issues/6497))
- Lower bound of `8.0.2` for `click` ([#8683](https://github.com/dbt-labs/dbt-core/issues/8683))
- Fixes test type edges filter ([#8692](https://github.com/dbt-labs/dbt-core/issues/8692))
- semantic models in graph selection ([#8589](https://github.com/dbt-labs/dbt-core/issues/8589))
- Support doc blocks in nested semantic model YAML ([#8509](https://github.com/dbt-labs/dbt-core/issues/8509))
- avoid double-rendering sql_header in dbt show ([#8739](https://github.com/dbt-labs/dbt-core/issues/8739))
- Fix tag selection for projects with semantic models ([#8749](https://github.com/dbt-labs/dbt-core/issues/8749))
- Foreign key constraint on incremental model results in Database Error ([#8022](https://github.com/dbt-labs/dbt-core/issues/8022))
- Support docs blocks on versioned model column descriptions ([#8540](https://github.com/dbt-labs/dbt-core/issues/8540))
- Enable seeds to be handled from stored manifest data ([#6875](https://github.com/dbt-labs/dbt-core/issues/6875))
- Override path-like args in dbt retry ([#8682](https://github.com/dbt-labs/dbt-core/issues/8682))
- Group updates on unmodified nodes are handled gracefully for state:modified ([#8371](https://github.com/dbt-labs/dbt-core/issues/8371))
- Partial parsing fix for adding groups and updating models at the same time ([#8697](https://github.com/dbt-labs/dbt-core/issues/8697))
- Fix partial parsing not working for semantic model change ([#8859](https://github.com/dbt-labs/dbt-core/issues/8859))
- Rework get_catalog implementation to retain previous adapter interface semantics ([#8846](https://github.com/dbt-labs/dbt-core/issues/8846))
- Add back contract enforcement for temporary tables on postgres ([#8857](https://github.com/dbt-labs/dbt-core/issues/8857))
- Add version to fqn when version==0 ([#8836](https://github.com/dbt-labs/dbt-core/issues/8836))
- Fix cased comparison in catalog-retrieval function. ([#8939](https://github.com/dbt-labs/dbt-core/issues/8939))
- Catalog queries now assign the correct type to materialized views ([#8864](https://github.com/dbt-labs/dbt-core/issues/8864))
- Make relation filtering None-tolerant for maximal flexibility across adapters. ([#8974](https://github.com/dbt-labs/dbt-core/issues/8974))
### Docs
- Corrected spelling of "Partiton" ([dbt-docs/#8100](https://github.com/dbt-labs/dbt-docs/issues/8100))
- Remove static SQL codeblock for metrics ([dbt-docs/#436](https://github.com/dbt-labs/dbt-docs/issues/436))
- fixed comment util.py ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Fix newline escapes and improve formatting in docker README ([dbt-docs/#8211](https://github.com/dbt-labs/dbt-docs/issues/8211))
- Display contract and column constraints on the model page ([dbt-docs/#433](https://github.com/dbt-labs/dbt-docs/issues/433))
- Display semantic model details in docs ([dbt-docs/#431](https://github.com/dbt-labs/dbt-docs/issues/431))
### Under the Hood
- Switch from hologram to mashumaro jsonschema ([#8426](https://github.com/dbt-labs/dbt-core/issues/8426))
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))
- format exception from dbtPlugin.initialize ([#8152](https://github.com/dbt-labs/dbt-core/issues/8152))
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
@@ -89,6 +198,20 @@
- Add return values to a number of functions for mypy ([#8389](https://github.com/dbt-labs/dbt-core/issues/8389))
- Fix mypy warnings for ManifestLoader.load() ([#8401](https://github.com/dbt-labs/dbt-core/issues/8401))
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))
- Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>, relation agnostic in /macros/relations ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Update typing to meet mypy standards ([#8396](https://github.com/dbt-labs/dbt-core/issues/8396))
- Mypy errors - adapters/factory.py ([#8387](https://github.com/dbt-labs/dbt-core/issues/8387))
- Added more type annotations. ([#8537](https://github.com/dbt-labs/dbt-core/issues/8537))
- Audit potential circular dependencies ([#8349](https://github.com/dbt-labs/dbt-core/issues/8349))
- Add functional test for advanced ref override ([#8566](https://github.com/dbt-labs/dbt-core/issues/8566))
- Add typing to __init__ in base.py ([#8398](https://github.com/dbt-labs/dbt-core/issues/8398))
- Fix untyped functions in task/runnable.py (mypy warning) ([#8402](https://github.com/dbt-labs/dbt-core/issues/8402))
- add a test for ephemeral cte injection ([#8225](https://github.com/dbt-labs/dbt-core/issues/8225))
- Fix test_numeric_values to look for more specific strings ([#8470](https://github.com/dbt-labs/dbt-core/issues/8470))
- Pin types-requests<2.31.0 in `dev-requirements.txt` ([#8789](https://github.com/dbt-labs/dbt-core/issues/8789))
- Add warning_tag to UnversionedBreakingChange ([#8827](https://github.com/dbt-labs/dbt-core/issues/8827))
- Update v10 manifest schema to match 1.6 for testing schema compatibility ([#8835](https://github.com/dbt-labs/dbt-core/issues/8835))
- Add a no-op runner for Saved Qeury ([#8893](https://github.com/dbt-labs/dbt-core/issues/8893))
### Dependencies
@@ -97,15 +220,27 @@
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))
- Support dbt-semantic-interfaces 0.2.0 ([#8250](https://github.com/dbt-labs/dbt-core/pull/8250))
- Bump docker/build-push-action from 4 to 5 ([#8783](https://github.com/dbt-labs/dbt-core/pull/8783))
- Upgrade dbt-semantic-interfaces dep to 0.3.0 ([#8819](https://github.com/dbt-labs/dbt-core/pull/8819))
- Begin using DSI 0.4.x ([#8892](https://github.com/dbt-labs/dbt-core/pull/8892))
### Contributors
- [@anjutiwari](https://github.com/anjutiwari) ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- [@benmosher](https://github.com/benmosher) ([#8480](https://github.com/dbt-labs/dbt-core/issues/8480))
- [@d-kaneshiro](https://github.com/d-kaneshiro) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#8153](https://github.com/dbt-labs/dbt-core/issues/8153), [#8589](https://github.com/dbt-labs/dbt-core/issues/8589))
- [@dylan-murray](https://github.com/dylan-murray) ([#8683](https://github.com/dbt-labs/dbt-core/issues/8683))
- [@ezraerb](https://github.com/ezraerb) ([#6154](https://github.com/dbt-labs/dbt-core/issues/6154))
- [@gem7318](https://github.com/gem7318) ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- [@jamezrin](https://github.com/jamezrin) ([#8211](https://github.com/dbt-labs/dbt-core/issues/8211))
- [@jusbaldw](https://github.com/jusbaldw) ([#6643](https://github.com/dbt-labs/dbt-core/issues/6643))
- [@lllong33](https://github.com/lllong33) ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- [@marcodamore](https://github.com/marcodamore) ([#436](https://github.com/dbt-labs/dbt-core/issues/436))
- [@mescanne](https://github.com/mescanne) ([#8614](https://github.com/dbt-labs/dbt-core/issues/8614))
- [@pgoslatara](https://github.com/pgoslatara) ([#8100](https://github.com/dbt-labs/dbt-core/issues/8100))
- [@philippeboyd](https://github.com/philippeboyd) ([#5374](https://github.com/dbt-labs/dbt-core/issues/5374))
- [@ramonvermeulen](https://github.com/ramonvermeulen) ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- [@renanleme](https://github.com/renanleme) ([#8692](https://github.com/dbt-labs/dbt-core/issues/8692))
## Previous Releases

View File

@@ -7,12 +7,12 @@ from dbt.exceptions import DbtRuntimeError
@dataclass
class Column:
# Note: This is automatically used by contract code
# No-op conversions (INTEGER => INT) have been removed.
# Any adapter that wants to take advantage of "translate_type"
# should create a ClassVar with the appropriate conversions.
TYPE_LABELS: ClassVar[Dict[str, str]] = {
"STRING": "TEXT",
"TIMESTAMP": "TIMESTAMP",
"FLOAT": "FLOAT",
"INTEGER": "INT",
"BOOLEAN": "BOOLEAN",
}
column: str
dtype: str

View File

@@ -17,9 +17,11 @@ from typing import (
Set,
Tuple,
Type,
TypedDict,
Union,
)
from dbt.adapters.capability import Capability, CapabilityDict
from dbt.contracts.graph.nodes import ColumnLevelConstraint, ConstraintType, ModelLevelConstraint
import agate
@@ -44,7 +46,13 @@ from dbt.exceptions import (
)
from dbt.adapters.protocol import AdapterConfig
from dbt.clients.agate_helper import empty_table, merge_tables, table_from_rows
from dbt.clients.agate_helper import (
empty_table,
get_column_value_uncased,
merge_tables,
table_from_rows,
Integer,
)
from dbt.clients.jinja import MacroGenerator
from dbt.contracts.graph.manifest import Manifest, MacroManifest
from dbt.contracts.graph.nodes import ResultNode
@@ -74,7 +82,9 @@ from dbt.adapters.cache import RelationsCache, _make_ref_key_dict
from dbt import deprecations
GET_CATALOG_MACRO_NAME = "get_catalog"
GET_CATALOG_RELATIONS_MACRO_NAME = "get_catalog_relations"
FRESHNESS_MACRO_NAME = "collect_freshness"
GET_RELATION_LAST_MODIFIED_MACRO_NAME = "get_relation_last_modified"
class ConstraintSupport(str, Enum):
@@ -109,7 +119,7 @@ def _catalog_filter_schemas(manifest: Manifest) -> Callable[[agate.Row], bool]:
return test
def _utc(dt: Optional[datetime], source: BaseRelation, field_name: str) -> datetime:
def _utc(dt: Optional[datetime], source: Optional[BaseRelation], field_name: str) -> datetime:
"""If dt has a timezone, return a new datetime that's in UTC. Otherwise,
assume the datetime is already for UTC and add the timezone.
"""
@@ -161,6 +171,12 @@ class PythonJobHelper:
raise NotImplementedError("PythonJobHelper submit function is not implemented yet")
class FreshnessResponse(TypedDict):
max_loaded_at: datetime
snapshotted_at: datetime
age: float # age in seconds
class BaseAdapter(metaclass=AdapterMeta):
"""The BaseAdapter provides an abstract base class for adapters.
@@ -222,6 +238,10 @@ class BaseAdapter(metaclass=AdapterMeta):
ConstraintType.foreign_key: ConstraintSupport.ENFORCED,
}
# This static member variable can be overriden in concrete adapter
# implementations to indicate adapter support for optional capabilities.
_capabilities = CapabilityDict({})
def __init__(self, config) -> None:
self.config = config
self.cache = RelationsCache()
@@ -415,7 +435,30 @@ class BaseAdapter(metaclass=AdapterMeta):
lowercase strings.
"""
info_schema_name_map = SchemaSearchMap()
nodes: Iterator[ResultNode] = chain(
relations = self._get_catalog_relations(manifest)
for relation in relations:
info_schema_name_map.add(relation)
# result is a map whose keys are information_schema Relations without
# identifiers that have appropriate database prefixes, and whose values
# are sets of lowercase schema names that are valid members of those
# databases
return info_schema_name_map
def _get_catalog_relations_by_info_schema(
self, relations
) -> Dict[InformationSchema, List[BaseRelation]]:
relations_by_info_schema: Dict[InformationSchema, List[BaseRelation]] = dict()
for relation in relations:
info_schema = relation.information_schema_only()
if info_schema not in relations_by_info_schema:
relations_by_info_schema[info_schema] = []
relations_by_info_schema[info_schema].append(relation)
return relations_by_info_schema
def _get_catalog_relations(self, manifest: Manifest) -> List[BaseRelation]:
nodes = chain(
[
node
for node in manifest.nodes.values()
@@ -423,14 +466,9 @@ class BaseAdapter(metaclass=AdapterMeta):
],
manifest.sources.values(),
)
for node in nodes:
relation = self.Relation.create_from(self.config, node)
info_schema_name_map.add(relation)
# result is a map whose keys are information_schema Relations without
# identifiers that have appropriate database prefixes, and whose values
# are sets of lowercase schema names that are valid members of those
# databases
return info_schema_name_map
relations = [self.Relation.create_from(self.config, n) for n in nodes]
return relations
def _relations_cache_for_schemas(
self, manifest: Manifest, cache_schemas: Optional[Set[BaseRelation]] = None
@@ -925,6 +963,17 @@ class BaseAdapter(metaclass=AdapterMeta):
"""
raise NotImplementedError("`convert_number_type` is not implemented for this adapter!")
@classmethod
def convert_integer_type(cls, agate_table: agate.Table, col_idx: int) -> str:
"""Return the type in the database that best maps to the agate.Number
type for the given agate table and column index.
:param agate_table: The table
:param col_idx: The index into the agate table for the column.
:return: The name of the type in the database
"""
return "integer"
@classmethod
@abc.abstractmethod
def convert_boolean_type(cls, agate_table: agate.Table, col_idx: int) -> str:
@@ -982,6 +1031,7 @@ class BaseAdapter(metaclass=AdapterMeta):
def convert_agate_type(cls, agate_table: agate.Table, col_idx: int) -> Optional[str]:
agate_type: Type = agate_table.column_types[col_idx]
conversions: List[Tuple[Type, Callable[..., str]]] = [
(Integer, cls.convert_integer_type),
(agate.Text, cls.convert_text_type),
(agate.Number, cls.convert_number_type),
(agate.Boolean, cls.convert_boolean_type),
@@ -1093,25 +1143,108 @@ class BaseAdapter(metaclass=AdapterMeta):
results = self._catalog_filter_table(table, manifest) # type: ignore[arg-type]
return results
def get_catalog(self, manifest: Manifest) -> Tuple[agate.Table, List[Exception]]:
schema_map = self._get_catalog_schemas(manifest)
def _get_one_catalog_by_relations(
self,
information_schema: InformationSchema,
relations: List[BaseRelation],
manifest: Manifest,
) -> agate.Table:
kwargs = {
"information_schema": information_schema,
"relations": relations,
}
table = self.execute_macro(
GET_CATALOG_RELATIONS_MACRO_NAME,
kwargs=kwargs,
# pass in the full manifest, so we get any local project
# overrides
manifest=manifest,
)
results = self._catalog_filter_table(table, manifest) # type: ignore[arg-type]
return results
def get_filtered_catalog(
self, manifest: Manifest, relations: Optional[Set[BaseRelation]] = None
):
catalogs: agate.Table
if (
relations is None
or len(relations) > 100
or not self.supports(Capability.SchemaMetadataByRelations)
):
# Do it the traditional way. We get the full catalog.
catalogs, exceptions = self.get_catalog(manifest)
else:
# Do it the new way. We try to save time by selecting information
# only for the exact set of relations we are interested in.
catalogs, exceptions = self.get_catalog_by_relations(manifest, relations)
if relations and catalogs:
relation_map = {
(
r.database.casefold() if r.database else None,
r.schema.casefold() if r.schema else None,
r.identifier.casefold() if r.identifier else None,
)
for r in relations
}
def in_map(row: agate.Row):
d = _expect_row_value("table_database", row)
s = _expect_row_value("table_schema", row)
i = _expect_row_value("table_name", row)
d = d.casefold() if d is not None else None
s = s.casefold() if s is not None else None
i = i.casefold() if i is not None else None
return (d, s, i) in relation_map
catalogs = catalogs.where(in_map)
return catalogs, exceptions
def row_matches_relation(self, row: agate.Row, relations: Set[BaseRelation]):
pass
def get_catalog(self, manifest: Manifest) -> Tuple[agate.Table, List[Exception]]:
with executor(self.config) as tpe:
futures: List[Future[agate.Table]] = []
schema_map: SchemaSearchMap = self._get_catalog_schemas(manifest)
for info, schemas in schema_map.items():
if len(schemas) == 0:
continue
name = ".".join([str(info.database), "information_schema"])
fut = tpe.submit_connected(
self, name, self._get_one_catalog, info, schemas, manifest
)
futures.append(fut)
catalogs, exceptions = catch_as_completed(futures)
catalogs, exceptions = catch_as_completed(futures)
return catalogs, exceptions
def get_catalog_by_relations(
self, manifest: Manifest, relations: Set[BaseRelation]
) -> Tuple[agate.Table, List[Exception]]:
with executor(self.config) as tpe:
futures: List[Future[agate.Table]] = []
relations_by_schema = self._get_catalog_relations_by_info_schema(relations)
for info_schema in relations_by_schema:
name = ".".join([str(info_schema.database), "information_schema"])
relations = set(relations_by_schema[info_schema])
fut = tpe.submit_connected(
self,
name,
self._get_one_catalog_by_relations,
info_schema,
relations,
manifest,
)
futures.append(fut)
catalogs, exceptions = catch_as_completed(futures)
return catalogs, exceptions
def cancel_open_connections(self):
"""Cancel all open connections."""
return self.connections.cancel_open()
@@ -1122,7 +1255,7 @@ class BaseAdapter(metaclass=AdapterMeta):
loaded_at_field: str,
filter: Optional[str],
manifest: Optional[Manifest] = None,
) -> Tuple[Optional[AdapterResponse], Dict[str, Any]]:
) -> Tuple[Optional[AdapterResponse], FreshnessResponse]:
"""Calculate the freshness of sources in dbt, and return it"""
kwargs: Dict[str, Any] = {
"source": source,
@@ -1157,13 +1290,52 @@ class BaseAdapter(metaclass=AdapterMeta):
snapshotted_at = _utc(table[0][1], source, loaded_at_field)
age = (snapshotted_at - max_loaded_at).total_seconds()
freshness = {
freshness: FreshnessResponse = {
"max_loaded_at": max_loaded_at,
"snapshotted_at": snapshotted_at,
"age": age,
}
return adapter_response, freshness
def calculate_freshness_from_metadata(
self,
source: BaseRelation,
manifest: Optional[Manifest] = None,
) -> Tuple[Optional[AdapterResponse], FreshnessResponse]:
kwargs: Dict[str, Any] = {
"information_schema": source.information_schema_only(),
"relations": [source],
}
result = self.execute_macro(
GET_RELATION_LAST_MODIFIED_MACRO_NAME, kwargs=kwargs, manifest=manifest
)
adapter_response, table = result.response, result.table # type: ignore[attr-defined]
try:
row = table[0]
last_modified_val = get_column_value_uncased("last_modified", row)
snapshotted_at_val = get_column_value_uncased("snapshotted_at", row)
except Exception:
raise MacroResultError(GET_RELATION_LAST_MODIFIED_MACRO_NAME, table)
if last_modified_val is None:
# Interpret missing value as "infinitely long ago"
max_loaded_at = datetime(1, 1, 1, 0, 0, 0, tzinfo=pytz.UTC)
else:
max_loaded_at = _utc(last_modified_val, None, "last_modified")
snapshotted_at = _utc(snapshotted_at_val, None, "snapshotted_at")
age = (snapshotted_at - max_loaded_at).total_seconds()
freshness: FreshnessResponse = {
"max_loaded_at": max_loaded_at,
"snapshotted_at": snapshotted_at,
"age": age,
}
return adapter_response, freshness
def pre_model_hook(self, config: Mapping[str, Any]) -> Any:
"""A hook for running some operation before the model materialization
runs. The hook can assume it has a connection available.
@@ -1437,6 +1609,14 @@ class BaseAdapter(metaclass=AdapterMeta):
else:
return None
@classmethod
def capabilities(cls) -> CapabilityDict:
return cls._capabilities
@classmethod
def supports(cls, capability: Capability) -> bool:
return bool(cls.capabilities()[capability])
COLUMNS_EQUAL_SQL = """
with diff_count as (

View File

@@ -459,11 +459,11 @@ class SchemaSearchMap(Dict[InformationSchema, Set[Optional[str]]]):
self[key].add(schema)
def search(self) -> Iterator[Tuple[InformationSchema, Optional[str]]]:
for information_schema_name, schemas in self.items():
for information_schema, schemas in self.items():
for schema in schemas:
yield information_schema_name, schema
yield information_schema, schema
def flatten(self, allow_multiple_databases: bool = False):
def flatten(self, allow_multiple_databases: bool = False) -> "SchemaSearchMap":
new = self.__class__()
# make sure we don't have multiple databases if allow_multiple_databases is set to False

View File

@@ -0,0 +1,52 @@
from dataclasses import dataclass
from enum import Enum
from typing import Optional, DefaultDict, Mapping
class Capability(str, Enum):
"""Enumeration of optional adapter features which can be probed using BaseAdapter.has_feature()"""
SchemaMetadataByRelations = "SchemaMetadataByRelations"
"""Indicates efficient support for retrieving schema metadata for a list of relations, rather than always retrieving
all the relations in a schema."""
TableLastModifiedMetadata = "TableLastModifiedMetadata"
"""Indicates support for determining the time of the last table modification by querying database metadata."""
class Support(str, Enum):
Unknown = "Unknown"
"""The adapter has not declared whether this capability is a feature of the underlying DBMS."""
Unsupported = "Unsupported"
"""This capability is not possible with the underlying DBMS, so the adapter does not implement related macros."""
NotImplemented = "NotImplemented"
"""This capability is available in the underlying DBMS, but support has not yet been implemented in the adapter."""
Versioned = "Versioned"
"""Some versions of the DBMS supported by the adapter support this capability and the adapter has implemented any
macros needed to use it."""
Full = "Full"
"""All versions of the DBMS supported by the adapter support this capability and the adapter has implemented any
macros needed to use it."""
@dataclass
class CapabilitySupport:
support: Support
first_version: Optional[str] = None
def __bool__(self):
return self.support == Support.Versioned or self.support == Support.Full
class CapabilityDict(DefaultDict[Capability, CapabilitySupport]):
def __init__(self, vals: Mapping[Capability, CapabilitySupport]):
super().__init__(self._default)
self.update(vals)
@staticmethod
def _default():
return CapabilitySupport(support=Support.Unknown)

View File

@@ -75,6 +75,10 @@ class SQLAdapter(BaseAdapter):
decimals = agate_table.aggregate(agate.MaxPrecision(col_idx)) # type: ignore[attr-defined]
return "float8" if decimals else "integer"
@classmethod
def convert_integer_type(cls, agate_table: agate.Table, col_idx: int) -> str:
return "integer"
@classmethod
def convert_boolean_type(cls, agate_table: agate.Table, col_idx: int) -> str:
return "boolean"

View File

@@ -24,6 +24,7 @@ if os.name != "nt":
FLAGS_DEFAULTS = {
"INDIRECT_SELECTION": "eager",
"TARGET_PATH": None,
"WARN_ERROR": None,
# Cli args without user_config or env var option.
"FULL_REFRESH": False,
"STRICT_MODE": False,
@@ -57,7 +58,7 @@ def args_to_context(args: List[str]) -> Context:
from dbt.cli.main import cli
cli_ctx = cli.make_context(cli.name, args)
# Split args if they're a comma seperated string.
# Split args if they're a comma separated string.
if len(args) == 1 and "," in args[0]:
args = args[0].split(",")
sub_command_name, sub_command, args = cli.resolve_command(cli_ctx, args)
@@ -78,7 +79,6 @@ class Flags:
def __init__(
self, ctx: Optional[Context] = None, user_config: Optional[UserConfig] = None
) -> None:
# Set the default flags.
for key, value in FLAGS_DEFAULTS.items():
object.__setattr__(self, key, value)
@@ -120,7 +120,6 @@ class Flags:
# respected over DBT_PRINT or --print.
new_name: Union[str, None] = None
if param_name in DEPRECATED_PARAMS:
# Deprecated env vars can only be set via env var.
# We use the deprecated option in click to serialize the value
# from the env var string.
@@ -315,7 +314,6 @@ def command_params(command: CliCommand, args_dict: Dict[str, Any]) -> CommandPar
default_args = set([x.lower() for x in FLAGS_DEFAULTS.keys()])
res = command.to_list()
for k, v in args_dict.items():
k = k.lower()
# if a "which" value exists in the args dict, it should match the command provided
@@ -327,7 +325,9 @@ def command_params(command: CliCommand, args_dict: Dict[str, Any]) -> CommandPar
continue
# param was assigned from defaults and should not be included
if k not in (cmd_args | prnt_args) - default_args:
if k not in (cmd_args | prnt_args) or (
k in default_args and v == FLAGS_DEFAULTS[k.upper()]
):
continue
# if the param is in parent args, it should come before the arg name
@@ -340,6 +340,10 @@ def command_params(command: CliCommand, args_dict: Dict[str, Any]) -> CommandPar
spinal_cased = k.replace("_", "-")
# MultiOption flags come back as lists, but we want to pass them as space separated strings
if isinstance(v, list):
v = " ".join(v)
if k == "macro" and command == CliCommand.RUN_OPERATION:
add_fn(v)
# None is a Singleton, False is a Flyweight, only one instance of each.

View File

@@ -1,3 +1,4 @@
import functools
from copy import copy
from dataclasses import dataclass
from typing import Callable, List, Optional, Union
@@ -118,6 +119,44 @@ class dbtRunner:
)
# approach from https://github.com/pallets/click/issues/108#issuecomment-280489786
def global_flags(func):
@p.cache_selected_only
@p.debug
@p.deprecated_print
@p.enable_legacy_logger
@p.fail_fast
@p.log_cache_events
@p.log_file_max_bytes
@p.log_format_file
@p.log_level
@p.log_level_file
@p.log_path
@p.macro_debugging
@p.partial_parse
@p.partial_parse_file_path
@p.partial_parse_file_diff
@p.populate_cache
@p.print
@p.printer_width
@p.quiet
@p.record_timing_info
@p.send_anonymous_usage_stats
@p.single_threaded
@p.static_parser
@p.use_colors
@p.use_colors_file
@p.use_experimental_parser
@p.version
@p.version_check
@p.write_json
@functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
# dbt
@click.group(
context_settings={"help_option_names": ["-h", "--help"]},
@@ -126,38 +165,11 @@ class dbtRunner:
epilog="Specify one of these sub-commands and you can find more help from there.",
)
@click.pass_context
@p.cache_selected_only
@p.debug
@p.deprecated_print
@p.enable_legacy_logger
@p.fail_fast
@p.log_cache_events
@p.log_file_max_bytes
@p.log_format
@p.log_format_file
@p.log_level
@p.log_level_file
@p.log_path
@p.macro_debugging
@p.partial_parse
@p.partial_parse_file_path
@p.partial_parse_file_diff
@p.populate_cache
@p.print
@p.printer_width
@p.quiet
@p.record_timing_info
@p.send_anonymous_usage_stats
@p.single_threaded
@p.static_parser
@p.use_colors
@p.use_colors_file
@p.use_experimental_parser
@p.version
@p.version_check
@global_flags
@p.warn_error
@p.warn_error_options
@p.write_json
@p.log_format
@p.show_resource_report
def cli(ctx, **kwargs):
"""An ELT tool for managing your SQL transformations and data models.
For more documentation on these commands, visit: docs.getdbt.com
@@ -167,13 +179,14 @@ def cli(ctx, **kwargs):
# dbt build
@cli.command("build")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@p.fail_fast
@p.favor_state
@p.deprecated_favor_state
@p.full_refresh
@p.include_saved_query
@p.indirect_selection
@p.profile
@p.profiles_dir
@@ -190,7 +203,6 @@ def cli(ctx, **kwargs):
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -213,6 +225,8 @@ def build(ctx, **kwargs):
# dbt clean
@cli.command("clean")
@click.pass_context
@global_flags
@p.clean_project_files_only
@p.profile
@p.profiles_dir
@p.project_dir
@@ -235,6 +249,7 @@ def clean(ctx, **kwargs):
# dbt docs
@cli.group()
@click.pass_context
@global_flags
def docs(ctx, **kwargs):
"""Generate or serve the documentation website for your project"""
@@ -242,6 +257,7 @@ def docs(ctx, **kwargs):
# dbt docs generate
@docs.command("generate")
@click.pass_context
@global_flags
@p.compile_docs
@p.defer
@p.deprecated_defer
@@ -254,6 +270,7 @@ def docs(ctx, **kwargs):
@p.select
@p.selector
@p.empty_catalog
@p.static
@p.state
@p.defer_state
@p.deprecated_state
@@ -261,7 +278,6 @@ def docs(ctx, **kwargs):
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -284,6 +300,7 @@ def docs_generate(ctx, **kwargs):
# dbt docs serve
@docs.command("serve")
@click.pass_context
@global_flags
@p.browser
@p.port
@p.profile
@@ -312,6 +329,7 @@ def docs_serve(ctx, **kwargs):
# dbt compile
@cli.command("compile")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@@ -335,7 +353,6 @@ def docs_serve(ctx, **kwargs):
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -359,6 +376,7 @@ def compile(ctx, **kwargs):
# dbt show
@cli.command("show")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@@ -382,7 +400,6 @@ def compile(ctx, **kwargs):
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -406,6 +423,7 @@ def show(ctx, **kwargs):
# dbt debug
@cli.command("debug")
@click.pass_context
@global_flags
@p.debug_connection
@p.config_dir
@p.profile
@@ -413,7 +431,6 @@ def show(ctx, **kwargs):
@p.project_dir
@p.target
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
def debug(ctx, **kwargs):
@@ -432,18 +449,46 @@ def debug(ctx, **kwargs):
# dbt deps
@cli.command("deps")
@click.pass_context
@global_flags
@p.profile
@p.profiles_dir_exists_false
@p.project_dir
@p.target
@p.vars
@p.source
@p.dry_run
@p.lock
@p.upgrade
@p.add_package
@requires.postflight
@requires.preflight
@requires.unset_profile
@requires.project
def deps(ctx, **kwargs):
"""Pull the most recent version of the dependencies listed in packages.yml"""
task = DepsTask(ctx.obj["flags"], ctx.obj["project"])
"""Install dbt packages specified.
In the following case, a new `package-lock.yml` will be generated and the packages are installed:
- user updated the packages.yml
- user specify the flag --update, which means for packages that are specified as a
range, dbt-core will try to install the newer version
Otherwise, deps will use `package-lock.yml` as source of truth to install packages.
There is a way to add new packages by providing an `--add-package` flag to deps command
which will allow user to specify a package they want to add in the format of packagename@version.
"""
flags = ctx.obj["flags"]
if flags.ADD_PACKAGE:
if not flags.ADD_PACKAGE["version"] and flags.SOURCE != "local":
raise BadOptionUsage(
message=f"Version is required in --add-package when a package when source is {flags.SOURCE}",
option_name="--add-package",
)
else:
if flags.DRY_RUN:
raise BadOptionUsage(
message="Invalid flag `--dry-run` when not using `--add-package`.",
option_name="--dry-run",
)
task = DepsTask(flags, ctx.obj["project"])
results = task.run()
success = task.interpret_results(results)
return results, success
@@ -452,6 +497,7 @@ def deps(ctx, **kwargs):
# dbt init
@cli.command("init")
@click.pass_context
@global_flags
# for backwards compatibility, accept 'project_name' as an optional positional argument
@click.argument("project_name", required=False)
@p.profile
@@ -474,6 +520,7 @@ def init(ctx, **kwargs):
# dbt list
@cli.command("list")
@click.pass_context
@global_flags
@p.exclude
@p.indirect_selection
@p.models
@@ -519,6 +566,7 @@ cli.add_command(ls, "ls")
# dbt parse
@cli.command("parse")
@click.pass_context
@global_flags
@p.profile
@p.profiles_dir
@p.project_dir
@@ -526,7 +574,6 @@ cli.add_command(ls, "ls")
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -536,19 +583,18 @@ cli.add_command(ls, "ls")
def parse(ctx, **kwargs):
"""Parses the project and provides information on performance"""
# manifest generation and writing happens in @requires.manifest
return ctx.obj["manifest"], True
# dbt run
@cli.command("run")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.favor_state
@p.deprecated_favor_state
@p.exclude
@p.fail_fast
@p.full_refresh
@p.profile
@p.profiles_dir
@@ -562,7 +608,6 @@ def parse(ctx, **kwargs):
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -585,6 +630,7 @@ def run(ctx, **kwargs):
# dbt retry
@cli.command("retry")
@click.pass_context
@global_flags
@p.project_dir
@p.profiles_dir
@p.vars
@@ -592,19 +638,18 @@ def run(ctx, **kwargs):
@p.target
@p.state
@p.threads
@p.fail_fast
@p.full_refresh
@requires.postflight
@requires.preflight
@requires.profile
@requires.project
@requires.runtime_config
@requires.manifest
def retry(ctx, **kwargs):
"""Retry the nodes that failed in the previous run."""
# Retry will parse manifest inside the task after we consolidate the flags
task = RetryTask(
ctx.obj["flags"],
ctx.obj["runtime_config"],
ctx.obj["manifest"],
)
results = task.run()
@@ -615,6 +660,7 @@ def retry(ctx, **kwargs):
# dbt clone
@cli.command("clone")
@click.pass_context
@global_flags
@p.defer_state
@p.exclude
@p.full_refresh
@@ -629,7 +675,6 @@ def retry(ctx, **kwargs):
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.preflight
@requires.profile
@requires.project
@@ -652,6 +697,7 @@ def clone(ctx, **kwargs):
# dbt run operation
@cli.command("run-operation")
@click.pass_context
@global_flags
@click.argument("macro")
@p.args
@p.profile
@@ -683,6 +729,7 @@ def run_operation(ctx, **kwargs):
# dbt seed
@cli.command("seed")
@click.pass_context
@global_flags
@p.exclude
@p.full_refresh
@p.profile
@@ -698,7 +745,6 @@ def run_operation(ctx, **kwargs):
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile
@@ -720,6 +766,7 @@ def seed(ctx, **kwargs):
# dbt snapshot
@cli.command("snapshot")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@@ -759,6 +806,7 @@ def snapshot(ctx, **kwargs):
# dbt source
@cli.group()
@click.pass_context
@global_flags
def source(ctx, **kwargs):
"""Manage your project's sources"""
@@ -766,6 +814,7 @@ def source(ctx, **kwargs):
# dbt source freshness
@source.command("freshness")
@click.pass_context
@global_flags
@p.exclude
@p.output_path # TODO: Is this ok to re-use? We have three different output params, how much can we consolidate?
@p.profile
@@ -808,10 +857,10 @@ cli.commands["source"].add_command(snapshot_freshness, "snapshot-freshness") #
# dbt test
@cli.command("test")
@click.pass_context
@global_flags
@p.defer
@p.deprecated_defer
@p.exclude
@p.fail_fast
@p.favor_state
@p.deprecated_favor_state
@p.indirect_selection
@@ -828,7 +877,6 @@ cli.commands["source"].add_command(snapshot_freshness, "snapshot-freshness") #
@p.target_path
@p.threads
@p.vars
@p.version_check
@requires.postflight
@requires.preflight
@requires.profile

Some files were not shown because too many files have changed in this diff Show More