Compare commits

..

79 Commits

Author SHA1 Message Date
Michelle Ark
60f87411d5 Merge branch 'feature/decouple-adapters-from-core' into macro-resolver-remove-lazy-loading 2023-12-07 08:27:45 +09:00
Michelle Ark
eb96e3deec Add RelationConfig Protocol for use in Relation.create_from (#9210)
* move relation contract to dbt.adapters

* changelog entry

* first pass: clean up relation.create_from

* type ignores

* type ignore

* changelog entry

* update RelationConfig variable names
2023-12-06 10:46:45 -08:00
Michelle Ark
f68af070f3 remove manifest from adapter.execute_macro, replace with MacroResolver + remove lazy loading 2023-12-06 23:27:42 +09:00
Michelle Ark
7ad1accf2b Merge branch 'feature/decouple-adapters-from-core' into relation-create-from-refactor 2023-12-06 11:19:16 +09:00
Michelle Ark
ed8f5d38e4 Remove ResultNode usage from connections (#9211) 2023-12-06 11:04:26 +09:00
Michelle Ark
7ad6aa18da update RelationConfig variable names 2023-12-06 11:02:39 +09:00
Michelle Ark
6796edd66e Merge branch 'feature/decouple-adapters-from-core' into relation-create-from-refactor 2023-12-06 10:58:41 +09:00
Michelle Ark
e01eb30884 Remove usage of dbt.contracts.relation in dbt/adapters (#9207) 2023-12-06 10:34:56 +09:00
Michelle Ark
ba53f053fd changelog entry 2023-12-05 17:07:32 +09:00
Michelle Ark
b8de881ed3 type ignore 2023-12-05 17:06:49 +09:00
Michelle Ark
d7d5e2335c type ignores 2023-12-05 15:41:13 +09:00
Michelle Ark
160d0db238 first pass: clean up relation.create_from 2023-12-05 15:03:16 +09:00
Michelle Ark
2cee8652a6 changelog entry 2023-12-05 12:06:06 +09:00
Michelle Ark
7f777f8a42 move relation contract to dbt.adapters 2023-12-05 12:03:16 +09:00
Michelle Ark
00f49206e9 Decouple macro generator from adapters (#9149) 2023-12-05 10:51:51 +09:00
Michelle Ark
1bca662883 first pass: adapter migration script (#9160) 2023-12-05 08:41:11 +09:00
colin-rogers-dbt
41ac915949 Refactor EventManager setup and interaction (#9180)
* moving types_pb2.py to common/events

* move event manager setup back to core, remove ref to global EVENT_MANAGER and clean up event manager functions

* move invocation_id from events to first class common concept

* move lowercase utils to common

* move lowercase utils to common

* ref CAPTURE_STREAM through method

* add changie
2023-12-01 08:48:19 -08:00
Mila Page
373125ecb8 Move the semver package to common and alter references. (#9166)
* Move the semver package to common and alter references.

* Alter leftover references to dbt.semver, this time using from syntax.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-11-29 11:03:13 -08:00
Michelle Ark
294ad82e50 delete accidentally merged types_pb2.py 2023-11-28 20:20:59 -05:00
Michelle Ark
12bd1e87fb Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-28 20:08:33 -05:00
colin-rogers-dbt
8bad75c65b Move adapter logger to adapters (#9165)
* moving types_pb2.py to common/events

* Move AdapterLogger to adapter folder

* add changie
2023-11-28 15:05:30 -08:00
Kshitij Aranke
c836b7585e Fix #9119: Get sources working again in dbt docs generate (#9159) 2023-11-28 18:21:44 +00:00
Michelle Ark
220f56d8d2 remove adapter.get_compiler (#9134) 2023-11-28 09:27:00 -05:00
Jeremy Cohen
32fde75504 Fixups for deps lock file (#9147)
* Update git revision with commit SHA

* Use PackageRenderer for lock file

* add unit tests for git and tarball packages

* deepcopy unrendered_packages_data before iteration, fix remaining tests

* Add functional tests

* Add changelog entries

* Assert one more

---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2023-11-28 10:06:05 +01:00
Michelle Ark
615ad1fe2d move include/global_project to adapters (#8930) 2023-11-27 18:14:02 -05:00
Mike Alfare
81236a3dca Fix test_current_timestamp_matches_utc (#9058)
* loosen the tolerance for matching timestamps due to a slow system clock on the macos runner
* changelog
2023-11-27 13:15:20 -05:00
Michelle Ark
6d834a18ed clean up plugin readmes (#9130) 2023-11-27 11:10:26 -05:00
Peter Webb
2ab0f7b26b Decouple adapter constraints from core (#9054)
* Move constraints to dbt.common

* Move constraints to contracts folder, per review

* Add a changelog entry.
2023-11-21 12:16:39 -05:00
Peter Webb
9bb970e6ef Cache plugin modules (#9031)
* Cache plugin modules

* Add changelog entry

* Use lru_cache to keep Python 3.8 happy.
2023-11-20 17:50:16 -05:00
Michelle Ark
e56a5dae8b Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific (#9051) 2023-11-20 17:26:11 -05:00
Jeremy Cohen
1c9cec1787 Roadmap update (Nov 2023) (#9120)
* Roadmap update (Nov 2023)

* fix typos

* Write out title

---------

Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
2023-11-18 14:54:03 -05:00
Quigley Malcolm
4d02ef637b Update comment about input_measures setting in metric parsing (#9118)
In [dbt-labs/dbt-core#7984](https://github.com/dbt-labs/dbt-core/pull/7984)
we began setting a metrics `type_params.input_measures` during metric
processing post parsing. However in that PR we didn't clean up the comment
in the parser about setting `input_measures`. This is that post fact
cleanup.
2023-11-17 12:24:58 -08:00
Adam Lopez
19f027b7a7 import nested tarball package dependencies (#8855)
* detected nested dependencies in tarball dependency

* rm git changes

* add changelog entry

* update tarball to use download_and_untar

* update tests + handle package lock entry creation for tarball

* use fixes from 017faf4bd1, add download_and_untar in install step

* quote version value

* Update .changes/unreleased/Fixes-20231013-130943.yaml

Co-authored-by: Jeremy Cohen <jtcohen6@gmail.com>

* Update .changes/unreleased/Fixes-20231013-130943.yaml

---------

Co-authored-by: Emily Rockman <ebuschang@gmail.com>
Co-authored-by: Jeremy Cohen <jtcohen6@gmail.com>
2023-11-16 12:38:47 -06:00
Michelle Ark
1d0a3e92c8 Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-16 11:28:51 -05:00
Quigley Malcolm
ab90c777d0 Add test asserting GraphRunnableTasks attempt to cancel connections on SystemExit (#9101)
* Add test asserting GraphRunnableTasks attempt to cancel connections on SystemExit

* Add test asserting GraphRunnableTasks attempt to cancel connections on KeyboardInterrupt

* Add test asserting GraphRunnableNode doesn't try to cancel connections on generic Exception
2023-11-15 16:03:43 -08:00
Michelle Ark
3902137dfc test pinning ddtrace (#9090) 2023-11-15 17:18:56 -05:00
Michelle Ark
0131feac68 document scripts/collect-artifact-schema (#9085) 2023-11-15 16:01:30 -05:00
Quigley Malcolm
017faf4bd1 Fix dbt deps failing on tarballs (#9068)
* tarball lockfile fix

* Add changie doc for tarball deps issue

* Add integration test for ensuring tarball package specification works

This test was written _after_ the fix was commited. However, I ran this
test against main without the fix and it failed. After running the test
with the tarball fix, it passed.

* Remove unnecessary `tarball` conditional logic in `PackageConfig.validate`

We had a conditional to skip validation for a package if the package
included the `tarball` key. However, this conditional always returned
false as it was nested inside a conditional that the package had the
default `package` key, which means it's not a tarball package, but a
package package (maybe we need better differentiation here). If we need
additional validation for tarballs down the road, we should do that one
level up. At this time we have no additional validaitons to add.

* Fix typos in changie doc for tarball deps issue

* Improve tarball package test naming and add related unhappy path test

* Remove unnecessary `setUp` fixture from tarball package tests

We initially included this fixture due to copy and pasting another
test. However, this `setUp` fixture isn't actually necessary for the
tarball dependency tests.

---------

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-11-14 11:38:40 -08:00
Quigley Malcolm
c2f7d75e9e Support hierarchical config setting for SavedQueryExport configs (#9065)
* Add test asserting `SavedQuery` configs can be set from `dbt_project.yml`

* Allow extraneous properties in Export configs

This brings the Export config object more in line with how other config
objects are specified in the unparsed definition. It allows for specifying
of extra configs, although they won't get propagate to the final config.

* Add `ExportConfig` options to `SavedQueryConfig` options

This allows for specifying `ExportConfig` options at the `SavedQueryConfig` level.
This also therefore allows these options to be specified in the dbt_project.yml
config. The plan in the follow up commit is to merge the `SavedQueryConfig` options
into all configs of `Exports` belonging to the saved query.

There are a couple caveots to call out:
1. We've used `schema` instead of `schema_name` on the `SavedQueryConfig` despite
it being called `schema_name` on the `ExportConfig`. This is because need `schema_name`
to be the name of the property on the `ExportConfig`, but `schema` is the user facing
specification.
2. We didn't add the `ExportConfig` `alias` property to the `SavedQueryConfig` This
is because `alias` will always be specific to a single export, and thus it doesn't
make sense to allow defining it on the `SavedQueryConfig` to then apply to all
`Exports` belonging to the `SavedQuery`

* Begin inheriting configs from saved query config, and transitively from project config

Export configs will now inherit from saved query configs, with a preference
for export config specifications. That is to say an export config will inherity
a config attr from the saved query config only if a value hasn't been supplied
on the export config directly. Additionally because the saved query config has
a similar relationship with the project config, exports configs can inherit
from the project config (again with a preference for export config specifications).

* Correct conditional in export config building for map schema to schema_name

I somehow wrote a really weird, but also valid, conditional statement. Previously
the conditional was
```
if combined.get("schema") is not combined.get("schema_name") is None:
```
which basically checked whether `schema` was a boolean that didn't match
the boolean of whether `schema_name` was None. This would pretty much
always evaluate to True because `schema` should be a string or none, not
a bool, and thus would never match the right hand side. Crazy. It has now
been fixed to do the thing we want to it to do. If `schema` isn't `None`,
and `schema_name` is `None`, then set `schema_name` to have the value of
`schema`.

* Update parameter names in `_get_export_config` to be more verbose
2023-11-14 11:13:41 -08:00
colin-rogers-dbt
51b94b26cc Refactor Base Exceptions (#8989)
* moving types_pb2.py to common/events

* Refactor Base Exceptions

* update make_log_dir_if_missing to handle str

* move remaining adapters exception imports to common/adapters
---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2023-11-10 09:54:36 -05:00
colin-rogers-dbt
e24f9b3da7 Revert "[Feature] Supports non half width alphanumeric for generic test (#8203)" (#9048)
This reverts commit f45b013321.
2023-11-09 15:05:12 -08:00
Mike Alfare
b58e8e3ffc Migrate TestCatalogRelationTypes to the shared adapter tests framework (#8952)
* changelog
* move test to shared adapter zone for reuse
* add comment explaining how to configure this test case
2023-11-09 14:57:45 -05:00
d-kaneshiro
f45b013321 [Feature] Supports non half width alphanumeric for generic test (#8203)
* Supports non half width alphanumeric for generic test

* Unify conditional statements

* add CHANGELOG entries

* added test for Japanese

* Move the fix further upstream

* Remove the changes in core/dbt/task/runnable.py

* Fix accidental removal of `_` substitution character

---------

Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
2023-11-09 10:50:59 -07:00
Thomas Lento
e547c0ec64 Update typing-extensions version to >= 4.4 (#9012) 2023-11-08 16:19:28 -05:00
colin-rogers-dbt
6871fc46b5 Add drop_schema_named macro (#8868)
* add drop_schema_named

* give drop_schema_named a default
2023-11-08 12:50:25 -08:00
Ben Mosher
931b2dbe40 During node execution, also treat SystemExit as an interrupt. (#8994)
IDE worker process raises SystemExit in multiple scenarios, including user abort of a command.
2023-11-08 15:34:18 -05:00
Leo Griffiths
bb35b3eb87 fix: get_custom_database docs describe postfix when default behaviour… (#9002)
* fix: get_custom_database docs describe postfix when default behaviour is to take the custom database value directly, if provided

* Changelog entry
2023-11-07 12:34:38 -07:00
Chenyu Li
01d481bc8d Fix lock for git subduer (#9019)
* wip

* add tests

* changelog

* nits

* pr feedback

* nits
2023-11-07 13:58:56 -05:00
Gerda Shank
46b9a1d621 Support new agate Integer data_type in adapter code (#9004)
Add test for empty seed file with only headers
2023-11-07 13:43:25 -05:00
Mike Alfare
839c720e91 Use MANIFEST.in to identify package data, allows recursive include (#9021)
* changelog

* use MANIFEST.in to identify package data
2023-11-07 13:21:15 -05:00
Quigley Malcolm
d88c6987a2 Remove documentable as it isn't used anywhere (#9008) 2023-11-07 09:12:36 -08:00
Michelle Ark
4ee950427a Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-07 11:38:24 -05:00
Jeremy Cohen
6c1822f186 Fix back compat for run_results pre-v5 (#9009)
* Fix back compat for run_results pre-v5

* Add type annotations

* Add functional testing

* Add inline annotations

* Add changelog entry.

* Consolidate upgrade_schema_version + upgrade_run_results_json

* Restore accidentally reverted test cases

* Pre-commit fixups

---------

Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
2023-11-06 16:55:36 -05:00
Anders
c7c3ac872c Update capability.py (#8842) 2023-11-06 10:42:52 -05:00
Michelle Ark
c4ff280436 remove dbt.flags.LOG_CACHE_EVENTS usage in dbt/adapters (#8933) 2023-11-01 18:44:59 -04:00
Peter Webb
7fddd6e448 Make relation filtering None-tolerant for maximal flexibility across adapters. (#8975) 2023-11-01 16:25:27 -04:00
Michelle Ark
1260782bd2 remove dbt.flags.MP_CONTEXT usage in dbt/adapters (#8931) 2023-11-01 16:13:39 -04:00
Michelle Ark
333120b111 Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-01 12:00:17 -04:00
Mike Alfare
bb21403c9e ADAP-974: Fix issue where materialized views were not showing up in catalog queries (#8945)
* changelog
* write test case demonstrating the issue
* update catalog query to reflect materialized views
2023-10-31 23:56:59 -04:00
Quigley Malcolm
ac972948b8 DSI 0.4.0 and Saved Query Exports (#8950) 2023-10-31 18:34:41 -07:00
Chenyu Li
211392c4a4 add a no-op runner for saved_query (#8937) 2023-10-31 16:43:33 -07:00
Peter Webb
7317de23a3 Fix cased comparison in catalog-retrieval function (#8940)
* Fix cased comparison in catalog-retrieval function.

* Fix cased comparison in catalog-retrieval function.
2023-10-30 11:23:41 -04:00
Michelle Ark
af916666a2 move agate_helper into common (#8911)
* move agate_helper into common

* add changie

---------

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
2023-10-26 16:22:04 -04:00
Michelle Ark
7de8930d1d move agate_helper unit tests under tests/unit/common 2023-10-26 16:19:13 -04:00
Michelle Ark
200bcdcd9f Merge branch 'move-agate-client-to-common' into feature/decouple-adapters-from-core 2023-10-26 16:18:30 -04:00
Michelle Ark
b9a603e3aa remove usage of dbt.config.PartialProject from dbt/adapters (#8909)
* remove usage of dbt.config.PartialProject from dbt/adapters

* add changie

---------

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
2023-10-26 16:04:39 -04:00
colin-rogers-dbt
1a825484fb Add utils module (#8910)
* moving types_pb2.py to common/events

* split out utils into core/common/adapters

* add changie
2023-10-26 15:59:07 -04:00
Peter Webb
a2a7b7d795 Fix issues around new get_catalog_by_relations macro (#8856)
* Fix issues around new get_catalog_by_relations macro

* Add changelog entry

* Fix unit test.

* Additional unit testing
2023-10-26 15:23:04 -04:00
Kshitij Aranke
4122f6c308 Fix #8836: Add version to fqn when version==0 (#8915) 2023-10-26 11:31:24 -05:00
Doug Beatty
6aeebc4c76 Handle unknown type_code for model contracts (#8887)
* Handle unknown `type_code` for model contracts

* Changelog entry

* Fix changelog entry

* Functional test for a `type_code` that is not recognized by psycopg2

* Functional tests for data type mismatches
2023-10-25 15:44:02 -06:00
Michelle Ark
f44d704801 move agate_helper into common 2023-10-25 17:05:08 -04:00
Michelle Ark
dbd02e54c2 move types_pb2.py from events to common/events 2023-10-25 15:46:41 -04:00
Michelle Ark
a89642a6f9 fix import 2023-10-25 15:04:17 -04:00
colin-rogers-dbt
c141148616 Move events to common (#8676)
* Move events to common

* More Type Annotations (#8536)

* Extend use of type annotations in the events module.

* Add return type of None to more __init__ definitions.

* Still more type annotations adding -> None to __init__

* Tweak per review

* Allow adapters to include python package logging in dbt logs (#8643)

* add set_package_log_level functionality

* set package handler

* set package handler

* add logging about stting up logging

* test event log handler

* add event log handler

* add event log level

* rename package and add unit tests

* revert logfile config change

* cleanup and add code comments

* add changie

* swap function for dict

* add additional unit tests

* fix unit test

* update README and protos

* fix formatting

* update precommit

---------

Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
2023-10-25 14:57:02 -04:00
Michelle Ark
469a9aca06 remove dbt.contracts.connection imports from adapter module 2023-10-25 14:56:55 -04:00
Emily Rockman
98310b6612 Contract enforcement on temporary tables (#8889)
* add test

* fix test

* first pass with constraint error

* add back column checks for temp tables

* changelog

* Update .changes/unreleased/Fixes-20231024-145504.yaml
2023-10-25 11:10:36 -05:00
Emily Rockman
ef9d6a870f add 1.7 changelog (#8886) 2023-10-24 11:51:28 -05:00
Chenyu Li
35f46dac8c Fix partial parsing issue not working for changing semantic model name (#8865)
* fix

* test

* changelog
2023-10-23 09:55:58 -07:00
FishtownBuildBot
efa6339e18 Cleanup main after cutting new 1.7.latest branch (#8840) 2023-10-12 16:46:37 +01:00
519 changed files with 17071 additions and 31370 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.7.18
current_version = 1.8.0a1
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

View File

@@ -3,6 +3,7 @@
For information on prior major and minor releases, see their changelogs:
* [1.7](https://github.com/dbt-labs/dbt-core/blob/1.7.latest/CHANGELOG.md)
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)

View File

@@ -1,157 +0,0 @@
## dbt-core 1.7.0 - November 02, 2023
### Breaking Changes
- Removed the FirstRunResultError and AfterFirstRunResultError event types, using the existing RunResultError in their place. ([#7963](https://github.com/dbt-labs/dbt-core/issues/7963))
### Features
- add log file of installed packages via dbt deps ([#6643](https://github.com/dbt-labs/dbt-core/issues/6643))
- Enable re-population of metadata vars post-environment change during programmatic invocation ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- Added support to configure a delimiter for a seed file, defaults to comma ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- Allow specification of `create_metric: true` on measures ([#8125](https://github.com/dbt-labs/dbt-core/issues/8125))
- Add node attributes related to compilation to run_results.json ([#7519](https://github.com/dbt-labs/dbt-core/issues/7519))
- Add --no-inject-ephemeral-ctes flag for `compile` command, for usage by linting. ([#8480](https://github.com/dbt-labs/dbt-core/issues/8480))
- Support configuration of semantic models with the addition of enable/disable and group enablement. ([#7968](https://github.com/dbt-labs/dbt-core/issues/7968))
- Accept a `dbt-cloud` config in dbt_project.yml ([#8438](https://github.com/dbt-labs/dbt-core/issues/8438))
- Support atomic replace in the global replace macro ([#8539](https://github.com/dbt-labs/dbt-core/issues/8539))
- Use translate_type on data_type in model.columns in templates by default, remove no op `TYPE_LABELS` ([#8007](https://github.com/dbt-labs/dbt-core/issues/8007))
- Add an option to generate static documentation ([#8614](https://github.com/dbt-labs/dbt-core/issues/8614))
- Allow setting "access" as a config in addition to as a property ([#8383](https://github.com/dbt-labs/dbt-core/issues/8383))
- Loosen typing requirement on renameable/replaceable relations to Iterable to allow adapters more flexibility in registering relation types, include docstrings as suggestions ([#8647](https://github.com/dbt-labs/dbt-core/issues/8647))
- Add support for optional label in semantic_models, measures, dimensions and entities. ([#8595](https://github.com/dbt-labs/dbt-core/issues/8595), [#8755](https://github.com/dbt-labs/dbt-core/issues/8755))
- Allow adapters to include package logs in dbt standard logging ([#7859](https://github.com/dbt-labs/dbt-core/issues/7859))
- Support storing test failures as views ([#6914](https://github.com/dbt-labs/dbt-core/issues/6914))
- resolve packages with same git repo and unique subdirectory ([#5374](https://github.com/dbt-labs/dbt-core/issues/5374))
- Add new ResourceReport event to record memory/cpu/io metrics ([#8342](https://github.com/dbt-labs/dbt-core/issues/8342))
- Adding `date_spine` macro (and supporting macros) from dbt-utils to dbt-core ([#8172](https://github.com/dbt-labs/dbt-core/issues/8172))
- Support `fill_nulls_with` and `join_to_timespine` for metric nodes ([#8593](https://github.com/dbt-labs/dbt-core/issues/8593), [#8755](https://github.com/dbt-labs/dbt-core/issues/8755))
- Raise a warning when a contracted model has a numeric field without scale defined ([#8183](https://github.com/dbt-labs/dbt-core/issues/8183))
- Added support for retrieving partial catalog information from a schema ([#8521](https://github.com/dbt-labs/dbt-core/issues/8521))
- Add meta attribute to SemanticModels config ([#8511](https://github.com/dbt-labs/dbt-core/issues/8511))
- Selectors with docs generate limits catalog generation ([#6014](https://github.com/dbt-labs/dbt-core/issues/6014))
- Allow freshness to be determined via DBMS metadata for supported adapters ([#8704](https://github.com/dbt-labs/dbt-core/issues/8704))
- Add support semantic layer SavedQuery node type ([#8594](https://github.com/dbt-labs/dbt-core/issues/8594))
- Add exports to SavedQuery spec ([#8892](https://github.com/dbt-labs/dbt-core/issues/8892))
### Fixes
- Copy dir during `dbt deps` if symlink fails ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- If --profile specified with dbt-init, create the project with the specified profile ([#6154](https://github.com/dbt-labs/dbt-core/issues/6154))
- Fixed double-underline ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Enable converting deprecation warnings to errors ([#8130](https://github.com/dbt-labs/dbt-core/issues/8130))
- Add status to Parse Inline Error ([#8173](https://github.com/dbt-labs/dbt-core/issues/8173))
- Ensure `warn_error_options` get serialized in `invocation_args_dict` ([#7694](https://github.com/dbt-labs/dbt-core/issues/7694))
- Stop detecting materialization macros based on macro name ([#6231](https://github.com/dbt-labs/dbt-core/issues/6231))
- Update `dbt deps` download retry logic to handle `EOFError` exceptions ([#6653](https://github.com/dbt-labs/dbt-core/issues/6653))
- Improve handling of CTE injection with ephemeral models ([#8213](https://github.com/dbt-labs/dbt-core/issues/8213))
- Fix unbound local variable error in `checked_agg_time_dimension_for_measure` ([#8230](https://github.com/dbt-labs/dbt-core/issues/8230))
- Ensure runtime errors are raised for graph runnable tasks (compile, show, run, etc) ([#8166](https://github.com/dbt-labs/dbt-core/issues/8166))
- Fix retry not working with log-file-max-bytes ([#8297](https://github.com/dbt-labs/dbt-core/issues/8297))
- Add explicit support for integers for the show command ([#8153](https://github.com/dbt-labs/dbt-core/issues/8153))
- Detect changes to model access, version, or latest_version in state:modified ([#8189](https://github.com/dbt-labs/dbt-core/issues/8189))
- Add connection status into list of statuses for dbt debug ([#8350](https://github.com/dbt-labs/dbt-core/issues/8350))
- fix fqn-selection for external versioned models ([#8374](https://github.com/dbt-labs/dbt-core/issues/8374))
- Fix: DbtInternalError after model that previously ref'd external model is deleted ([#8375](https://github.com/dbt-labs/dbt-core/issues/8375))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
- Remedy performance regression by only writing run_results.json once. ([#8360](https://github.com/dbt-labs/dbt-core/issues/8360))
- Add support for swapping materialized views with tables/views and vice versa ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Turn breaking changes to contracted models into warnings for unversioned models ([#8384](https://github.com/dbt-labs/dbt-core/issues/8384), [#8282](https://github.com/dbt-labs/dbt-core/issues/8282))
- Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension` ([#8453](https://github.com/dbt-labs/dbt-core/issues/8453))
- fix ambiguous reference error for tests and versions when model name is duplicated across packages ([#8327](https://github.com/dbt-labs/dbt-core/issues/8327), [#8493](https://github.com/dbt-labs/dbt-core/issues/8493))
- Fix "Internal Error: Expected node <unique-id> not found in manifest" when depends_on set on ModelNodeArgs ([#8506](https://github.com/dbt-labs/dbt-core/issues/8506))
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
- Parse the correct schema version from manifest ([#8544](https://github.com/dbt-labs/dbt-core/issues/8544))
- make version comparison insensitive to order ([#8571](https://github.com/dbt-labs/dbt-core/issues/8571))
- Update metric helper functions to work with new semantic layer metrics ([#8134](https://github.com/dbt-labs/dbt-core/issues/8134))
- Disallow cleaning paths outside current working directory ([#8318](https://github.com/dbt-labs/dbt-core/issues/8318))
- Warn when --state == --target ([#8160](https://github.com/dbt-labs/dbt-core/issues/8160))
- update dbt show to include limit in DWH query ([#8496,](https://github.com/dbt-labs/dbt-core/issues/8496,), [#8417](https://github.com/dbt-labs/dbt-core/issues/8417))
- Support quoted parameter list for MultiOption CLI options. ([#8598](https://github.com/dbt-labs/dbt-core/issues/8598))
- Support global flags passed in after subcommands ([#6497](https://github.com/dbt-labs/dbt-core/issues/6497))
- Lower bound of `8.0.2` for `click` ([#8683](https://github.com/dbt-labs/dbt-core/issues/8683))
- Fixes test type edges filter ([#8692](https://github.com/dbt-labs/dbt-core/issues/8692))
- semantic models in graph selection ([#8589](https://github.com/dbt-labs/dbt-core/issues/8589))
- Support doc blocks in nested semantic model YAML ([#8509](https://github.com/dbt-labs/dbt-core/issues/8509))
- avoid double-rendering sql_header in dbt show ([#8739](https://github.com/dbt-labs/dbt-core/issues/8739))
- Fix tag selection for projects with semantic models ([#8749](https://github.com/dbt-labs/dbt-core/issues/8749))
- Foreign key constraint on incremental model results in Database Error ([#8022](https://github.com/dbt-labs/dbt-core/issues/8022))
- Support docs blocks on versioned model column descriptions ([#8540](https://github.com/dbt-labs/dbt-core/issues/8540))
- Enable seeds to be handled from stored manifest data ([#6875](https://github.com/dbt-labs/dbt-core/issues/6875))
- Override path-like args in dbt retry ([#8682](https://github.com/dbt-labs/dbt-core/issues/8682))
- Group updates on unmodified nodes are handled gracefully for state:modified ([#8371](https://github.com/dbt-labs/dbt-core/issues/8371))
- Partial parsing fix for adding groups and updating models at the same time ([#8697](https://github.com/dbt-labs/dbt-core/issues/8697))
- Fix partial parsing not working for semantic model change ([#8859](https://github.com/dbt-labs/dbt-core/issues/8859))
- Rework get_catalog implementation to retain previous adapter interface semantics ([#8846](https://github.com/dbt-labs/dbt-core/issues/8846))
- Add back contract enforcement for temporary tables on postgres ([#8857](https://github.com/dbt-labs/dbt-core/issues/8857))
- Add version to fqn when version==0 ([#8836](https://github.com/dbt-labs/dbt-core/issues/8836))
- Fix cased comparison in catalog-retrieval function. ([#8939](https://github.com/dbt-labs/dbt-core/issues/8939))
- Catalog queries now assign the correct type to materialized views ([#8864](https://github.com/dbt-labs/dbt-core/issues/8864))
- Make relation filtering None-tolerant for maximal flexibility across adapters. ([#8974](https://github.com/dbt-labs/dbt-core/issues/8974))
### Docs
- Corrected spelling of "Partiton" ([dbt-docs/#8100](https://github.com/dbt-labs/dbt-docs/issues/8100))
- Remove static SQL codeblock for metrics ([dbt-docs/#436](https://github.com/dbt-labs/dbt-docs/issues/436))
- fixed comment util.py ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Fix newline escapes and improve formatting in docker README ([dbt-docs/#8211](https://github.com/dbt-labs/dbt-docs/issues/8211))
- Display contract and column constraints on the model page ([dbt-docs/#433](https://github.com/dbt-labs/dbt-docs/issues/433))
- Display semantic model details in docs ([dbt-docs/#431](https://github.com/dbt-labs/dbt-docs/issues/431))
### Under the Hood
- Switch from hologram to mashumaro jsonschema ([#8426](https://github.com/dbt-labs/dbt-core/issues/8426))
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))
- format exception from dbtPlugin.initialize ([#8152](https://github.com/dbt-labs/dbt-core/issues/8152))
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
- Ref expressions with version can now be processed by the latest version of the high-performance dbt-extractor library. ([#7688](https://github.com/dbt-labs/dbt-core/issues/7688))
- Bump manifest schema version to v11, freeze manifest v10 ([#8333](https://github.com/dbt-labs/dbt-core/issues/8333))
- add tracking for plugin.get_nodes calls ([#8344](https://github.com/dbt-labs/dbt-core/issues/8344))
- add internal flag: --no-partial-parse-file-diff to inform whether to compute a file diff during partial parsing ([#8363](https://github.com/dbt-labs/dbt-core/issues/8363))
- Add return values to a number of functions for mypy ([#8389](https://github.com/dbt-labs/dbt-core/issues/8389))
- Fix mypy warnings for ManifestLoader.load() ([#8401](https://github.com/dbt-labs/dbt-core/issues/8401))
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))
- Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>, relation agnostic in /macros/relations ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Update typing to meet mypy standards ([#8396](https://github.com/dbt-labs/dbt-core/issues/8396))
- Mypy errors - adapters/factory.py ([#8387](https://github.com/dbt-labs/dbt-core/issues/8387))
- Added more type annotations. ([#8537](https://github.com/dbt-labs/dbt-core/issues/8537))
- Audit potential circular dependencies ([#8349](https://github.com/dbt-labs/dbt-core/issues/8349))
- Add functional test for advanced ref override ([#8566](https://github.com/dbt-labs/dbt-core/issues/8566))
- Add typing to __init__ in base.py ([#8398](https://github.com/dbt-labs/dbt-core/issues/8398))
- Fix untyped functions in task/runnable.py (mypy warning) ([#8402](https://github.com/dbt-labs/dbt-core/issues/8402))
- add a test for ephemeral cte injection ([#8225](https://github.com/dbt-labs/dbt-core/issues/8225))
- Fix test_numeric_values to look for more specific strings ([#8470](https://github.com/dbt-labs/dbt-core/issues/8470))
- Pin types-requests<2.31.0 in `dev-requirements.txt` ([#8789](https://github.com/dbt-labs/dbt-core/issues/8789))
- Add warning_tag to UnversionedBreakingChange ([#8827](https://github.com/dbt-labs/dbt-core/issues/8827))
- Update v10 manifest schema to match 1.6 for testing schema compatibility ([#8835](https://github.com/dbt-labs/dbt-core/issues/8835))
- Add a no-op runner for Saved Qeury ([#8893](https://github.com/dbt-labs/dbt-core/issues/8893))
### Dependencies
- Bump mypy from 1.3.0 to 1.4.0 ([#7912](https://github.com/dbt-labs/dbt-core/pull/7912))
- Bump mypy from 1.4.0 to 1.4.1 ([#8219](https://github.com/dbt-labs/dbt-core/pull/8219))
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))
- Support dbt-semantic-interfaces 0.2.0 ([#8250](https://github.com/dbt-labs/dbt-core/pull/8250))
- Bump docker/build-push-action from 4 to 5 ([#8783](https://github.com/dbt-labs/dbt-core/pull/8783))
- Upgrade dbt-semantic-interfaces dep to 0.3.0 ([#8819](https://github.com/dbt-labs/dbt-core/pull/8819))
- Begin using DSI 0.4.x ([#8892](https://github.com/dbt-labs/dbt-core/pull/8892))
### Contributors
- [@anjutiwari](https://github.com/anjutiwari) ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- [@benmosher](https://github.com/benmosher) ([#8480](https://github.com/dbt-labs/dbt-core/issues/8480))
- [@d-kaneshiro](https://github.com/d-kaneshiro) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#8153](https://github.com/dbt-labs/dbt-core/issues/8153), [#8589](https://github.com/dbt-labs/dbt-core/issues/8589))
- [@dylan-murray](https://github.com/dylan-murray) ([#8683](https://github.com/dbt-labs/dbt-core/issues/8683))
- [@ezraerb](https://github.com/ezraerb) ([#6154](https://github.com/dbt-labs/dbt-core/issues/6154))
- [@gem7318](https://github.com/gem7318) ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- [@jamezrin](https://github.com/jamezrin) ([#8211](https://github.com/dbt-labs/dbt-core/issues/8211))
- [@jusbaldw](https://github.com/jusbaldw) ([#6643](https://github.com/dbt-labs/dbt-core/issues/6643))
- [@lllong33](https://github.com/lllong33) ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- [@marcodamore](https://github.com/marcodamore) ([#436](https://github.com/dbt-labs/dbt-core/issues/436))
- [@mescanne](https://github.com/mescanne) ([#8614](https://github.com/dbt-labs/dbt-core/issues/8614))
- [@pgoslatara](https://github.com/pgoslatara) ([#8100](https://github.com/dbt-labs/dbt-core/issues/8100))
- [@philippeboyd](https://github.com/philippeboyd) ([#5374](https://github.com/dbt-labs/dbt-core/issues/5374))
- [@ramonvermeulen](https://github.com/ramonvermeulen) ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- [@renanleme](https://github.com/renanleme) ([#8692](https://github.com/dbt-labs/dbt-core/issues/8692))

View File

@@ -1,8 +0,0 @@
## dbt-core 1.7.1 - November 07, 2023
### Fixes
- Fix compilation exception running empty seed file and support new Integer agate data_type ([#8895](https://github.com/dbt-labs/dbt-core/issues/8895))
- Update run_results.json from previous versions of dbt to support deferral and rerun from failure ([#9010](https://github.com/dbt-labs/dbt-core/issues/9010))
- Use MANIFEST.in to recursively include all jinja templates; fixes issue where some templates were not included in the distribution ([#9016](https://github.com/dbt-labs/dbt-core/issues/9016))
- Fix git repository with subdirectory for Deps ([#9000](https://github.com/dbt-labs/dbt-core/issues/9000))

View File

@@ -1,11 +0,0 @@
## dbt-core 1.7.10 - March 14, 2024
### Fixes
- Do not add duplicate input_measures ([#9360](https://github.com/dbt-labs/dbt-core/issues/9360))
- Fix partial parsing `KeyError` on deleted schema files ([#8860](https://github.com/dbt-labs/dbt-core/issues/8860))
- Support saved queries in `dbt list` ([#9532](https://github.com/dbt-labs/dbt-core/issues/9532))
### Dependencies
- Restrict protobuf to 4.* versions ([#9566](https://github.com/dbt-labs/dbt-core/pull/9566))

View File

@@ -1,6 +0,0 @@
## dbt-core 1.7.11 - March 28, 2024
### Fixes
- Tighten exception handling to avoid worker thread hangs. ([#9583](https://github.com/dbt-labs/dbt-core/issues/9583))
- Add field wrapper to BaseRelation members that were missing it. ([#9681](https://github.com/dbt-labs/dbt-core/issues/9681))

View File

@@ -1,6 +0,0 @@
## dbt-core 1.7.12 - April 16, 2024
### Fixes
- Fix assorted source freshness edgecases so check is run or actionable information is given ([#9078](https://github.com/dbt-labs/dbt-core/issues/9078))
- Exclude password-like fields for considering reparse ([#9795](https://github.com/dbt-labs/dbt-core/issues/9795))

View File

@@ -1,8 +0,0 @@
## dbt-core 1.7.13 - April 18, 2024
### Security
- Bump sqlparse to >=0.5.0, <0.6.0 to address GHSA-2m57-hf25-phgg ([#9951](https://github.com/dbt-labs/dbt-core/pull/9951))
### Contributors
- [@emmoop](https://github.com/emmoop) ([#9951](https://github.com/dbt-labs/dbt-core/pull/9951))

View File

@@ -1,16 +0,0 @@
## dbt-core 1.7.14 - May 02, 2024
### Features
- Move flags from UserConfig in profiles.yml to flags in dbt_project.yml ([#9183](https://github.com/dbt-labs/dbt-core/issues/9183))
- Add require_explicit_package_overrides_for_builtin_materializations to dbt_project.yml flags, which can be used to opt-out of overriding built-in materializations from packages ([#10007](https://github.com/dbt-labs/dbt-core/issues/10007))
### Fixes
- remove materialized views from renambeable relation and remove a quote ([#127](https://github.com/dbt-labs/dbt-core/issues/127))
- Replace usage of `Set` with `List` to fix issue with index updates intermittently happening out of order ([#72](https://github.com/dbt-labs/dbt-core/issues/72))
### Under the Hood
- Raise deprecation warning if installed package overrides built-in materialization ([#9971](https://github.com/dbt-labs/dbt-core/issues/9971))
- Remove the final underscore from secret environment variable constants. ([#10052](https://github.com/dbt-labs/dbt-core/issues/10052))

View File

@@ -1,9 +0,0 @@
## dbt-core 1.7.15 - May 22, 2024
### Fixes
- Fix the semicolon semantics for indexes while respecting other bug fix ([#85](https://github.com/dbt-labs/dbt-core/issues/85))
### Security
- Explicitly bind to localhost in docs serve ([#10209](https://github.com/dbt-labs/dbt-core/issues/10209))

View File

@@ -1,5 +0,0 @@
## dbt-core 1.7.16 - June 05, 2024
### Features
- Add --host flag to dbt docs serve, defaulting to '127.0.0.1' ([#10229](https://github.com/dbt-labs/dbt-core/issues/10229))

View File

@@ -1,5 +0,0 @@
## dbt-core 1.7.17 - June 20, 2024
### Docs
- Fix npm security vulnerabilities as of June 2024 ([dbt-docs/#513](https://github.com/dbt-labs/dbt-docs/issues/513))

View File

@@ -1,5 +0,0 @@
## dbt-core 1.7.18 - August 07, 2024
### Fixes
- respect --quiet and --warn-error-options for flag deprecations ([#10105](https://github.com/dbt-labs/dbt-core/issues/10105))

View File

@@ -1,16 +0,0 @@
## dbt-core 1.7.2 - November 16, 2023
### Features
- Support setting export configs hierarchically via saved query and project configs ([#8956](https://github.com/dbt-labs/dbt-core/issues/8956))
### Fixes
- Fix formatting of tarball information in packages-lock.yml ([#9062](https://github.com/dbt-labs/dbt-core/issues/9062))
### Under the Hood
- Treat SystemExit as an interrupt if raised during node execution. ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))
### Contributors
- [@benmosher](https://github.com/benmosher) ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))

View File

@@ -1,7 +0,0 @@
## dbt-core 1.7.3 - November 29, 2023
### Fixes
- deps: Lock git packages to commit SHA during resolution ([#9050](https://github.com/dbt-labs/dbt-core/issues/9050))
- deps: Use PackageRenderer to read package-lock.json ([#9127](https://github.com/dbt-labs/dbt-core/issues/9127))
- Get sources working again in dbt docs generate ([#9119](https://github.com/dbt-labs/dbt-core/issues/9119))

View File

@@ -1,12 +0,0 @@
## dbt-core 1.7.4 - December 14, 2023
### Features
- Adds support for parsing conversion metric related properties for the semantic layer. ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))
### Fixes
- Ensure we produce valid jsonschema schemas for manifest, catalog, run-results, and sources ([#8991](https://github.com/dbt-labs/dbt-core/issues/8991))
### Contributors
- [@WilliamDee](https://github.com/WilliamDee) ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))

View File

@@ -1,8 +0,0 @@
## dbt-core 1.7.5 - January 18, 2024
### Fixes
- Preserve the value of vars and the --full-refresh flags when using retry. ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))
### Contributors
- [@peterallenwebb,](https://github.com/peterallenwebb,) ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))

View File

@@ -1,6 +0,0 @@
## dbt-core 1.7.6 - January 25, 2024
### Fixes
- Handle unknown `type_code` for model contracts ([#8877](https://github.com/dbt-labs/dbt-core/issues/8877), [#8353](https://github.com/dbt-labs/dbt-core/issues/8353))
- Fix retry command run from CLI ([#9444](https://github.com/dbt-labs/dbt-core/issues/9444))

View File

@@ -1,6 +0,0 @@
## dbt-core 1.7.7 - February 01, 2024
### Fixes
- Fix seed and source selection in `dbt docs generate` ([#9161](https://github.com/dbt-labs/dbt-core/issues/9161))
- Add TestGenerateCatalogWithExternalNodes, include empty nodes in node selection during docs generate ([#9456](https://github.com/dbt-labs/dbt-core/issues/9456))

View File

@@ -1,6 +0,0 @@
## dbt-core 1.7.8 - February 14, 2024
### Fixes
- When patching versioned models, set constraints after config ([#9364](https://github.com/dbt-labs/dbt-core/issues/9364))
- Store node_info in node associated logging events ([#9557](https://github.com/dbt-labs/dbt-core/issues/9557))

View File

@@ -1,18 +0,0 @@
## dbt-core 1.7.9 - February 28, 2024
### Fixes
- Fix node_info contextvar handling so incorrect node_info doesn't persist ([#8866](https://github.com/dbt-labs/dbt-core/issues/8866))
- Add target-path to retry ([#8948](https://github.com/dbt-labs/dbt-core/issues/8948))
### Under the Hood
- Make dbt-core compatible with Python 3.12 ([#9007](https://github.com/dbt-labs/dbt-core/issues/9007))
- Restrict protobuf to major version 4. ([#9566](https://github.com/dbt-labs/dbt-core/issues/9566))
### Security
- Update Jinja2 to >= 3.1.3 to address CVE-2024-22195 ([#CVE-2024-22195](https://github.com/dbt-labs/dbt-core/pull/CVE-2024-22195))
### Contributors
- [@l1xnan](https://github.com/l1xnan) ([#9007](https://github.com/dbt-labs/dbt-core/issues/9007))

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Remove adapter.get_compiler interface
time: 2023-11-27T11:47:57.443202-05:00
custom:
Author: michelleark
Issue: "9148"

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Move AdapterLogger to adapters folder
time: 2023-11-28T13:43:56.853925-08:00
custom:
Author: colin-rogers-dbt
Issue: "9151"

View File

@@ -0,0 +1,7 @@
kind: Breaking Changes
body: move event manager setup back to core, remove ref to global EVENT_MANAGER and
clean up event manager functions
time: 2023-11-30T13:53:48.645192-08:00
custom:
Author: colin-rogers-dbt
Issue: "9150"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Begin using DSI 0.4.x
time: 2023-10-31T13:19:54.750009-07:00
custom:
Author: QMalcolm peterallenwebb
PR: "8892"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Update typing-extensions version to >=4.4
time: 2023-11-06T13:00:51.062386-08:00
custom:
Author: tlento
PR: "9012"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: fix get_custom_database docstring
time: 2023-11-06T12:31:57.525711Z
custom:
Author: LeoTheGriff
Issue: "9003"

View File

@@ -0,0 +1,6 @@
kind: Features
body: 'Allow adapters to include package logs in dbt standard logging '
time: 2023-09-15T12:37:33.862862-07:00
custom:
Author: colin-rogers-dbt
Issue: "7859"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add drop_schema_named macro
time: 2023-10-17T14:36:20.612289-07:00
custom:
Author: colin-rogers-dbt
Issue: "8025"

View File

@@ -0,0 +1,6 @@
kind: Features
body: migrate utils to common and adapters folders
time: 2023-10-26T11:08:21.458709-07:00
custom:
Author: colin-rogers-dbt
Issue: "8924"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Move Agate helper client into common
time: 2023-10-26T12:35:56.538587-07:00
custom:
Author: MichelleArk
Issue: "8926"

View File

@@ -0,0 +1,6 @@
kind: Features
body: remove usage of dbt.config.PartialProject from dbt/adapters
time: 2023-10-26T12:39:13.904116-07:00
custom:
Author: MichelleArk
Issue: "8928"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add exports to SavedQuery spec
time: 2023-10-31T13:20:22.448158-07:00
custom:
Author: QMalcolm peterallenwebb
Issue: "8892"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Remove legacy logger
time: 2023-11-07T13:56:35.186648-08:00
custom:
Author: colin-rogers-dbt
Issue: "8027"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support setting export configs hierarchically via saved query and project configs
time: 2023-11-10T15:42:55.042317-08:00
custom:
Author: QMalcolm
Issue: "8956"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: For packages installed with tarball method, fetch metadata to resolve nested dependencies
time: 2023-10-13T13:09:43.188308-04:00
custom:
Author: adamlopez
Issue: "8621"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix partial parsing not working for semantic model change
time: 2023-10-16T16:39:53.05058-07:00
custom:
Author: ChenyuLInx
Issue: "8859"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Handle unknown `type_code` for model contracts
time: 2023-10-24T11:01:51.980781-06:00
custom:
Author: dbeatty10
Issue: 8877 8353

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add back contract enforcement for temporary tables on postgres
time: 2023-10-24T14:55:04.051683-05:00
custom:
Author: emmyoop
Issue: "8857"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Rework get_catalog implementation to retain previous adapter interface semantics
time: 2023-10-24T15:54:00.628086-04:00
custom:
Author: peterallenwebb
Issue: "8846"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add version to fqn when version==0
time: 2023-10-26T00:25:36.259356-05:00
custom:
Author: aranke
Issue: "8836"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix cased comparison in catalog-retrieval function.
time: 2023-10-30T09:37:34.258612-04:00
custom:
Author: peterallenwebb
Issue: "8939"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Catalog queries now assign the correct type to materialized views
time: 2023-10-31T00:53:45.486203-04:00
custom:
Author: mikealfare
Issue: "8864"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix compilation exception running empty seed file and support new Integer agate data_type
time: 2023-10-31T14:48:37.774871-04:00
custom:
Author: gshank
Issue: "8895"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Make relation filtering None-tolerant for maximal flexibility across adapters.
time: 2023-11-01T15:58:24.552054-04:00
custom:
Author: peterallenwebb
Issue: "8974"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Update run_results.json from previous versions of dbt to support deferral and
rerun from failure
time: 2023-11-06T15:59:33.677915-05:00
custom:
Author: jtcohen6 peterallenwebb
Issue: "9010"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix git repository with subdirectory for Deps
time: 2023-11-07T09:23:58.214271-08:00
custom:
Author: ChenyuLInx
Issue: "9000"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Use MANIFEST.in to recursively include all jinja templates; fixes issue where
some templates were not included in the distribution
time: 2023-11-07T09:41:30.121733-05:00
custom:
Author: mikealfare
Issue: "9016"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix formatting of tarball information in packages-lock.yml
time: 2023-11-13T11:49:56.437007-08:00
custom:
Author: ChenyuLInx QMalcolm
Issue: "9062"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'deps: Lock git packages to commit SHA during resolution'
time: 2023-11-27T15:43:10.122069+01:00
custom:
Author: jtcohen6
Issue: "9050"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'deps: Use PackageRenderer to read package-lock.json'
time: 2023-11-27T15:43:47.842423+01:00
custom:
Author: jtcohen6
Issue: "9127"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Get sources working again in dbt docs generate'
time: 2023-11-28T15:52:25.738256Z
custom:
Author: aranke
Issue: "9119"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Added more type annotations.
time: 2023-08-31T16:44:35.737954-04:00
custom:
Author: peterallenwebb
Issue: "8537"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.include.global_project in dbt/adapters
time: 2023-10-26T18:49:53.36449-04:00
custom:
Author: michelleark
Issue: "8925"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add a no-op runner for Saved Qeury
time: 2023-10-27T14:00:48.4755-07:00
custom:
Author: ChenyuLInx
Issue: "8893"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: remove dbt.flags.MP_CONTEXT usage in dbt/adapters
time: 2023-11-01T10:27:58.790153-04:00
custom:
Author: michelleark
Issue: "8967"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: 'Remove usage of dbt.flags.LOG_CACHE_EVENTS in dbt/adapters'
time: 2023-11-01T17:31:24.974093-04:00
custom:
Author: michelleark
Issue: "8969"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move CatalogRelationTypes test case to the shared test suite to be reused by
adapter maintainers
time: 2023-11-03T19:52:22.694394-04:00
custom:
Author: mikealfare
Issue: "8952"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Treat SystemExit as an interrupt if raised during node execution.
time: 2023-11-06T08:04:22.022179-05:00
custom:
Author: benmosher
Issue: n/a

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Removing unused 'documentable'
time: 2023-11-06T10:57:30.694056-08:00
custom:
Author: QMalcolm
Issue: "8871"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove use of dbt/core exceptions in dbt/adapter
time: 2023-11-07T13:57:28.683727-08:00
custom:
Author: colin-rogers-dbt MichelleArk
Issue: "8920"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Cache dbt plugin modules to improve integration test performance
time: 2023-11-07T19:15:46.170151-05:00
custom:
Author: peterallenwebb
Issue: "9029"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Fix test_current_timestamp_matches_utc test; allow for MacOS runner system clock
variance
time: 2023-11-11T17:53:50.098843-05:00
custom:
Author: mikealfare
Issue: "9057"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific
event types and protos
time: 2023-11-16T17:42:51.005023-05:00
custom:
Author: michelleark
Issue: 8927 8918

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Clean up unused adaptor folders
time: 2023-11-20T13:47:35.923794-08:00
custom:
Author: ChenyuLInx
Issue: "9123"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move column constraints into common/contracts, removing another dependency of
adapters on core.
time: 2023-11-20T18:32:14.859503-05:00
custom:
Author: peterallenwebb
Issue: "9024"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move dbt.semver to dbt.common.semver and update references.
time: 2023-11-28T17:07:32.172421-08:00
custom:
Author: versusfacit
Issue: "9039"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move lowercase utils method to common
time: 2023-11-30T13:54:32.561673-08:00
custom:
Author: colin-rogers-dbt
Issue: "9180"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usages of dbt.clients.jinja in dbt/adapters
time: 2023-12-05T09:35:44.845352+09:00
custom:
Author: michelleark
Issue: "9205"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.contracts in dbt/adapters
time: 2023-12-05T12:05:59.936775+09:00
custom:
Author: michelleark
Issue: "9208"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.contracts.graph.nodes.ResultNode in dbt/adapters
time: 2023-12-05T16:58:12.932172+09:00
custom:
Author: michelleark
Issue: "9214"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Introduce RelationConfig Protocol, consolidate Relation.create_from
time: 2023-12-05T17:07:25.33861+09:00
custom:
Author: michelleark
Issue: "9215"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Remove support and testing for Python 3.8, which is now EOL.
time: 2024-10-16T14:40:56.451972-04:00
custom:
Author: gshank peterallenwebb
Issue: "10861"

View File

@@ -31,7 +31,43 @@ kinds:
- {{.Body}} ({{ range $index, $element := $IssueList }}{{if $index}}, {{end}}{{$element}}{{end}})
- label: Under the Hood
- label: Dependencies
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
- label: Security
changeFormat: |-
{{- $PRList := list }}
{{- $changes := splitList " " $.Custom.PR }}
{{- range $pullrequest := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $pullrequest }}
{{- $PRList = append $PRList $changeLink }}
{{- end -}}
- {{.Body}} ({{ range $index, $element := $PRList }}{{if $index}}, {{end}}{{$element}}{{end}})
skipGlobalChoices: true
additionalChoices:
- key: Author
label: GitHub Username(s) (separated by a single space if multiple)
type: string
minLength: 3
- key: PR
label: GitHub Pull Request Number (separated by a single space if multiple)
type: string
minLength: 1
newlines:
afterChangelogHeader: 1
@@ -70,10 +106,18 @@ footerFormat: |
{{- $changeList := splitList " " $change.Custom.Author }}
{{- $IssueList := list }}
{{- $changeLink := $change.Kind }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- if or (eq $change.Kind "Dependencies") (eq $change.Kind "Security") }}
{{- $changes := splitList " " $change.Custom.PR }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/pull/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- else }}
{{- $changes := splitList " " $change.Custom.Issue }}
{{- range $issueNbr := $changes }}
{{- $changeLink := "[#nbr](https://github.com/dbt-labs/dbt-core/issues/nbr)" | replace "nbr" $issueNbr }}
{{- $IssueList = append $IssueList $changeLink }}
{{- end -}}
{{- end }}
{{- /* check if this contributor has other changes associated with them already */}}
{{- if hasKey $contributorDict $author }}

View File

@@ -10,3 +10,5 @@ ignore =
E741
E501 # long line checking is done in black
exclude = test/
per-file-ignores =
*/__init__.py: F401

2
.gitattributes vendored
View File

@@ -1,4 +1,4 @@
core/dbt/include/index.html binary
core/dbt/task/docs/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary
core/dbt/docs/build/html/searchindex.js binary
core/dbt/docs/build/html/index.html binary

View File

@@ -13,7 +13,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler

View File

@@ -58,4 +58,4 @@ jobs:
commit_message: "Add automated changelog yaml from template for bot PR"
changie_kind: ${{ matrix.changie_kind }}
label: ${{ matrix.label }}
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n Issue: ${{ github.event.pull_request.number }}"
custom_changelog_string: "custom:\n Author: ${{ github.event.pull_request.user.login }}\n PR: ${{ github.event.pull_request.number }}"

View File

@@ -47,12 +47,12 @@ jobs:
steps:
- name: Check out the repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: '3.9'
python-version: '3.8'
- name: Install python dependencies
run: |
@@ -74,17 +74,17 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9", "3.10", "3.11", "3.12" ]
python-version: ["3.8", "3.9", "3.10", "3.11"]
env:
TOXENV: "unit"
steps:
- name: Check out the repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
@@ -95,12 +95,8 @@ jobs:
python -m pip install tox
tox --version
- name: Run unit tests
uses: nick-fields/retry@v3
with:
timeout_minutes: 10
max_attempts: 3
command: tox -e unit
- name: Run tox
run: tox
- name: Get current date
if: always()
@@ -111,7 +107,7 @@ jobs:
- name: Upload Unit Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v4
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: unit
@@ -139,7 +135,7 @@ jobs:
- name: generate include
id: generate-include
run: |
INCLUDE=('"python-version":"3.9","os":"windows-latest"' '"python-version":"3.9","os":"macos-12"' )
INCLUDE=('"python-version":"3.8","os":"windows-latest"' '"python-version":"3.8","os":"macos-latest"' )
INCLUDE_GROUPS="["
for include in ${INCLUDE[@]}; do
for group in $(seq 1 ${{ env.PYTHON_INTEGRATION_TEST_WORKERS }}); do
@@ -161,7 +157,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9", "3.10", "3.11", "3.12" ]
python-version: ["3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-20.04]
split-group: ${{ fromJson(needs.integration-metadata.outputs.split-groups) }}
include: ${{ fromJson(needs.integration-metadata.outputs.include) }}
@@ -179,10 +175,10 @@ jobs:
steps:
- name: Check out the repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
@@ -205,12 +201,8 @@ jobs:
python -m pip install tox
tox --version
- name: Run integration tests
uses: nick-fields/retry@v3
with:
timeout_minutes: 30
max_attempts: 3
command: tox -- --ddtrace
- name: Run tests
run: tox -- --ddtrace
env:
PYTEST_ADDOPTS: ${{ format('--splits {0} --group {1}', env.PYTHON_INTEGRATION_TEST_WORKERS, matrix.split-group) }}
@@ -221,15 +213,15 @@ jobs:
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@v3
if: always()
with:
name: logs_${{ matrix.python-version }}_${{ matrix.os }}_${{ matrix.split-group }}_${{ steps.date.outputs.date }}
name: logs_${{ matrix.python-version }}_${{ matrix.os }}_${{ steps.date.outputs.date }}
path: ./logs
- name: Upload Integration Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v4
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: integration
@@ -258,12 +250,12 @@ jobs:
steps:
- name: Check out the repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: '3.9'
python-version: '3.8'
- name: Install python dependencies
run: |
@@ -296,7 +288,7 @@ jobs:
- name: Install source distributions
# ignore dbt-1.0.0, which intentionally raises an error when installed from source
run: |
find ./dist/*.gz -maxdepth 1 -type f | xargs python -m pip install --force-reinstall --find-links=dist/
find ./dist/dbt-[a-z]*.gz -maxdepth 1 -type f | xargs python -m pip install --force-reinstall --find-links=dist/
- name: Check source distributions
run: |

View File

@@ -48,7 +48,7 @@ jobs:
# explicitly checkout the performance runner from main regardless of which
# version we are modeling.
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
ref: main
@@ -87,12 +87,12 @@ jobs:
# explicitly checkout the performance runner from main regardless of which
# version we are modeling.
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
ref: main
# attempts to access a previously cached runner
- uses: actions/cache@v4
- uses: actions/cache@v3
id: cache
with:
path: ${{ env.RUNNER_CACHE_PATH }}
@@ -148,9 +148,9 @@ jobs:
echo "release_branch: ${{ needs.set-variables.outputs.release_branch }}"
- name: Setup Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: "3.9"
python-version: "3.8"
- name: Install dbt
run: pip install dbt-postgres==${{ needs.set-variables.outputs.release_id }}
@@ -160,13 +160,13 @@ jobs:
# explicitly checkout main to get the latest project definitions
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
ref: main
# this was built in the previous job so it will be there.
- name: Fetch Runner
uses: actions/cache@v4
uses: actions/cache@v3
id: cache
with:
path: ${{ env.RUNNER_CACHE_PATH }}
@@ -195,7 +195,7 @@ jobs:
- name: '[DEBUG] ls baseline directory after run'
run: ls -R performance/baselines/
- uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@v3
with:
name: baseline
path: performance/baselines/${{ needs.set-variables.outputs.release_id }}/
@@ -225,7 +225,7 @@ jobs:
echo "release_branch: ${{ needs.set-variables.outputs.release_branch }}"
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
ref: ${{ matrix.base-branch }}
@@ -235,7 +235,7 @@ jobs:
git push origin ${{ matrix.target-branch }}
git branch --set-upstream-to=origin/${{ matrix.target-branch }} ${{ matrix.target-branch }}
- uses: actions/download-artifact@v4
- uses: actions/download-artifact@v3
with:
name: baseline
path: performance/baselines/${{ needs.set-variables.outputs.release_id }}

View File

@@ -39,7 +39,7 @@ jobs:
steps:
- name: "Checkout ${{ github.repository }} Branch ${{ env.RELEASE_BRANCH }}"
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
ref: ${{ env.RELEASE_BRANCH }}

View File

@@ -36,7 +36,7 @@ jobs:
latest: ${{ steps.latest.outputs.latest }}
minor_latest: ${{ steps.latest.outputs.minor_latest }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Split version
id: version
run: |

View File

@@ -37,20 +37,21 @@ jobs:
steps:
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: 3.9
python-version: 3.8
- name: Checkout dbt repo
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: ${{ env.DBT_REPO_DIRECTORY }}
- name: Checkout schemas.getdbt.com repo
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
repository: dbt-labs/schemas.getdbt.com
ref: 'main'
ssh-key: ${{ secrets.SCHEMA_SSH_PRIVATE_KEY }}
path: ${{ env.SCHEMA_REPO_DIRECTORY }}
- name: Generate current schema
@@ -62,14 +63,27 @@ jobs:
pip install -r dev-requirements.txt -r editable-requirements.txt
python scripts/collect-artifact-schema.py --path ${{ env.LATEST_SCHEMA_PATH }}
# Copy generated schema files into the schemas.getdbt.com repo
# Do a git diff to find any changes
# Ignore any date or version changes though
- name: Compare schemas
run: |
cp -r ${{ env.LATEST_SCHEMA_PATH }}/dbt ${{ env.SCHEMA_REPO_DIRECTORY }}
cd ${{ env.SCHEMA_REPO_DIRECTORY }}
git diff -I='*[0-9]{4}-[0-9]{2}-[0-9]{2}' -I='*[0-9]+\.[0-9]+\.[0-9]+' --exit-code > ${{ env.SCHEMA_DIFF_ARTIFACT }}
diff_results=$(git diff -I='*[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])T' \
-I='*[0-9]{1}.[0-9]{2}.[0-9]{1}(rc[0-9]|b[0-9]| )' --compact-summary)
if [[ $(echo diff_results) ]]; then
echo $diff_results
echo "Schema changes detected!"
git diff -I='*[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])T' \
-I='*[0-9]{1}.[0-9]{2}.[0-9]{1}(rc[0-9]|b[0-9]| )' > ${{ env.SCHEMA_DIFF_ARTIFACT }}
exit 1
else
echo "No schema changes detected"
fi
- name: Upload schema diff
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v3
if: ${{ failure() }}
with:
name: 'schema_schanges.txt'

View File

@@ -69,14 +69,14 @@ jobs:
steps:
- name: checkout dev
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
persist-credentials: false
- name: Setup Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: "3.9"
python-version: "3.8"
- name: Install python dependencies
run: |
@@ -94,11 +94,7 @@ jobs:
# integration tests generate a ton of logs in different files. the next step will find them all.
# we actually care if these pass, because the normal test run doesn't usually include many json log outputs
- name: Run integration tests
uses: nick-fields/retry@v3
with:
timeout_minutes: 30
max_attempts: 3
command: tox -e integration -- -nauto
run: tox -e integration -- -nauto
env:
PYTEST_ADDOPTS: ${{ format('--splits {0} --group {1}', env.PYTHON_INTEGRATION_TEST_WORKERS, matrix.split-group) }}

View File

@@ -27,6 +27,7 @@ on:
description: 'Version of Python to Test Against'
type: choice
options:
- '3.8'
- '3.9'
- '3.10'
- '3.11'
@@ -35,7 +36,7 @@ on:
type: choice
options:
- 'ubuntu-latest'
- 'macos-12'
- 'macos-latest'
- 'windows-latest'
num_runs_per_batch:
description: 'Max number of times to run the test per batch. We always run 10 batches.'
@@ -82,12 +83,12 @@ jobs:
steps:
- name: "Checkout code"
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
ref: ${{ inputs.branch }}
- name: "Setup Python"
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: "${{ inputs.python_version }}"
@@ -100,7 +101,7 @@ jobs:
# mac and windows don't use make due to limitations with docker with those runners in GitHub
- name: "Set up postgres (macos)"
if: inputs.os == 'macos-12'
if: inputs.os == 'macos-latest'
uses: ./.github/actions/setup-postgres-macos
- name: "Set up postgres (windows)"

View File

@@ -1,9 +1,9 @@
# Configuration for pre-commit hooks (see https://pre-commit.com/).
# Eventually the hooks described here will be run as tests before merging each PR.
exclude: ^(core/dbt/docs/build/|core/dbt/events/types_pb2.py)
exclude: ^(core/dbt/docs/build/|core/dbt/common/events/types_pb2.py|core/dbt/events/core_types_pb2.py|core/dbt/adapters/events/adapter_types_pb2.py)
# Force all unspecified python hooks to run python 3.9
# Force all unspecified python hooks to run python 3.8
default_language_version:
python: python3

View File

@@ -26,7 +26,7 @@ Legacy tests are found in the 'test' directory:
The "tasks" map to top-level dbt commands. So `dbt run` => task.run.RunTask, etc. Some are more like abstract base classes (GraphRunnableTask, for example) but all the concrete types outside of task should map to tasks. Currently one executes at a time. The tasks kick off their “Runners” and those do execute in parallel. The parallelism is managed via a thread pool, in GraphRunnableTask.
core/dbt/include/index.html
core/dbt/task/docs/index.html
This is the docs website code. It comes from the dbt-docs repository, and is generated when a release is packaged.
## Adapters

View File

@@ -5,347 +5,12 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.7.18 - August 07, 2024
### Fixes
- respect --quiet and --warn-error-options for flag deprecations ([#10105](https://github.com/dbt-labs/dbt-core/issues/10105))
## dbt-core 1.7.17 - June 20, 2024
### Docs
- Fix npm security vulnerabilities as of June 2024 ([dbt-docs/#513](https://github.com/dbt-labs/dbt-docs/issues/513))
## dbt-core 1.7.16 - June 05, 2024
### Features
- Add --host flag to dbt docs serve, defaulting to '127.0.0.1' ([#10229](https://github.com/dbt-labs/dbt-core/issues/10229))
## dbt-core 1.7.15 - May 22, 2024
### Fixes
- Fix the semicolon semantics for indexes while respecting other bug fix ([#85](https://github.com/dbt-labs/dbt-core/issues/85))
### Security
- Explicitly bind to localhost in docs serve ([#10209](https://github.com/dbt-labs/dbt-core/issues/10209))
## dbt-core 1.7.14 - May 02, 2024
### Features
- Move flags from UserConfig in profiles.yml to flags in dbt_project.yml ([#9183](https://github.com/dbt-labs/dbt-core/issues/9183))
- Add require_explicit_package_overrides_for_builtin_materializations to dbt_project.yml flags, which can be used to opt-out of overriding built-in materializations from packages ([#10007](https://github.com/dbt-labs/dbt-core/issues/10007))
### Fixes
- remove materialized views from renambeable relation and remove a quote ([#127](https://github.com/dbt-labs/dbt-core/issues/127))
- Replace usage of `Set` with `List` to fix issue with index updates intermittently happening out of order ([#72](https://github.com/dbt-labs/dbt-core/issues/72))
### Under the Hood
- Raise deprecation warning if installed package overrides built-in materialization ([#9971](https://github.com/dbt-labs/dbt-core/issues/9971))
- Remove the final underscore from secret environment variable constants. ([#10052](https://github.com/dbt-labs/dbt-core/issues/10052))
## dbt-core 1.7.13 - April 18, 2024
### Security
- Bump sqlparse to >=0.5.0, <0.6.0 to address GHSA-2m57-hf25-phgg ([#9951](https://github.com/dbt-labs/dbt-core/pull/9951))
### Contributors
- [@emmoop](https://github.com/emmoop) ([#9951](https://github.com/dbt-labs/dbt-core/pull/9951))
## dbt-core 1.7.12 - April 16, 2024
### Fixes
- Fix assorted source freshness edgecases so check is run or actionable information is given ([#9078](https://github.com/dbt-labs/dbt-core/issues/9078))
- Exclude password-like fields for considering reparse ([#9795](https://github.com/dbt-labs/dbt-core/issues/9795))
## dbt-core 1.7.11 - March 28, 2024
### Fixes
- Tighten exception handling to avoid worker thread hangs. ([#9583](https://github.com/dbt-labs/dbt-core/issues/9583))
- Add field wrapper to BaseRelation members that were missing it. ([#9681](https://github.com/dbt-labs/dbt-core/issues/9681))
## dbt-core 1.7.10 - March 14, 2024
### Fixes
- Do not add duplicate input_measures ([#9360](https://github.com/dbt-labs/dbt-core/issues/9360))
- Fix partial parsing `KeyError` on deleted schema files ([#8860](https://github.com/dbt-labs/dbt-core/issues/8860))
- Support saved queries in `dbt list` ([#9532](https://github.com/dbt-labs/dbt-core/issues/9532))
### Dependencies
- Restrict protobuf to 4.* versions ([#9566](https://github.com/dbt-labs/dbt-core/pull/9566))
## dbt-core 1.7.9 - February 28, 2024
### Fixes
- Fix node_info contextvar handling so incorrect node_info doesn't persist ([#8866](https://github.com/dbt-labs/dbt-core/issues/8866))
- Add target-path to retry ([#8948](https://github.com/dbt-labs/dbt-core/issues/8948))
### Under the Hood
- Make dbt-core compatible with Python 3.12 ([#9007](https://github.com/dbt-labs/dbt-core/issues/9007))
- Restrict protobuf to major version 4. ([#9566](https://github.com/dbt-labs/dbt-core/issues/9566))
### Security
- Update Jinja2 to >= 3.1.3 to address CVE-2024-22195 ([#CVE-2024-22195](https://github.com/dbt-labs/dbt-core/pull/CVE-2024-22195))
### Contributors
- [@l1xnan](https://github.com/l1xnan) ([#9007](https://github.com/dbt-labs/dbt-core/issues/9007))
## dbt-core 1.7.8 - February 14, 2024
### Fixes
- When patching versioned models, set constraints after config ([#9364](https://github.com/dbt-labs/dbt-core/issues/9364))
- Store node_info in node associated logging events ([#9557](https://github.com/dbt-labs/dbt-core/issues/9557))
## dbt-core 1.7.7 - February 01, 2024
### Fixes
- Fix seed and source selection in `dbt docs generate` ([#9161](https://github.com/dbt-labs/dbt-core/issues/9161))
- Add TestGenerateCatalogWithExternalNodes, include empty nodes in node selection during docs generate ([#9456](https://github.com/dbt-labs/dbt-core/issues/9456))
## dbt-core 1.7.6 - January 25, 2024
### Fixes
- Handle unknown `type_code` for model contracts ([#8877](https://github.com/dbt-labs/dbt-core/issues/8877), [#8353](https://github.com/dbt-labs/dbt-core/issues/8353))
- Fix retry command run from CLI ([#9444](https://github.com/dbt-labs/dbt-core/issues/9444))
## dbt-core 1.7.5 - January 18, 2024
### Fixes
- Preserve the value of vars and the --full-refresh flags when using retry. ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))
### Contributors
- [@peterallenwebb,](https://github.com/peterallenwebb,) ([#9112](https://github.com/dbt-labs/dbt-core/issues/9112))
## dbt-core 1.7.4 - December 14, 2023
### Features
- Adds support for parsing conversion metric related properties for the semantic layer. ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))
### Fixes
- Ensure we produce valid jsonschema schemas for manifest, catalog, run-results, and sources ([#8991](https://github.com/dbt-labs/dbt-core/issues/8991))
### Contributors
- [@WilliamDee](https://github.com/WilliamDee) ([#9203](https://github.com/dbt-labs/dbt-core/issues/9203))
## dbt-core 1.7.3 - November 29, 2023
### Fixes
- deps: Lock git packages to commit SHA during resolution ([#9050](https://github.com/dbt-labs/dbt-core/issues/9050))
- deps: Use PackageRenderer to read package-lock.json ([#9127](https://github.com/dbt-labs/dbt-core/issues/9127))
- Get sources working again in dbt docs generate ([#9119](https://github.com/dbt-labs/dbt-core/issues/9119))
## dbt-core 1.7.2 - November 16, 2023
### Features
- Support setting export configs hierarchically via saved query and project configs ([#8956](https://github.com/dbt-labs/dbt-core/issues/8956))
### Fixes
- Fix formatting of tarball information in packages-lock.yml ([#9062](https://github.com/dbt-labs/dbt-core/issues/9062))
### Under the Hood
- Treat SystemExit as an interrupt if raised during node execution. ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))
### Contributors
- [@benmosher](https://github.com/benmosher) ([#n/a](https://github.com/dbt-labs/dbt-core/issues/n/a))
## dbt-core 1.7.1 - November 07, 2023
### Fixes
- Fix compilation exception running empty seed file and support new Integer agate data_type ([#8895](https://github.com/dbt-labs/dbt-core/issues/8895))
- Update run_results.json from previous versions of dbt to support deferral and rerun from failure ([#9010](https://github.com/dbt-labs/dbt-core/issues/9010))
- Use MANIFEST.in to recursively include all jinja templates; fixes issue where some templates were not included in the distribution ([#9016](https://github.com/dbt-labs/dbt-core/issues/9016))
- Fix git repository with subdirectory for Deps ([#9000](https://github.com/dbt-labs/dbt-core/issues/9000))
## dbt-core 1.7.0 - November 02, 2023
### Breaking Changes
- Removed the FirstRunResultError and AfterFirstRunResultError event types, using the existing RunResultError in their place. ([#7963](https://github.com/dbt-labs/dbt-core/issues/7963))
### Features
- add log file of installed packages via dbt deps ([#6643](https://github.com/dbt-labs/dbt-core/issues/6643))
- Enable re-population of metadata vars post-environment change during programmatic invocation ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- Added support to configure a delimiter for a seed file, defaults to comma ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- Allow specification of `create_metric: true` on measures ([#8125](https://github.com/dbt-labs/dbt-core/issues/8125))
- Add node attributes related to compilation to run_results.json ([#7519](https://github.com/dbt-labs/dbt-core/issues/7519))
- Add --no-inject-ephemeral-ctes flag for `compile` command, for usage by linting. ([#8480](https://github.com/dbt-labs/dbt-core/issues/8480))
- Support configuration of semantic models with the addition of enable/disable and group enablement. ([#7968](https://github.com/dbt-labs/dbt-core/issues/7968))
- Accept a `dbt-cloud` config in dbt_project.yml ([#8438](https://github.com/dbt-labs/dbt-core/issues/8438))
- Support atomic replace in the global replace macro ([#8539](https://github.com/dbt-labs/dbt-core/issues/8539))
- Use translate_type on data_type in model.columns in templates by default, remove no op `TYPE_LABELS` ([#8007](https://github.com/dbt-labs/dbt-core/issues/8007))
- Add an option to generate static documentation ([#8614](https://github.com/dbt-labs/dbt-core/issues/8614))
- Allow setting "access" as a config in addition to as a property ([#8383](https://github.com/dbt-labs/dbt-core/issues/8383))
- Loosen typing requirement on renameable/replaceable relations to Iterable to allow adapters more flexibility in registering relation types, include docstrings as suggestions ([#8647](https://github.com/dbt-labs/dbt-core/issues/8647))
- Add support for optional label in semantic_models, measures, dimensions and entities. ([#8595](https://github.com/dbt-labs/dbt-core/issues/8595), [#8755](https://github.com/dbt-labs/dbt-core/issues/8755))
- Allow adapters to include package logs in dbt standard logging ([#7859](https://github.com/dbt-labs/dbt-core/issues/7859))
- Support storing test failures as views ([#6914](https://github.com/dbt-labs/dbt-core/issues/6914))
- resolve packages with same git repo and unique subdirectory ([#5374](https://github.com/dbt-labs/dbt-core/issues/5374))
- Add new ResourceReport event to record memory/cpu/io metrics ([#8342](https://github.com/dbt-labs/dbt-core/issues/8342))
- Adding `date_spine` macro (and supporting macros) from dbt-utils to dbt-core ([#8172](https://github.com/dbt-labs/dbt-core/issues/8172))
- Support `fill_nulls_with` and `join_to_timespine` for metric nodes ([#8593](https://github.com/dbt-labs/dbt-core/issues/8593), [#8755](https://github.com/dbt-labs/dbt-core/issues/8755))
- Raise a warning when a contracted model has a numeric field without scale defined ([#8183](https://github.com/dbt-labs/dbt-core/issues/8183))
- Added support for retrieving partial catalog information from a schema ([#8521](https://github.com/dbt-labs/dbt-core/issues/8521))
- Add meta attribute to SemanticModels config ([#8511](https://github.com/dbt-labs/dbt-core/issues/8511))
- Selectors with docs generate limits catalog generation ([#6014](https://github.com/dbt-labs/dbt-core/issues/6014))
- Allow freshness to be determined via DBMS metadata for supported adapters ([#8704](https://github.com/dbt-labs/dbt-core/issues/8704))
- Add support semantic layer SavedQuery node type ([#8594](https://github.com/dbt-labs/dbt-core/issues/8594))
- Add exports to SavedQuery spec ([#8892](https://github.com/dbt-labs/dbt-core/issues/8892))
### Fixes
- Copy dir during `dbt deps` if symlink fails ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- If --profile specified with dbt-init, create the project with the specified profile ([#6154](https://github.com/dbt-labs/dbt-core/issues/6154))
- Fixed double-underline ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Enable converting deprecation warnings to errors ([#8130](https://github.com/dbt-labs/dbt-core/issues/8130))
- Add status to Parse Inline Error ([#8173](https://github.com/dbt-labs/dbt-core/issues/8173))
- Ensure `warn_error_options` get serialized in `invocation_args_dict` ([#7694](https://github.com/dbt-labs/dbt-core/issues/7694))
- Stop detecting materialization macros based on macro name ([#6231](https://github.com/dbt-labs/dbt-core/issues/6231))
- Update `dbt deps` download retry logic to handle `EOFError` exceptions ([#6653](https://github.com/dbt-labs/dbt-core/issues/6653))
- Improve handling of CTE injection with ephemeral models ([#8213](https://github.com/dbt-labs/dbt-core/issues/8213))
- Fix unbound local variable error in `checked_agg_time_dimension_for_measure` ([#8230](https://github.com/dbt-labs/dbt-core/issues/8230))
- Ensure runtime errors are raised for graph runnable tasks (compile, show, run, etc) ([#8166](https://github.com/dbt-labs/dbt-core/issues/8166))
- Fix retry not working with log-file-max-bytes ([#8297](https://github.com/dbt-labs/dbt-core/issues/8297))
- Add explicit support for integers for the show command ([#8153](https://github.com/dbt-labs/dbt-core/issues/8153))
- Detect changes to model access, version, or latest_version in state:modified ([#8189](https://github.com/dbt-labs/dbt-core/issues/8189))
- Add connection status into list of statuses for dbt debug ([#8350](https://github.com/dbt-labs/dbt-core/issues/8350))
- fix fqn-selection for external versioned models ([#8374](https://github.com/dbt-labs/dbt-core/issues/8374))
- Fix: DbtInternalError after model that previously ref'd external model is deleted ([#8375](https://github.com/dbt-labs/dbt-core/issues/8375))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
- Remedy performance regression by only writing run_results.json once. ([#8360](https://github.com/dbt-labs/dbt-core/issues/8360))
- Add support for swapping materialized views with tables/views and vice versa ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Turn breaking changes to contracted models into warnings for unversioned models ([#8384](https://github.com/dbt-labs/dbt-core/issues/8384), [#8282](https://github.com/dbt-labs/dbt-core/issues/8282))
- Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension` ([#8453](https://github.com/dbt-labs/dbt-core/issues/8453))
- fix ambiguous reference error for tests and versions when model name is duplicated across packages ([#8327](https://github.com/dbt-labs/dbt-core/issues/8327), [#8493](https://github.com/dbt-labs/dbt-core/issues/8493))
- Fix "Internal Error: Expected node <unique-id> not found in manifest" when depends_on set on ModelNodeArgs ([#8506](https://github.com/dbt-labs/dbt-core/issues/8506))
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
- Parse the correct schema version from manifest ([#8544](https://github.com/dbt-labs/dbt-core/issues/8544))
- make version comparison insensitive to order ([#8571](https://github.com/dbt-labs/dbt-core/issues/8571))
- Update metric helper functions to work with new semantic layer metrics ([#8134](https://github.com/dbt-labs/dbt-core/issues/8134))
- Disallow cleaning paths outside current working directory ([#8318](https://github.com/dbt-labs/dbt-core/issues/8318))
- Warn when --state == --target ([#8160](https://github.com/dbt-labs/dbt-core/issues/8160))
- update dbt show to include limit in DWH query ([#8496,](https://github.com/dbt-labs/dbt-core/issues/8496,), [#8417](https://github.com/dbt-labs/dbt-core/issues/8417))
- Support quoted parameter list for MultiOption CLI options. ([#8598](https://github.com/dbt-labs/dbt-core/issues/8598))
- Support global flags passed in after subcommands ([#6497](https://github.com/dbt-labs/dbt-core/issues/6497))
- Lower bound of `8.0.2` for `click` ([#8683](https://github.com/dbt-labs/dbt-core/issues/8683))
- Fixes test type edges filter ([#8692](https://github.com/dbt-labs/dbt-core/issues/8692))
- semantic models in graph selection ([#8589](https://github.com/dbt-labs/dbt-core/issues/8589))
- Support doc blocks in nested semantic model YAML ([#8509](https://github.com/dbt-labs/dbt-core/issues/8509))
- avoid double-rendering sql_header in dbt show ([#8739](https://github.com/dbt-labs/dbt-core/issues/8739))
- Fix tag selection for projects with semantic models ([#8749](https://github.com/dbt-labs/dbt-core/issues/8749))
- Foreign key constraint on incremental model results in Database Error ([#8022](https://github.com/dbt-labs/dbt-core/issues/8022))
- Support docs blocks on versioned model column descriptions ([#8540](https://github.com/dbt-labs/dbt-core/issues/8540))
- Enable seeds to be handled from stored manifest data ([#6875](https://github.com/dbt-labs/dbt-core/issues/6875))
- Override path-like args in dbt retry ([#8682](https://github.com/dbt-labs/dbt-core/issues/8682))
- Group updates on unmodified nodes are handled gracefully for state:modified ([#8371](https://github.com/dbt-labs/dbt-core/issues/8371))
- Partial parsing fix for adding groups and updating models at the same time ([#8697](https://github.com/dbt-labs/dbt-core/issues/8697))
- Fix partial parsing not working for semantic model change ([#8859](https://github.com/dbt-labs/dbt-core/issues/8859))
- Rework get_catalog implementation to retain previous adapter interface semantics ([#8846](https://github.com/dbt-labs/dbt-core/issues/8846))
- Add back contract enforcement for temporary tables on postgres ([#8857](https://github.com/dbt-labs/dbt-core/issues/8857))
- Add version to fqn when version==0 ([#8836](https://github.com/dbt-labs/dbt-core/issues/8836))
- Fix cased comparison in catalog-retrieval function. ([#8939](https://github.com/dbt-labs/dbt-core/issues/8939))
- Catalog queries now assign the correct type to materialized views ([#8864](https://github.com/dbt-labs/dbt-core/issues/8864))
- Make relation filtering None-tolerant for maximal flexibility across adapters. ([#8974](https://github.com/dbt-labs/dbt-core/issues/8974))
### Docs
- Corrected spelling of "Partiton" ([dbt-docs/#8100](https://github.com/dbt-labs/dbt-docs/issues/8100))
- Remove static SQL codeblock for metrics ([dbt-docs/#436](https://github.com/dbt-labs/dbt-docs/issues/436))
- fixed comment util.py ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Fix newline escapes and improve formatting in docker README ([dbt-docs/#8211](https://github.com/dbt-labs/dbt-docs/issues/8211))
- Display contract and column constraints on the model page ([dbt-docs/#433](https://github.com/dbt-labs/dbt-docs/issues/433))
- Display semantic model details in docs ([dbt-docs/#431](https://github.com/dbt-labs/dbt-docs/issues/431))
### Under the Hood
- Switch from hologram to mashumaro jsonschema ([#8426](https://github.com/dbt-labs/dbt-core/issues/8426))
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))
- format exception from dbtPlugin.initialize ([#8152](https://github.com/dbt-labs/dbt-core/issues/8152))
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
- Ref expressions with version can now be processed by the latest version of the high-performance dbt-extractor library. ([#7688](https://github.com/dbt-labs/dbt-core/issues/7688))
- Bump manifest schema version to v11, freeze manifest v10 ([#8333](https://github.com/dbt-labs/dbt-core/issues/8333))
- add tracking for plugin.get_nodes calls ([#8344](https://github.com/dbt-labs/dbt-core/issues/8344))
- add internal flag: --no-partial-parse-file-diff to inform whether to compute a file diff during partial parsing ([#8363](https://github.com/dbt-labs/dbt-core/issues/8363))
- Add return values to a number of functions for mypy ([#8389](https://github.com/dbt-labs/dbt-core/issues/8389))
- Fix mypy warnings for ManifestLoader.load() ([#8401](https://github.com/dbt-labs/dbt-core/issues/8401))
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))
- Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>, relation agnostic in /macros/relations ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Update typing to meet mypy standards ([#8396](https://github.com/dbt-labs/dbt-core/issues/8396))
- Mypy errors - adapters/factory.py ([#8387](https://github.com/dbt-labs/dbt-core/issues/8387))
- Added more type annotations. ([#8537](https://github.com/dbt-labs/dbt-core/issues/8537))
- Audit potential circular dependencies ([#8349](https://github.com/dbt-labs/dbt-core/issues/8349))
- Add functional test for advanced ref override ([#8566](https://github.com/dbt-labs/dbt-core/issues/8566))
- Add typing to __init__ in base.py ([#8398](https://github.com/dbt-labs/dbt-core/issues/8398))
- Fix untyped functions in task/runnable.py (mypy warning) ([#8402](https://github.com/dbt-labs/dbt-core/issues/8402))
- add a test for ephemeral cte injection ([#8225](https://github.com/dbt-labs/dbt-core/issues/8225))
- Fix test_numeric_values to look for more specific strings ([#8470](https://github.com/dbt-labs/dbt-core/issues/8470))
- Pin types-requests<2.31.0 in `dev-requirements.txt` ([#8789](https://github.com/dbt-labs/dbt-core/issues/8789))
- Add warning_tag to UnversionedBreakingChange ([#8827](https://github.com/dbt-labs/dbt-core/issues/8827))
- Update v10 manifest schema to match 1.6 for testing schema compatibility ([#8835](https://github.com/dbt-labs/dbt-core/issues/8835))
- Add a no-op runner for Saved Qeury ([#8893](https://github.com/dbt-labs/dbt-core/issues/8893))
### Dependencies
- Bump mypy from 1.3.0 to 1.4.0 ([#7912](https://github.com/dbt-labs/dbt-core/pull/7912))
- Bump mypy from 1.4.0 to 1.4.1 ([#8219](https://github.com/dbt-labs/dbt-core/pull/8219))
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))
- Support dbt-semantic-interfaces 0.2.0 ([#8250](https://github.com/dbt-labs/dbt-core/pull/8250))
- Bump docker/build-push-action from 4 to 5 ([#8783](https://github.com/dbt-labs/dbt-core/pull/8783))
- Upgrade dbt-semantic-interfaces dep to 0.3.0 ([#8819](https://github.com/dbt-labs/dbt-core/pull/8819))
- Begin using DSI 0.4.x ([#8892](https://github.com/dbt-labs/dbt-core/pull/8892))
### Contributors
- [@anjutiwari](https://github.com/anjutiwari) ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- [@benmosher](https://github.com/benmosher) ([#8480](https://github.com/dbt-labs/dbt-core/issues/8480))
- [@d-kaneshiro](https://github.com/d-kaneshiro) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@dave-connors-3](https://github.com/dave-connors-3) ([#8153](https://github.com/dbt-labs/dbt-core/issues/8153), [#8589](https://github.com/dbt-labs/dbt-core/issues/8589))
- [@dylan-murray](https://github.com/dylan-murray) ([#8683](https://github.com/dbt-labs/dbt-core/issues/8683))
- [@ezraerb](https://github.com/ezraerb) ([#6154](https://github.com/dbt-labs/dbt-core/issues/6154))
- [@gem7318](https://github.com/gem7318) ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- [@jamezrin](https://github.com/jamezrin) ([#8211](https://github.com/dbt-labs/dbt-core/issues/8211))
- [@jusbaldw](https://github.com/jusbaldw) ([#6643](https://github.com/dbt-labs/dbt-core/issues/6643))
- [@lllong33](https://github.com/lllong33) ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- [@marcodamore](https://github.com/marcodamore) ([#436](https://github.com/dbt-labs/dbt-core/issues/436))
- [@mescanne](https://github.com/mescanne) ([#8614](https://github.com/dbt-labs/dbt-core/issues/8614))
- [@pgoslatara](https://github.com/pgoslatara) ([#8100](https://github.com/dbt-labs/dbt-core/issues/8100))
- [@philippeboyd](https://github.com/philippeboyd) ([#5374](https://github.com/dbt-labs/dbt-core/issues/5374))
- [@ramonvermeulen](https://github.com/ramonvermeulen) ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- [@renanleme](https://github.com/renanleme) ([#8692](https://github.com/dbt-labs/dbt-core/issues/8692))
## Previous Releases
For information on prior major and minor releases, see their changelogs:
* [1.7](https://github.com/dbt-labs/dbt-core/blob/1.7.latest/CHANGELOG.md)
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)

View File

@@ -33,6 +33,9 @@ RUN apt-get update \
python-is-python3 \
python-dev-is-python3 \
python3-pip \
python3.8 \
python3.8-dev \
python3.8-venv \
python3.9 \
python3.9-dev \
python3.9-venv \

View File

@@ -40,7 +40,16 @@ dev: dev_req ## Installs dbt-* packages in develop mode along with development d
.PHONY: proto_types
proto_types: ## generates google protobuf python file from types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/types.proto
protoc -I=./core/dbt/common/events --python_out=./core/dbt/common/events ./core/dbt/common/events/types.proto
.PHONY: core_proto_types
core_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/core_types.proto
.PHONY: adapter_proto_types
adapter_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/adapters/events --python_out=./core/dbt/adapters/events ./core/dbt/adapters/events/adapter_types.proto
.PHONY: mypy
mypy: .env ## Runs mypy against staged changes for static type checking.

View File

@@ -1,7 +1,7 @@
# these are all just exports, #noqa them so flake8 will be happy
# TODO: Should we still include this in the `adapters` namespace?
from dbt.contracts.connection import Credentials # noqa: F401
from dbt.adapters.contracts.connection import Credentials # noqa: F401
from dbt.adapters.base.meta import available # noqa: F401
from dbt.adapters.base.connections import BaseConnectionManager # noqa: F401
from dbt.adapters.base.relation import ( # noqa: F401

View File

@@ -2,7 +2,7 @@ from dataclasses import dataclass
import re
from typing import Dict, ClassVar, Any, Optional
from dbt.exceptions import DbtRuntimeError
from dbt.common.exceptions import DbtRuntimeError
@dataclass

View File

@@ -6,6 +6,7 @@ import traceback
# multiprocessing.RLock is a function returning this type
from multiprocessing.synchronize import RLock
from multiprocessing.context import SpawnContext
from threading import get_ident
from typing import (
Any,
@@ -23,8 +24,9 @@ from typing import (
import agate
import dbt.exceptions
from dbt.contracts.connection import (
import dbt.adapters.exceptions
import dbt.common.exceptions.base
from dbt.adapters.contracts.connection import (
Connection,
Identifier,
ConnectionState,
@@ -36,9 +38,9 @@ from dbt.contracts.graph.manifest import Manifest
from dbt.adapters.base.query_headers import (
MacroQueryStringSetter,
)
from dbt.events import AdapterLogger
from dbt.events.functions import fire_event
from dbt.events.types import (
from dbt.adapters.events.logging import AdapterLogger
from dbt.common.events.functions import fire_event
from dbt.adapters.events.types import (
NewConnection,
ConnectionReused,
ConnectionLeftOpenInCleanup,
@@ -48,9 +50,8 @@ from dbt.events.types import (
Rollback,
RollbackFailed,
)
from dbt.events.contextvars import get_node_info
from dbt import flags
from dbt.utils import cast_to_str
from dbt.common.events.contextvars import get_node_info
from dbt.common.utils import cast_to_str
SleepTime = Union[int, float] # As taken by time.sleep.
AdapterHandle = Any # Adapter connection handle objects can be any class.
@@ -72,10 +73,10 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
TYPE: str = NotImplemented
def __init__(self, profile: AdapterRequiredConfig) -> None:
def __init__(self, profile: AdapterRequiredConfig, mp_context: SpawnContext) -> None:
self.profile = profile
self.thread_connections: Dict[Hashable, Connection] = {}
self.lock: RLock = flags.MP_CONTEXT.RLock()
self.lock: RLock = mp_context.RLock()
self.query_header: Optional[MacroQueryStringSetter] = None
def set_query_header(self, manifest: Manifest) -> None:
@@ -91,13 +92,15 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
key = self.get_thread_identifier()
with self.lock:
if key not in self.thread_connections:
raise dbt.exceptions.InvalidConnectionError(key, list(self.thread_connections))
raise dbt.adapters.exceptions.InvalidConnectionError(
key, list(self.thread_connections)
)
return self.thread_connections[key]
def set_thread_connection(self, conn: Connection) -> None:
key = self.get_thread_identifier()
if key in self.thread_connections:
raise dbt.exceptions.DbtInternalError(
raise dbt.common.exceptions.DbtInternalError(
"In set_thread_connection, existing connection exists for {}"
)
self.thread_connections[key] = conn
@@ -137,13 +140,13 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:return: A context manager that handles exceptions raised by the
underlying database.
"""
raise dbt.exceptions.NotImplementedError(
raise dbt.common.exceptions.base.NotImplementedError(
"`exception_handler` is not implemented for this adapter!"
)
def set_connection_name(self, name: Optional[str] = None) -> Connection:
"""Called by 'acquire_connection' in BaseAdapter, which is called by
'connection_named', called by 'connection_for(node)'.
'connection_named'.
Creates a connection for this thread if one doesn't already
exist, and will rename an existing connection."""
@@ -220,14 +223,14 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:param int _attempts: Parameter used to keep track of the number of attempts in calling the
connect function across recursive calls. Passed as an argument to retry_timeout if it
is a Callable. This parameter should not be set by the initial caller.
:raises dbt.exceptions.FailedToConnectError: Upon exhausting all retry attempts without
:raises dbt.adapters.exceptions.FailedToConnectError: Upon exhausting all retry attempts without
successfully acquiring a handle.
:return: The given connection with its appropriate state and handle attributes set
depending on whether we successfully acquired a handle or not.
"""
timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout
if timeout < 0:
raise dbt.exceptions.FailedToConnectError(
raise dbt.adapters.exceptions.FailedToConnectError(
"retry_timeout cannot be negative or return a negative time."
)
@@ -235,7 +238,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
# This guard is not perfect others may add to the recursion limit (e.g. built-ins).
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError("retry_limit cannot be negative")
raise dbt.adapters.exceptions.FailedToConnectError("retry_limit cannot be negative")
try:
connection.handle = connect()
@@ -246,7 +249,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
if retry_limit <= 0:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError(str(e))
raise dbt.adapters.exceptions.FailedToConnectError(str(e))
logger.debug(
f"Got a retryable error when attempting to open a {cls.TYPE} connection.\n"
@@ -268,12 +271,12 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
except Exception as e:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError(str(e))
raise dbt.adapters.exceptions.FailedToConnectError(str(e))
@abc.abstractmethod
def cancel_open(self) -> Optional[List[str]]:
"""Cancel all open connections on the adapter. (passable)"""
raise dbt.exceptions.NotImplementedError(
raise dbt.common.exceptions.base.NotImplementedError(
"`cancel_open` is not implemented for this adapter!"
)
@@ -288,7 +291,9 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
This should be thread-safe, or hold the lock if necessary. The given
connection should not be in either in_use or available.
"""
raise dbt.exceptions.NotImplementedError("`open` is not implemented for this adapter!")
raise dbt.common.exceptions.base.NotImplementedError(
"`open` is not implemented for this adapter!"
)
def release(self) -> None:
with self.lock:
@@ -320,12 +325,16 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
@abc.abstractmethod
def begin(self) -> None:
"""Begin a transaction. (passable)"""
raise dbt.exceptions.NotImplementedError("`begin` is not implemented for this adapter!")
raise dbt.common.exceptions.base.NotImplementedError(
"`begin` is not implemented for this adapter!"
)
@abc.abstractmethod
def commit(self) -> None:
"""Commit a transaction. (passable)"""
raise dbt.exceptions.NotImplementedError("`commit` is not implemented for this adapter!")
raise dbt.common.exceptions.base.NotImplementedError(
"`commit` is not implemented for this adapter!"
)
@classmethod
def _rollback_handle(cls, connection: Connection) -> None:
@@ -361,7 +370,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
def _rollback(cls, connection: Connection) -> None:
"""Roll back the given connection."""
if connection.transaction_open is False:
raise dbt.exceptions.DbtInternalError(
raise dbt.common.exceptions.DbtInternalError(
f"Tried to rollback transaction on connection "
f'"{connection.name}", but it does not have one open!'
)
@@ -412,7 +421,9 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:return: A tuple of the query status and results (empty if fetch=False).
:rtype: Tuple[AdapterResponse, agate.Table]
"""
raise dbt.exceptions.NotImplementedError("`execute` is not implemented for this adapter!")
raise dbt.common.exceptions.base.NotImplementedError(
"`execute` is not implemented for this adapter!"
)
def add_select_query(self, sql: str) -> Tuple[Connection, Any]:
"""
@@ -422,7 +433,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
See https://github.com/dbt-labs/dbt-core/issues/8396 for more information.
"""
raise dbt.exceptions.NotImplementedError(
raise dbt.common.exceptions.base.NotImplementedError(
"`add_select_query` is not implemented for this adapter!"
)
@@ -430,6 +441,6 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
def data_type_code_to_name(cls, type_code: Union[int, str]) -> str:
"""Get the string representation of the data type from the type_code."""
# https://peps.python.org/pep-0249/#type-objects
raise dbt.exceptions.NotImplementedError(
raise dbt.common.exceptions.base.NotImplementedError(
"`data_type_code_to_name` is not implemented for this adapter!"
)

View File

@@ -9,7 +9,6 @@ from typing import (
Any,
Callable,
Dict,
Iterable,
Iterator,
List,
Mapping,
@@ -20,44 +19,52 @@ from typing import (
TypedDict,
Union,
)
from multiprocessing.context import SpawnContext
from dbt.adapters.capability import Capability, CapabilityDict
from dbt.contracts.graph.nodes import ColumnLevelConstraint, ConstraintType, ModelLevelConstraint
from dbt.common.contracts.constraints import (
ColumnLevelConstraint,
ConstraintType,
ModelLevelConstraint,
)
from dbt.adapters.contracts.macros import MacroResolver
import agate
import pytz
from dbt.exceptions import (
from dbt.adapters.exceptions import (
SnapshotTargetIncompleteError,
SnapshotTargetNotSnapshotTableError,
NullRelationDropAttemptedError,
NullRelationCacheAttemptedError,
RelationReturnedMultipleResultsError,
UnexpectedNonTimestampError,
RenameToNoneAttemptedError,
QuoteConfigTypeError,
)
from dbt.common.exceptions import (
NotImplementedError,
DbtInternalError,
DbtRuntimeError,
DbtValidationError,
UnexpectedNullError,
MacroArgTypeError,
MacroResultError,
NotImplementedError,
NullRelationCacheAttemptedError,
NullRelationDropAttemptedError,
QuoteConfigTypeError,
RelationReturnedMultipleResultsError,
RenameToNoneAttemptedError,
SnapshotTargetIncompleteError,
SnapshotTargetNotSnapshotTableError,
UnexpectedNonTimestampError,
UnexpectedNullError,
)
from dbt.adapters.protocol import AdapterConfig
from dbt.clients.agate_helper import (
from dbt.common.clients.agate_helper import (
empty_table,
get_column_value_uncased,
merge_tables,
table_from_rows,
Integer,
)
from dbt.clients.jinja import MacroGenerator
from dbt.contracts.graph.manifest import Manifest, MacroManifest
from dbt.contracts.graph.nodes import ResultNode
from dbt.events.functions import fire_event, warn_or_error
from dbt.events.types import (
from dbt.common.clients.jinja import CallableMacroGenerator
from dbt.contracts.graph.manifest import Manifest
from dbt.common.events.functions import fire_event, warn_or_error
from dbt.adapters.events.types import (
CacheMiss,
ListRelations,
CodeExecution,
@@ -66,7 +73,7 @@ from dbt.events.types import (
ConstraintNotSupported,
ConstraintNotEnforced,
)
from dbt.utils import filter_null_values, executor, cast_to_str, AttrDict
from dbt.common.utils import filter_null_values, executor, cast_to_str, AttrDict
from dbt.adapters.base.connections import Connection, AdapterResponse, BaseConnectionManager
from dbt.adapters.base.meta import AdapterMeta, available
@@ -79,7 +86,8 @@ from dbt.adapters.base.relation import (
from dbt.adapters.base import Column as BaseColumn
from dbt.adapters.base import Credentials
from dbt.adapters.cache import RelationsCache, _make_ref_key_dict
from dbt import deprecations
from dbt.adapters.events.types import CollectFreshnessReturnSignature
GET_CATALOG_MACRO_NAME = "get_catalog"
GET_CATALOG_RELATIONS_MACRO_NAME = "get_catalog_relations"
@@ -242,11 +250,24 @@ class BaseAdapter(metaclass=AdapterMeta):
# implementations to indicate adapter support for optional capabilities.
_capabilities = CapabilityDict({})
def __init__(self, config) -> None:
def __init__(self, config, mp_context: SpawnContext) -> None:
self.config = config
self.cache = RelationsCache()
self.connections = self.ConnectionManager(config)
self._macro_manifest_lazy: Optional[MacroManifest] = None
self.cache = RelationsCache(log_cache_events=config.log_cache_events)
self.connections = self.ConnectionManager(config, mp_context)
self._macro_resolver: Optional[MacroResolver] = None
###
# Methods to set / access a macro resolver
###
def set_macro_resolver(self, macro_resolver: MacroResolver) -> None:
self._macro_resolver = macro_resolver
def get_macro_resolver(self) -> Optional[MacroResolver]:
return self._macro_resolver
def clear_macro_resolver(self) -> None:
if self._macro_resolver is not None:
self._macro_resolver = None
###
# Methods that pass through to the connection manager
@@ -276,10 +297,10 @@ class BaseAdapter(metaclass=AdapterMeta):
return conn.name
@contextmanager
def connection_named(self, name: str, node: Optional[ResultNode] = None) -> Iterator[None]:
def connection_named(self, name: str, query_header_context: Any = None) -> Iterator[None]:
try:
if self.connections.query_header is not None:
self.connections.query_header.set(name, node)
self.connections.query_header.set(name, query_header_context)
self.acquire_connection(name)
yield
finally:
@@ -287,11 +308,6 @@ class BaseAdapter(metaclass=AdapterMeta):
if self.connections.query_header is not None:
self.connections.query_header.reset()
@contextmanager
def connection_for(self, node: ResultNode) -> Iterator[None]:
with self.connection_named(node.unique_id, node):
yield
@available.parse(lambda *a, **k: ("", empty_table()))
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
@@ -364,39 +380,6 @@ class BaseAdapter(metaclass=AdapterMeta):
"""
return cls.ConnectionManager.TYPE
@property
def _macro_manifest(self) -> MacroManifest:
if self._macro_manifest_lazy is None:
return self.load_macro_manifest()
return self._macro_manifest_lazy
def check_macro_manifest(self) -> Optional[MacroManifest]:
"""Return the internal manifest (used for executing macros) if it's
been initialized, otherwise return None.
"""
return self._macro_manifest_lazy
def load_macro_manifest(self, base_macros_only=False) -> MacroManifest:
# base_macros_only is for the test framework
if self._macro_manifest_lazy is None:
# avoid a circular import
from dbt.parser.manifest import ManifestLoader
manifest = ManifestLoader.load_macros(
self.config,
self.connections.set_query_header,
base_macros_only=base_macros_only,
)
# TODO CT-211
self._macro_manifest_lazy = manifest # type: ignore[assignment]
# TODO CT-211
return self._macro_manifest_lazy # type: ignore[return-value]
def clear_macro_manifest(self):
if self._macro_manifest_lazy is not None:
self._macro_manifest_lazy = None
###
# Caching methods
###
def _schema_is_cached(self, database: Optional[str], schema: str) -> bool:
@@ -420,7 +403,7 @@ class BaseAdapter(metaclass=AdapterMeta):
"""
# the cache only cares about executable nodes
return {
self.Relation.create_from(self.config, node).without_identifier()
self.Relation.create_from(self.config, node).without_identifier() # type: ignore[arg-type]
for node in manifest.nodes.values()
if (node.is_relational and not node.is_ephemeral_model and not node.is_external_node)
}
@@ -467,7 +450,7 @@ class BaseAdapter(metaclass=AdapterMeta):
manifest.sources.values(),
)
relations = [self.Relation.create_from(self.config, n) for n in nodes]
relations = [self.Relation.create_from(self.config, n) for n in nodes] # type: ignore[arg-type]
return relations
def _relations_cache_for_schemas(
@@ -1051,11 +1034,10 @@ class BaseAdapter(metaclass=AdapterMeta):
def execute_macro(
self,
macro_name: str,
manifest: Optional[Manifest] = None,
macro_resolver: Optional[MacroResolver] = None,
project: Optional[str] = None,
context_override: Optional[Dict[str, Any]] = None,
kwargs: Optional[Dict[str, Any]] = None,
text_only_columns: Optional[Iterable[str]] = None,
) -> AttrDict:
"""Look macro_name up in the manifest and execute its results.
@@ -1075,13 +1057,11 @@ class BaseAdapter(metaclass=AdapterMeta):
if context_override is None:
context_override = {}
if manifest is None:
# TODO CT-211
manifest = self._macro_manifest # type: ignore[assignment]
# TODO CT-211
macro = manifest.find_macro_by_name( # type: ignore[union-attr]
macro_name, self.config.project_name, project
)
resolver = macro_resolver or self._macro_resolver
if resolver is None:
raise DbtInternalError("macro resolver was None when calling execute_macro!")
macro = resolver.find_macro_by_name(macro_name, self.config.project_name, project)
if macro is None:
if project is None:
package_name = "any package"
@@ -1101,12 +1081,12 @@ class BaseAdapter(metaclass=AdapterMeta):
# TODO CT-211
macro=macro,
config=self.config,
manifest=manifest, # type: ignore[arg-type]
manifest=resolver, # type: ignore[arg-type]
package_name=project,
)
macro_context.update(context_override)
macro_function = MacroGenerator(macro, macro_context)
macro_function = CallableMacroGenerator(macro, macro_context)
with self.connections.exception_handler(f"macro {macro_name}"):
result = macro_function(**kwargs)
@@ -1137,7 +1117,7 @@ class BaseAdapter(metaclass=AdapterMeta):
kwargs=kwargs,
# pass in the full manifest so we get any local project
# overrides
manifest=manifest,
macro_resolver=manifest,
)
results = self._catalog_filter_table(table, manifest) # type: ignore[arg-type]
@@ -1159,7 +1139,7 @@ class BaseAdapter(metaclass=AdapterMeta):
kwargs=kwargs,
# pass in the full manifest, so we get any local project
# overrides
manifest=manifest,
macro_resolver=manifest,
)
results = self._catalog_filter_table(table, manifest) # type: ignore[arg-type]
@@ -1270,9 +1250,9 @@ class BaseAdapter(metaclass=AdapterMeta):
AttrDict, # current: contains AdapterResponse + agate.Table
agate.Table, # previous: just table
]
result = self.execute_macro(FRESHNESS_MACRO_NAME, kwargs=kwargs, manifest=manifest)
result = self.execute_macro(FRESHNESS_MACRO_NAME, kwargs=kwargs, macro_resolver=manifest)
if isinstance(result, agate.Table):
deprecations.warn("collect-freshness-return-signature")
warn_or_error(CollectFreshnessReturnSignature())
adapter_response = None
table = result
else:
@@ -1307,7 +1287,7 @@ class BaseAdapter(metaclass=AdapterMeta):
"relations": [source],
}
result = self.execute_macro(
GET_RELATION_LAST_MODIFIED_MACRO_NAME, kwargs=kwargs, manifest=manifest
GET_RELATION_LAST_MODIFIED_MACRO_NAME, kwargs=kwargs, macro_resolver=manifest
)
adapter_response, table = result.response, result.table # type: ignore[attr-defined]
@@ -1361,11 +1341,6 @@ class BaseAdapter(metaclass=AdapterMeta):
"""
pass
def get_compiler(self):
from dbt.compilation import Compiler
return Compiler(self.config)
# Methods used in adapter tests
def update_column_sql(
self,
@@ -1485,7 +1460,7 @@ class BaseAdapter(metaclass=AdapterMeta):
strategy = strategy.replace("+", "_")
macro_name = f"get_incremental_{strategy}_sql"
# The model_context should have MacroGenerator callable objects for all macros
# The model_context should have callable objects for all macros
if macro_name not in model_context:
raise DbtRuntimeError(
'dbt could not find an incremental strategy macro with the name "{}" in {}'.format(

View File

@@ -1,9 +1,8 @@
import abc
from functools import wraps
from typing import Callable, Optional, Any, FrozenSet, Dict, Set
from dbt.deprecations import warn, renamed_method
from dbt.common.events.functions import warn_or_error
from dbt.adapters.events.types import AdapterDeprecationWarning
Decorator = Callable[[Any], Callable]
@@ -62,11 +61,12 @@ class _Available:
def wrapper(func):
func_name = func.__name__
renamed_method(func_name, supported_name)
@wraps(func)
def inner(*args, **kwargs):
warn("adapter:{}".format(func_name))
warn_or_error(
AdapterDeprecationWarning(old_name=func_name, new_name=supported_name)
)
return func(*args, **kwargs)
if parse_replacement:

View File

@@ -1,20 +1,10 @@
from typing import List, Optional, Type
from pathlib import Path
from dbt.adapters.base import Credentials
from dbt.exceptions import CompilationError
from dbt.adapters.protocol import AdapterProtocol
def project_name_from_path(include_path: str) -> str:
# avoid an import cycle
from dbt.config.project import PartialProject
partial = PartialProject.from_project_root(include_path)
if partial.project_name is None:
raise CompilationError(f"Invalid project at {include_path}: name not set!")
return partial.project_name
class AdapterPlugin:
"""Defines the basic requirements for a dbt adapter plugin.
@@ -29,12 +19,13 @@ class AdapterPlugin:
credentials: Type[Credentials],
include_path: str,
dependencies: Optional[List[str]] = None,
project_name: Optional[str] = None,
) -> None:
self.adapter: Type[AdapterProtocol] = adapter
self.credentials: Type[Credentials] = credentials
self.include_path: str = include_path
self.project_name: str = project_name_from_path(include_path)
self.project_name: str = project_name or f"dbt_{Path(include_path).name}"
self.dependencies: List[str]
if dependencies is None:
self.dependencies = []

View File

@@ -1,21 +1,20 @@
from threading import local
from typing import Optional, Callable, Dict, Any
from dbt.clients.jinja import QueryStringGenerator
from dbt.adapters.clients.jinja import QueryStringGenerator
from dbt.context.manifest import generate_query_header_context
from dbt.contracts.connection import AdapterRequiredConfig, QueryComment
from dbt.contracts.graph.nodes import ResultNode
from dbt.adapters.contracts.connection import AdapterRequiredConfig, QueryComment
from dbt.contracts.graph.manifest import Manifest
from dbt.exceptions import DbtRuntimeError
from dbt.common.exceptions import DbtRuntimeError
class NodeWrapper:
def __init__(self, node) -> None:
self._inner_node = node
class QueryHeaderContextWrapper:
def __init__(self, context) -> None:
self._inner_context = context
def __getattr__(self, name):
return getattr(self._inner_node, name, "")
return getattr(self._inner_context, name, "")
class _QueryComment(local):
@@ -53,7 +52,7 @@ class _QueryComment(local):
self.append = append
QueryStringFunc = Callable[[str, Optional[NodeWrapper]], str]
QueryStringFunc = Callable[[str, Optional[QueryHeaderContextWrapper]], str]
class MacroQueryStringSetter:
@@ -90,10 +89,10 @@ class MacroQueryStringSetter:
def reset(self):
self.set("master", None)
def set(self, name: str, node: Optional[ResultNode]):
wrapped: Optional[NodeWrapper] = None
if node is not None:
wrapped = NodeWrapper(node)
def set(self, name: str, query_header_context: Any):
wrapped: Optional[QueryHeaderContextWrapper] = None
if query_header_context is not None:
wrapped = QueryHeaderContextWrapper(query_header_context)
comment_str = self.generator(name, wrapped)
append = False

View File

@@ -2,8 +2,8 @@ from collections.abc import Hashable
from dataclasses import dataclass, field
from typing import Optional, TypeVar, Any, Type, Dict, Iterator, Tuple, Set, Union, FrozenSet
from dbt.contracts.graph.nodes import SourceDefinition, ManifestNode, ResultNode, ParsedNode
from dbt.contracts.relation import (
from dbt.adapters.contracts.relation import (
RelationConfig,
RelationType,
ComponentName,
HasQuoting,
@@ -11,15 +11,11 @@ from dbt.contracts.relation import (
Policy,
Path,
)
from dbt.exceptions import (
ApproximateMatchError,
DbtInternalError,
MultipleDatabasesNotAllowedError,
)
from dbt.node_types import NodeType
from dbt.utils import filter_null_values, deep_merge, classproperty
from dbt.adapters.exceptions import MultipleDatabasesNotAllowedError, ApproximateMatchError
from dbt.common.utils import filter_null_values, deep_merge
from dbt.adapters.utils import classproperty
import dbt.exceptions
import dbt.common.exceptions
Self = TypeVar("Self", bound="BaseRelation")
@@ -41,13 +37,13 @@ class BaseRelation(FakeAPIObject, Hashable):
# adding a relation type here also requires defining the associated rename macro
# e.g. adding RelationType.View in dbt-postgres requires that you define:
# include/postgres/macros/relations/view/rename.sql::postgres__get_rename_view_sql()
renameable_relations: SerializableIterable = field(default_factory=frozenset)
renameable_relations: SerializableIterable = ()
# register relation types that are atomically replaceable, e.g. they have "create or replace" syntax
# adding a relation type here also requires defining the associated replace macro
# e.g. adding RelationType.View in dbt-postgres requires that you define:
# include/postgres/macros/relations/view/replace.sql::postgres__get_replace_view_sql()
replaceable_relations: SerializableIterable = field(default_factory=frozenset)
replaceable_relations: SerializableIterable = ()
def _is_exactish_match(self, field: ComponentName, value: str) -> bool:
if self.dbt_created and self.quote_policy.get_part(field) is False:
@@ -100,7 +96,7 @@ class BaseRelation(FakeAPIObject, Hashable):
if not search:
# nothing was passed in
raise dbt.exceptions.DbtRuntimeError(
raise dbt.common.exceptions.DbtRuntimeError(
"Tried to match relation, but no search path was passed!"
)
@@ -200,83 +196,50 @@ class BaseRelation(FakeAPIObject, Hashable):
identifier=identifier,
)
@classmethod
def create_from_source(cls: Type[Self], source: SourceDefinition, **kwargs: Any) -> Self:
source_quoting = source.quoting.to_dict(omit_none=True)
source_quoting.pop("column", None)
quote_policy = deep_merge(
cls.get_default_quote_policy().to_dict(omit_none=True),
source_quoting,
kwargs.get("quote_policy", {}),
)
return cls.create(
database=source.database,
schema=source.schema,
identifier=source.identifier,
quote_policy=quote_policy,
**kwargs,
)
@staticmethod
def add_ephemeral_prefix(name: str):
return f"__dbt__cte__{name}"
@classmethod
def create_ephemeral_from_node(
def create_ephemeral_from(
cls: Type[Self],
config: HasQuoting,
node: ManifestNode,
relation_config: RelationConfig,
) -> Self:
# Note that ephemeral models are based on the name.
identifier = cls.add_ephemeral_prefix(node.name)
identifier = cls.add_ephemeral_prefix(relation_config.name)
return cls.create(
type=cls.CTE,
identifier=identifier,
).quote(identifier=False)
@classmethod
def create_from_node(
def create_from(
cls: Type[Self],
config: HasQuoting,
node,
quote_policy: Optional[Dict[str, bool]] = None,
quoting: HasQuoting,
relation_config: RelationConfig,
**kwargs: Any,
) -> Self:
if quote_policy is None:
quote_policy = {}
quote_policy = kwargs.pop("quote_policy", {})
quote_policy = dbt.utils.merge(config.quoting, quote_policy)
config_quoting = relation_config.quoting_dict
config_quoting.pop("column", None)
# precedence: kwargs quoting > relation config quoting > base quoting > default quoting
quote_policy = deep_merge(
cls.get_default_quote_policy().to_dict(omit_none=True),
quoting.quoting,
config_quoting,
quote_policy,
)
return cls.create(
database=node.database,
schema=node.schema,
identifier=node.alias,
database=relation_config.database,
schema=relation_config.schema,
identifier=relation_config.identifier,
quote_policy=quote_policy,
**kwargs,
)
@classmethod
def create_from(
cls: Type[Self],
config: HasQuoting,
node: ResultNode,
**kwargs: Any,
) -> Self:
if node.resource_type == NodeType.Source:
if not isinstance(node, SourceDefinition):
raise DbtInternalError(
"type mismatch, expected SourceDefinition but got {}".format(type(node))
)
return cls.create_from_source(node, **kwargs)
else:
# Can't use ManifestNode here because of parameterized generics
if not isinstance(node, (ParsedNode)):
raise DbtInternalError(
f"type mismatch, expected ManifestNode but got {type(node)}"
)
return cls.create_from_node(config, node, **kwargs)
@classmethod
def create(
cls: Type[Self],
@@ -386,7 +349,7 @@ class InformationSchema(BaseRelation):
def __post_init__(self):
if not isinstance(self.information_schema_view, (type(None), str)):
raise dbt.exceptions.CompilationError(
raise dbt.common.exceptions.CompilationError(
"Got an invalid name: {}".format(self.information_schema_view)
)

View File

@@ -7,17 +7,16 @@ from dbt.adapters.reference_keys import (
_make_ref_key_dict,
_ReferenceKey,
)
from dbt.exceptions import (
DependentLinkNotCachedError,
from dbt.common.exceptions.cache import (
NewNameAlreadyInCacheError,
NoneRelationFoundError,
ReferencedLinkNotCachedError,
DependentLinkNotCachedError,
TruncatedModelNameCausedCollisionError,
NoneRelationFoundError,
)
from dbt.events.functions import fire_event, fire_event_if
from dbt.events.types import CacheAction, CacheDumpGraph
from dbt.flags import get_flags
from dbt.utils import lowercase
from dbt.common.events.functions import fire_event, fire_event_if
from dbt.adapters.events.types import CacheAction, CacheDumpGraph
from dbt.common.utils.formatting import lowercase
def dot_separated(key: _ReferenceKey) -> str:
@@ -165,10 +164,11 @@ class RelationsCache:
:attr Set[str] schemas: The set of known/cached schemas, all lowercased.
"""
def __init__(self) -> None:
def __init__(self, log_cache_events: bool = False) -> None:
self.relations: Dict[_ReferenceKey, _CachedRelation] = {}
self.lock = threading.RLock()
self.schemas: Set[Tuple[Optional[str], Optional[str]]] = set()
self.log_cache_events = log_cache_events
def add_schema(
self,
@@ -318,10 +318,9 @@ class RelationsCache:
:param BaseRelation relation: The underlying relation.
"""
flags = get_flags()
cached = _CachedRelation(relation)
fire_event_if(
flags.LOG_CACHE_EVENTS,
self.log_cache_events,
lambda: CacheDumpGraph(before_after="before", action="adding", dump=self.dump_graph()),
)
fire_event(CacheAction(action="add_relation", ref_key=_make_ref_key_dict(cached)))
@@ -329,7 +328,7 @@ class RelationsCache:
with self.lock:
self._setdefault(cached)
fire_event_if(
flags.LOG_CACHE_EVENTS,
self.log_cache_events,
lambda: CacheDumpGraph(before_after="after", action="adding", dump=self.dump_graph()),
)
@@ -454,9 +453,8 @@ class RelationsCache:
ref_key_2=new_key._asdict(),
)
)
flags = get_flags()
fire_event_if(
flags.LOG_CACHE_EVENTS,
self.log_cache_events,
lambda: CacheDumpGraph(before_after="before", action="rename", dump=self.dump_graph()),
)
@@ -467,7 +465,7 @@ class RelationsCache:
self._setdefault(_CachedRelation(new))
fire_event_if(
flags.LOG_CACHE_EVENTS,
self.log_cache_events,
lambda: CacheDumpGraph(before_after="after", action="rename", dump=self.dump_graph()),
)

View File

@@ -4,7 +4,7 @@ from typing import Optional, DefaultDict, Mapping
class Capability(str, Enum):
"""Enumeration of optional adapter features which can be probed using BaseAdapter.has_feature()"""
"""Enumeration of optional adapter features which can be probed using BaseAdapter.capabilities()"""
SchemaMetadataByRelations = "SchemaMetadataByRelations"
"""Indicates efficient support for retrieving schema metadata for a list of relations, rather than always retrieving

Some files were not shown because too many files have changed in this diff Show More