Compare commits

...

294 Commits

Author SHA1 Message Date
Michelle Ark
d16988c9bf finish rename 2023-12-08 09:30:50 +09:00
Michelle Ark
8973f99eb9 replace MacroResolver with MacroClient 2023-12-08 01:40:14 +09:00
Michelle Ark
bb1237505c changelog entry 2023-12-08 00:49:26 +09:00
Michelle Ark
325717f642 fix adapter test setup 2023-12-08 00:39:43 +09:00
Michelle Ark
f61da7bfe6 add macro_context_generator on adapter 2023-12-08 00:26:18 +09:00
Michelle Ark
538481af61 fix adapter.calculate_freshness call 2023-12-07 21:49:03 +09:00
Michelle Ark
3620ce680a changelog entry 2023-12-07 11:15:59 +09:00
Michelle Ark
7b1e3323df pass MacroResolverProtcol in adapter.calculate_freshness_from_metadata 2023-12-07 11:11:58 +09:00
Michelle Ark
a70edd6a09 Merge branch 'feature/decouple-adapters-from-core' into macro-resolver-remove-lazy-loading2 2023-12-07 11:09:16 +09:00
Michelle Ark
26ddaaf51f Remove manifest from catalog and connection method signatures (#9242) 2023-12-07 11:05:27 +09:00
Michelle Ark
3e5515bb51 rename to MacroResolverProtocol 2023-12-07 11:03:21 +09:00
Michelle Ark
4dec100b58 remove manifest from adapter.execute_macro, replace with MacroResolver + remove lazy loading 2023-12-07 10:49:01 +09:00
colin-rogers-dbt
f1c2f06d1e Move BaseConfig to Common (#9224)
* moving types_pb2.py to common/events

* move BaseConfig and assorted dependencies to common

* move ShowBehavior and OnConfigurationChange to common

* add changie
2023-12-06 17:25:22 -08:00
Michelle Ark
c2734c503a Merge branch 'main' into feature/decouple-adapters-from-core 2023-12-07 09:00:03 +09:00
colin-rogers-dbt
45b35701d4 Merge main into feature/decouple-adapters-from-core (#9240)
* moving types_pb2.py to common/events

* Restore warning on unpinned git packages (#9157)

* Support --empty flag for schema-only dry runs (#8971)

* Fix ensuring we produce valid jsonschema artifacts for manifest, catalog, sources, and run-results (#9155)

* Drop `all_refs=True` from jsonschema-ization build process

Passing `all_refs=True` makes it so that Everything is a ref, even
the top level schema. In jsonschema land, this essentially makes the
produced artifact not a full schema, but a fractal object to be included
in a schema. Thus when `$id` is passed in, jsonschema tools blow up
because `$id` is for identifying a schema, which we explicitly weren't
creating. The alternative was to drop the inclusion of `$id`. Howver, we're
intending to create a schema, and having an `$id` is recommended best
practice. Additionally since we were intending to create a schema,
not a fractal, it seemed best to create to full schema.

* Explicity produce jsonschemas using DRAFT_2020_12 dialect

Previously were were implicitly using the `DRAFT_2020_12` dialect through
mashumaro. It felt wise to begin explicitly specifying this. First, it
is closest in available mashumaro provided dialects to what we produced
pre 1.7. Secondly, if mashumaro changes its default for whatever reason
(say a new dialect is added, and mashumaro moves to that), we don't want
to automatically inherit that.

* Bump manifest version to v12

Core 1.7 released with manifest v11, and we don't want to be overriding
that with 1.8. It'd be weird for 1.7 and 1.8 to both have v11 manifests,
but for them to be different, right?

* Begin including schema dialect specification in produced jsonschema

In jsonschema's documentation they state
> It's not always easy to tell which draft a JSON Schema is using.
> You can use the $schema keyword to declare which version of the JSON Schema specification the schema is written to.
> It's generally good practice to include it, though it is not required.

and

> For brevity, the $schema keyword isn't included in most of the examples in this book, but it should always be used in the real world.

Basically, to know how to parse a schema, it's important to include what
schema dialect is being used for the schema specification. The change in
this commit ensures we include that information.

* Create manifest v12 jsonschema specification

* Add change documentation for jsonschema schema production fix

* Bump run-results version to v6

* Generate new v6 run-results jsonschema

* Regenerate catalog v1 and sources v3 with fixed jsonschema production

* Update tests to handle bumped versions of manifest and run-results

---------

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
2023-12-06 15:46:21 -08:00
Michelle Ark
eb96e3deec Add RelationConfig Protocol for use in Relation.create_from (#9210)
* move relation contract to dbt.adapters

* changelog entry

* first pass: clean up relation.create_from

* type ignores

* type ignore

* changelog entry

* update RelationConfig variable names
2023-12-06 10:46:45 -08:00
Michelle Ark
ed8f5d38e4 Remove ResultNode usage from connections (#9211) 2023-12-06 11:04:26 +09:00
Quigley Malcolm
0ab954e1af Fix ensuring we produce valid jsonschema artifacts for manifest, catalog, sources, and run-results (#9155)
* Drop `all_refs=True` from jsonschema-ization build process

Passing `all_refs=True` makes it so that Everything is a ref, even
the top level schema. In jsonschema land, this essentially makes the
produced artifact not a full schema, but a fractal object to be included
in a schema. Thus when `$id` is passed in, jsonschema tools blow up
because `$id` is for identifying a schema, which we explicitly weren't
creating. The alternative was to drop the inclusion of `$id`. Howver, we're
intending to create a schema, and having an `$id` is recommended best
practice. Additionally since we were intending to create a schema,
not a fractal, it seemed best to create to full schema.

* Explicity produce jsonschemas using DRAFT_2020_12 dialect

Previously were were implicitly using the `DRAFT_2020_12` dialect through
mashumaro. It felt wise to begin explicitly specifying this. First, it
is closest in available mashumaro provided dialects to what we produced
pre 1.7. Secondly, if mashumaro changes its default for whatever reason
(say a new dialect is added, and mashumaro moves to that), we don't want
to automatically inherit that.

* Bump manifest version to v12

Core 1.7 released with manifest v11, and we don't want to be overriding
that with 1.8. It'd be weird for 1.7 and 1.8 to both have v11 manifests,
but for them to be different, right?

* Begin including schema dialect specification in produced jsonschema

In jsonschema's documentation they state
> It's not always easy to tell which draft a JSON Schema is using.
> You can use the $schema keyword to declare which version of the JSON Schema specification the schema is written to.
> It's generally good practice to include it, though it is not required.

and

> For brevity, the $schema keyword isn't included in most of the examples in this book, but it should always be used in the real world.

Basically, to know how to parse a schema, it's important to include what
schema dialect is being used for the schema specification. The change in
this commit ensures we include that information.

* Create manifest v12 jsonschema specification

* Add change documentation for jsonschema schema production fix

* Bump run-results version to v6

* Generate new v6 run-results jsonschema

* Regenerate catalog v1 and sources v3 with fixed jsonschema production

* Update tests to handle bumped versions of manifest and run-results
2023-12-05 17:36:43 -08:00
Michelle Ark
e01eb30884 Remove usage of dbt.contracts.relation in dbt/adapters (#9207) 2023-12-06 10:34:56 +09:00
Michelle Ark
00f49206e9 Decouple macro generator from adapters (#9149) 2023-12-05 10:51:51 +09:00
Michelle Ark
1bca662883 first pass: adapter migration script (#9160) 2023-12-05 08:41:11 +09:00
colin-rogers-dbt
41ac915949 Refactor EventManager setup and interaction (#9180)
* moving types_pb2.py to common/events

* move event manager setup back to core, remove ref to global EVENT_MANAGER and clean up event manager functions

* move invocation_id from events to first class common concept

* move lowercase utils to common

* move lowercase utils to common

* ref CAPTURE_STREAM through method

* add changie
2023-12-01 08:48:19 -08:00
Mila Page
373125ecb8 Move the semver package to common and alter references. (#9166)
* Move the semver package to common and alter references.

* Alter leftover references to dbt.semver, this time using from syntax.

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
2023-11-29 11:03:13 -08:00
Michelle Ark
5488dfb992 Support --empty flag for schema-only dry runs (#8971) 2023-11-29 11:06:06 -05:00
Jeremy Cohen
09355701f6 Restore warning on unpinned git packages (#9157) 2023-11-29 08:40:47 +01:00
Michelle Ark
294ad82e50 delete accidentally merged types_pb2.py 2023-11-28 20:20:59 -05:00
Michelle Ark
12bd1e87fb Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-28 20:08:33 -05:00
colin-rogers-dbt
8bad75c65b Move adapter logger to adapters (#9165)
* moving types_pb2.py to common/events

* Move AdapterLogger to adapter folder

* add changie
2023-11-28 15:05:30 -08:00
Kshitij Aranke
c836b7585e Fix #9119: Get sources working again in dbt docs generate (#9159) 2023-11-28 18:21:44 +00:00
Michelle Ark
220f56d8d2 remove adapter.get_compiler (#9134) 2023-11-28 09:27:00 -05:00
Jeremy Cohen
32fde75504 Fixups for deps lock file (#9147)
* Update git revision with commit SHA

* Use PackageRenderer for lock file

* add unit tests for git and tarball packages

* deepcopy unrendered_packages_data before iteration, fix remaining tests

* Add functional tests

* Add changelog entries

* Assert one more

---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2023-11-28 10:06:05 +01:00
Michelle Ark
615ad1fe2d move include/global_project to adapters (#8930) 2023-11-27 18:14:02 -05:00
Mike Alfare
81236a3dca Fix test_current_timestamp_matches_utc (#9058)
* loosen the tolerance for matching timestamps due to a slow system clock on the macos runner
* changelog
2023-11-27 13:15:20 -05:00
Michelle Ark
6d834a18ed clean up plugin readmes (#9130) 2023-11-27 11:10:26 -05:00
Peter Webb
2ab0f7b26b Decouple adapter constraints from core (#9054)
* Move constraints to dbt.common

* Move constraints to contracts folder, per review

* Add a changelog entry.
2023-11-21 12:16:39 -05:00
Peter Webb
9bb970e6ef Cache plugin modules (#9031)
* Cache plugin modules

* Add changelog entry

* Use lru_cache to keep Python 3.8 happy.
2023-11-20 17:50:16 -05:00
Michelle Ark
e56a5dae8b Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific (#9051) 2023-11-20 17:26:11 -05:00
Jeremy Cohen
1c9cec1787 Roadmap update (Nov 2023) (#9120)
* Roadmap update (Nov 2023)

* fix typos

* Write out title

---------

Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
2023-11-18 14:54:03 -05:00
Quigley Malcolm
4d02ef637b Update comment about input_measures setting in metric parsing (#9118)
In [dbt-labs/dbt-core#7984](https://github.com/dbt-labs/dbt-core/pull/7984)
we began setting a metrics `type_params.input_measures` during metric
processing post parsing. However in that PR we didn't clean up the comment
in the parser about setting `input_measures`. This is that post fact
cleanup.
2023-11-17 12:24:58 -08:00
Adam Lopez
19f027b7a7 import nested tarball package dependencies (#8855)
* detected nested dependencies in tarball dependency

* rm git changes

* add changelog entry

* update tarball to use download_and_untar

* update tests + handle package lock entry creation for tarball

* use fixes from 017faf4bd1, add download_and_untar in install step

* quote version value

* Update .changes/unreleased/Fixes-20231013-130943.yaml

Co-authored-by: Jeremy Cohen <jtcohen6@gmail.com>

* Update .changes/unreleased/Fixes-20231013-130943.yaml

---------

Co-authored-by: Emily Rockman <ebuschang@gmail.com>
Co-authored-by: Jeremy Cohen <jtcohen6@gmail.com>
2023-11-16 12:38:47 -06:00
Michelle Ark
1d0a3e92c8 Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-16 11:28:51 -05:00
Quigley Malcolm
ab90c777d0 Add test asserting GraphRunnableTasks attempt to cancel connections on SystemExit (#9101)
* Add test asserting GraphRunnableTasks attempt to cancel connections on SystemExit

* Add test asserting GraphRunnableTasks attempt to cancel connections on KeyboardInterrupt

* Add test asserting GraphRunnableNode doesn't try to cancel connections on generic Exception
2023-11-15 16:03:43 -08:00
Michelle Ark
3902137dfc test pinning ddtrace (#9090) 2023-11-15 17:18:56 -05:00
Michelle Ark
0131feac68 document scripts/collect-artifact-schema (#9085) 2023-11-15 16:01:30 -05:00
Quigley Malcolm
017faf4bd1 Fix dbt deps failing on tarballs (#9068)
* tarball lockfile fix

* Add changie doc for tarball deps issue

* Add integration test for ensuring tarball package specification works

This test was written _after_ the fix was commited. However, I ran this
test against main without the fix and it failed. After running the test
with the tarball fix, it passed.

* Remove unnecessary `tarball` conditional logic in `PackageConfig.validate`

We had a conditional to skip validation for a package if the package
included the `tarball` key. However, this conditional always returned
false as it was nested inside a conditional that the package had the
default `package` key, which means it's not a tarball package, but a
package package (maybe we need better differentiation here). If we need
additional validation for tarballs down the road, we should do that one
level up. At this time we have no additional validaitons to add.

* Fix typos in changie doc for tarball deps issue

* Improve tarball package test naming and add related unhappy path test

* Remove unnecessary `setUp` fixture from tarball package tests

We initially included this fixture due to copy and pasting another
test. However, this `setUp` fixture isn't actually necessary for the
tarball dependency tests.

---------

Co-authored-by: Chenyu Li <chenyu.li@dbtlabs.com>
2023-11-14 11:38:40 -08:00
Quigley Malcolm
c2f7d75e9e Support hierarchical config setting for SavedQueryExport configs (#9065)
* Add test asserting `SavedQuery` configs can be set from `dbt_project.yml`

* Allow extraneous properties in Export configs

This brings the Export config object more in line with how other config
objects are specified in the unparsed definition. It allows for specifying
of extra configs, although they won't get propagate to the final config.

* Add `ExportConfig` options to `SavedQueryConfig` options

This allows for specifying `ExportConfig` options at the `SavedQueryConfig` level.
This also therefore allows these options to be specified in the dbt_project.yml
config. The plan in the follow up commit is to merge the `SavedQueryConfig` options
into all configs of `Exports` belonging to the saved query.

There are a couple caveots to call out:
1. We've used `schema` instead of `schema_name` on the `SavedQueryConfig` despite
it being called `schema_name` on the `ExportConfig`. This is because need `schema_name`
to be the name of the property on the `ExportConfig`, but `schema` is the user facing
specification.
2. We didn't add the `ExportConfig` `alias` property to the `SavedQueryConfig` This
is because `alias` will always be specific to a single export, and thus it doesn't
make sense to allow defining it on the `SavedQueryConfig` to then apply to all
`Exports` belonging to the `SavedQuery`

* Begin inheriting configs from saved query config, and transitively from project config

Export configs will now inherit from saved query configs, with a preference
for export config specifications. That is to say an export config will inherity
a config attr from the saved query config only if a value hasn't been supplied
on the export config directly. Additionally because the saved query config has
a similar relationship with the project config, exports configs can inherit
from the project config (again with a preference for export config specifications).

* Correct conditional in export config building for map schema to schema_name

I somehow wrote a really weird, but also valid, conditional statement. Previously
the conditional was
```
if combined.get("schema") is not combined.get("schema_name") is None:
```
which basically checked whether `schema` was a boolean that didn't match
the boolean of whether `schema_name` was None. This would pretty much
always evaluate to True because `schema` should be a string or none, not
a bool, and thus would never match the right hand side. Crazy. It has now
been fixed to do the thing we want to it to do. If `schema` isn't `None`,
and `schema_name` is `None`, then set `schema_name` to have the value of
`schema`.

* Update parameter names in `_get_export_config` to be more verbose
2023-11-14 11:13:41 -08:00
colin-rogers-dbt
51b94b26cc Refactor Base Exceptions (#8989)
* moving types_pb2.py to common/events

* Refactor Base Exceptions

* update make_log_dir_if_missing to handle str

* move remaining adapters exception imports to common/adapters
---------

Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
2023-11-10 09:54:36 -05:00
colin-rogers-dbt
e24f9b3da7 Revert "[Feature] Supports non half width alphanumeric for generic test (#8203)" (#9048)
This reverts commit f45b013321.
2023-11-09 15:05:12 -08:00
Mike Alfare
b58e8e3ffc Migrate TestCatalogRelationTypes to the shared adapter tests framework (#8952)
* changelog
* move test to shared adapter zone for reuse
* add comment explaining how to configure this test case
2023-11-09 14:57:45 -05:00
d-kaneshiro
f45b013321 [Feature] Supports non half width alphanumeric for generic test (#8203)
* Supports non half width alphanumeric for generic test

* Unify conditional statements

* add CHANGELOG entries

* added test for Japanese

* Move the fix further upstream

* Remove the changes in core/dbt/task/runnable.py

* Fix accidental removal of `_` substitution character

---------

Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
2023-11-09 10:50:59 -07:00
Thomas Lento
e547c0ec64 Update typing-extensions version to >= 4.4 (#9012) 2023-11-08 16:19:28 -05:00
colin-rogers-dbt
6871fc46b5 Add drop_schema_named macro (#8868)
* add drop_schema_named

* give drop_schema_named a default
2023-11-08 12:50:25 -08:00
Ben Mosher
931b2dbe40 During node execution, also treat SystemExit as an interrupt. (#8994)
IDE worker process raises SystemExit in multiple scenarios, including user abort of a command.
2023-11-08 15:34:18 -05:00
Leo Griffiths
bb35b3eb87 fix: get_custom_database docs describe postfix when default behaviour… (#9002)
* fix: get_custom_database docs describe postfix when default behaviour is to take the custom database value directly, if provided

* Changelog entry
2023-11-07 12:34:38 -07:00
Chenyu Li
01d481bc8d Fix lock for git subduer (#9019)
* wip

* add tests

* changelog

* nits

* pr feedback

* nits
2023-11-07 13:58:56 -05:00
Gerda Shank
46b9a1d621 Support new agate Integer data_type in adapter code (#9004)
Add test for empty seed file with only headers
2023-11-07 13:43:25 -05:00
Mike Alfare
839c720e91 Use MANIFEST.in to identify package data, allows recursive include (#9021)
* changelog

* use MANIFEST.in to identify package data
2023-11-07 13:21:15 -05:00
Quigley Malcolm
d88c6987a2 Remove documentable as it isn't used anywhere (#9008) 2023-11-07 09:12:36 -08:00
Michelle Ark
4ee950427a Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-07 11:38:24 -05:00
Jeremy Cohen
6c1822f186 Fix back compat for run_results pre-v5 (#9009)
* Fix back compat for run_results pre-v5

* Add type annotations

* Add functional testing

* Add inline annotations

* Add changelog entry.

* Consolidate upgrade_schema_version + upgrade_run_results_json

* Restore accidentally reverted test cases

* Pre-commit fixups

---------

Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
2023-11-06 16:55:36 -05:00
Anders
c7c3ac872c Update capability.py (#8842) 2023-11-06 10:42:52 -05:00
Michelle Ark
c4ff280436 remove dbt.flags.LOG_CACHE_EVENTS usage in dbt/adapters (#8933) 2023-11-01 18:44:59 -04:00
Peter Webb
7fddd6e448 Make relation filtering None-tolerant for maximal flexibility across adapters. (#8975) 2023-11-01 16:25:27 -04:00
Michelle Ark
1260782bd2 remove dbt.flags.MP_CONTEXT usage in dbt/adapters (#8931) 2023-11-01 16:13:39 -04:00
Michelle Ark
333120b111 Merge branch 'main' into feature/decouple-adapters-from-core 2023-11-01 12:00:17 -04:00
Mike Alfare
bb21403c9e ADAP-974: Fix issue where materialized views were not showing up in catalog queries (#8945)
* changelog
* write test case demonstrating the issue
* update catalog query to reflect materialized views
2023-10-31 23:56:59 -04:00
Quigley Malcolm
ac972948b8 DSI 0.4.0 and Saved Query Exports (#8950) 2023-10-31 18:34:41 -07:00
Chenyu Li
211392c4a4 add a no-op runner for saved_query (#8937) 2023-10-31 16:43:33 -07:00
Peter Webb
7317de23a3 Fix cased comparison in catalog-retrieval function (#8940)
* Fix cased comparison in catalog-retrieval function.

* Fix cased comparison in catalog-retrieval function.
2023-10-30 11:23:41 -04:00
Michelle Ark
af916666a2 move agate_helper into common (#8911)
* move agate_helper into common

* add changie

---------

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
2023-10-26 16:22:04 -04:00
Michelle Ark
7de8930d1d move agate_helper unit tests under tests/unit/common 2023-10-26 16:19:13 -04:00
Michelle Ark
200bcdcd9f Merge branch 'move-agate-client-to-common' into feature/decouple-adapters-from-core 2023-10-26 16:18:30 -04:00
Michelle Ark
b9a603e3aa remove usage of dbt.config.PartialProject from dbt/adapters (#8909)
* remove usage of dbt.config.PartialProject from dbt/adapters

* add changie

---------

Co-authored-by: Colin <colin.rogers@dbtlabs.com>
2023-10-26 16:04:39 -04:00
colin-rogers-dbt
1a825484fb Add utils module (#8910)
* moving types_pb2.py to common/events

* split out utils into core/common/adapters

* add changie
2023-10-26 15:59:07 -04:00
Peter Webb
a2a7b7d795 Fix issues around new get_catalog_by_relations macro (#8856)
* Fix issues around new get_catalog_by_relations macro

* Add changelog entry

* Fix unit test.

* Additional unit testing
2023-10-26 15:23:04 -04:00
Kshitij Aranke
4122f6c308 Fix #8836: Add version to fqn when version==0 (#8915) 2023-10-26 11:31:24 -05:00
Doug Beatty
6aeebc4c76 Handle unknown type_code for model contracts (#8887)
* Handle unknown `type_code` for model contracts

* Changelog entry

* Fix changelog entry

* Functional test for a `type_code` that is not recognized by psycopg2

* Functional tests for data type mismatches
2023-10-25 15:44:02 -06:00
Michelle Ark
f44d704801 move agate_helper into common 2023-10-25 17:05:08 -04:00
Michelle Ark
dbd02e54c2 move types_pb2.py from events to common/events 2023-10-25 15:46:41 -04:00
Michelle Ark
a89642a6f9 fix import 2023-10-25 15:04:17 -04:00
colin-rogers-dbt
c141148616 Move events to common (#8676)
* Move events to common

* More Type Annotations (#8536)

* Extend use of type annotations in the events module.

* Add return type of None to more __init__ definitions.

* Still more type annotations adding -> None to __init__

* Tweak per review

* Allow adapters to include python package logging in dbt logs (#8643)

* add set_package_log_level functionality

* set package handler

* set package handler

* add logging about stting up logging

* test event log handler

* add event log handler

* add event log level

* rename package and add unit tests

* revert logfile config change

* cleanup and add code comments

* add changie

* swap function for dict

* add additional unit tests

* fix unit test

* update README and protos

* fix formatting

* update precommit

---------

Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
2023-10-25 14:57:02 -04:00
Michelle Ark
469a9aca06 remove dbt.contracts.connection imports from adapter module 2023-10-25 14:56:55 -04:00
Emily Rockman
98310b6612 Contract enforcement on temporary tables (#8889)
* add test

* fix test

* first pass with constraint error

* add back column checks for temp tables

* changelog

* Update .changes/unreleased/Fixes-20231024-145504.yaml
2023-10-25 11:10:36 -05:00
Emily Rockman
ef9d6a870f add 1.7 changelog (#8886) 2023-10-24 11:51:28 -05:00
Chenyu Li
35f46dac8c Fix partial parsing issue not working for changing semantic model name (#8865)
* fix

* test

* changelog
2023-10-23 09:55:58 -07:00
FishtownBuildBot
efa6339e18 Cleanup main after cutting new 1.7.latest branch (#8840) 2023-10-12 16:46:37 +01:00
FishtownBuildBot
1baebb423c [Automated] Merged prep-release/1.7.0rc1_6495873283 into target main during release process 2023-10-12 09:50:47 -04:00
Github Build Bot
462df8395e Bumping version to 1.7.0rc1 and generate changelog 2023-10-12 12:53:26 +00:00
Quigley Malcolm
35f214d9db Support dbt-semantic-interfaces 0.3.0 (#8820)
* changie doc for DSI 0.3.0 upgrade

* Gracefully handle v10 metric filters

* Fix iteration over metrics in `upgrade_v10_metric_filters`

* Update previous manifest version test fixtures to have more expressive metrics

* Regenerate the test v10 manifest artifact using the more expressive metrics from 904cc1ef

To do this I cherry-picked 904cc1ef onto my local 1.6.latest branch,
had the test regenerate the test v10 manifest artifact, and then over
wrote the test v10 manifest artifact on this branch (cherry-picking it
across the branches didn't work, had to copy paste :grimmace:)

* Regenerate test v11 manifest artifact using the fixture changes in 904cc1ef

* Update `upgrade_v10_metric_filters` to handled disabled metrics

Regenerating the v10 and v11 test manifest artifacts uncovered an
issue wherein we weren't handling disabled metrics that need to
get upgraded. This commit fixes that. Additionally, the
`upgrade_v10_metric_filters` was getting a bit unwieldy, so I broke
extracted the abstracted sub functions.

* Fix `test_backwards_compatible_versions` test

When we regenerated the v10 test manifest artifact, it started having
the `metricflow_time_sine` model, and it didn't previously. This caused
`test_backwards_compatible_versions` to start failing because it was
no longer identified as having modified state for v10. The test has
been altered accordingly
2023-10-11 16:46:26 -07:00
Quigley Malcolm
af0cbcb6a5 Add SavedQuery nodes (#8798)
* Bump to dbt-semantic-interfaces 0.3.0b1

* Update import path of `WhereFilterParser` from `dbt-semantic-interfaces`

In 0.3.x of `dbt-semantic-intefaces` the location of the WhereFilterParser
moved to be grouped in with a bunch of new adjacent code. As such,
we needed to correct our import path of it.

* Create basic `SavedQuery` node type based on `SavedQuery` protocol from DSI

* Add ability to add SavedQueries to the manifest

* Define unparsed SavedQuery node

* Begin parsing saved_query objects to manifest

* Skip jinja rendering of `SavedQuery.where` property

* Begin propagating `SavedQueries` on the manifest to the semantic manifest

* Add tests for basic saved query parsing

* Add custom pluralization handling of SavedQuery node type

* Add a config subclass to SavedQuery node

* Move the SavedQuery node to nodes.py

Unfortunately things are a bit too intertwined currently for SavedQuery
to be in it's own file. We need to add the SavedQuery node to the
GraphMemberNode, unfortunately with SavedQuery in it's own file,
importing it would have caused a circular dependency. We'll need
to separately come in and split things up as a cleanup portion of
work.

* Add basic plumbing of saved query configs to projects

* Add basic lookup utility for saved queries, SavedQueryLookup

* Handle disabled SavedQuery nodes in parsing and lookups

* Add SavedQuery nodes to grouping process

Our grouping logic seems to be in a weird spot. It seems liek we're
moving to setting the `group` for a node in the node's `config` however,
all of the logic around grouping is still focused on the top level `group`
property on a nodes. To get group stuff plumbed I've thus added `group`
as a top level property of the `SavedQuery` node, and populated it from
the config group value.

* Plumb through saved query in a lot more places

I don't like making scatter shot commits like this. However, a lot
of this commit was written ~4am, soooo yea. Things were broken, I wanted
things to be unbroken. I mostly searched for `semantic_models` and added
the equivalent necessary `saved_queries`. Some stuff is in support of
writing out the manifest, some stuff helps with node selection, it's a
lot of miscelaneous stuff that I don't fully understand.

* Add `depends_on` to `SavedQuery` nodes and populate from `metrics` property

* Add partial parsing support to SavedQuery nodes

* Add `docs` support for SavedQuery descriptions

* Support selctor methods for SavedQuery nodes

* Add `refs` property to SavedQuery node

We don't actually append anything to `refs` for SavedQuery nodes currently.
I'm not sure if anything needs to be appended to them. Regardless, we
access the `refs` property throughout the codebase while iterating over
nodes. It seems wise to support this attribute as to not accidently blow
something up with it not existing.

* Support `saved_queries` when upgrading from manifests <= v10 (and regenerate v11)

* Add changie doc for saved query node support

* Pin to dbt-semantic-interfaces 0.3.0b1 for saved query work

We're gonna release DSI 0.3.0, and if this PR automatically pulls that
in things will break. But the things that need fixing should be handled
separately from this PR. After releasing DSI 0.3.0 I'm going to create
a branch off/ontop of this one, and open a stacked PR with the associated
changes.

* Bump supported DSI version to 0.3.x

* Switch metric filters and saved query where to use ne WhereFilterIntersection

* Update schema yaml readers to create WhereFilterInterfaces

* Expand metric filters and saved query where property to handle both str and list of strs

* Update tests which were broken by where filter changes

* Regeneate v11 manifest

* Fixup: Update `SavedQueryLookup.perform_lookup` to operate on saved queries

I missed this when I was copy and pasting 🤦
2023-10-11 15:54:11 -07:00
Peter Webb
2e35426d11 Add support for getting freshness from DBMS metadata (#8795)
* Add support for getting freshness from DBMS metadata

* Add changelog entry

* Add simple test case

* Change parsing error to warning and add new event type for warning

* Code review simplification of capability dict.

* Revisions to the capability mechanism per review

* Move utility function.

* Reduce try/except scope

* Clean up imports.

* Simplify typing per review

* Unit test fix
2023-10-11 15:55:57 -04:00
Emily Rockman
bf10a29f06 update v10 manifest on main (#8834)
* update manifest

* add changelog
2023-10-11 14:52:01 -05:00
Gerda Shank
a7e2d9bc40 Partial parsing issue when adding groups and updating models at the same time (#8817) 2023-10-11 15:01:49 -04:00
Michelle Ark
a3777496b5 fix changelog (#8833) 2023-10-11 13:22:05 -04:00
dependabot[bot]
edf6aedc51 Bump docker/build-push-action from 4 to 5 (#8783)
* Bump docker/build-push-action from 4 to 5

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4 to 5.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
2023-10-11 09:51:51 -07:00
Jeremy Cohen
53845d0277 Add warning_tag to UnversionedBreakingChange (#8828) 2023-10-11 17:54:16 +02:00
Mike Alfare
3d27483658 ADAP-850: Support test results as a view (#8653)
* add `store_failures_as` parameter to TestConfig, catch strategy parameter in test materialization
* create test results as views
* updated test expected values for new config option
* break up tests into reusable tests and adapter specific configuration, update test to check for relation type and confirm views update
* move test configuration into base test class
* allow `store_failures_as` to drive whether failures are stored
* update expected test config dicts to include the new default value for store_failures_as
* Add `store_failures_as` config for generic tests
* cover --store-failures on CLI gap
* add generic tests test case for store_failures_as
* update object names for generic test case tests for store_failures_as
* remove unique generic test, it was not testing `store_failures_as`
* pull generic run and assertion into base test class to turn tests into quasi-parameterized tests
* add ephemeral option for store_failures_as, as a way to easily turn off store_failures at the model level
* add compilation error for invalid setting of store_failures_as

---------

Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
2023-10-10 17:26:01 -04:00
Emily Rockman
4f9bd0cb38 Fix uncaught exception for group updates (#8792)
* add test

* write test

* fix test

* updating test

* add clean

* cleanup

* more tests, fix comment

* add new test, move fixtures
2023-10-10 15:37:47 -05:00
Kshitij Aranke
3f7f7de179 Fix #8682: Override path-like args in dbt retry (#8803) 2023-10-10 19:31:54 +01:00
Kshitij Aranke
6461f5aacf Fix #8022: Foreign key constraint on incremental model results in Database Error (#8768) 2023-10-10 18:32:51 +01:00
Doug Beatty
339957b42c Explanation of Parsing vs. Compilation vs. Runtime (#8744)
* Explanation of Parsing vs. Compilation vs. Runtime

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Update core/dbt/parser/parsing-vs-compilation-vs-runtime.md

* Apply suggestions from code review

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

* Fix a couple markdown rendering issues

* Move to the "explain it like im 64" folder

When ELI5 just isnt detailed enough.

* Disambiguate Python references

Disambiguate Python references and delineate SQL models ("Jinja-SQL") from Python models ("dbt-py")

---------

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
2023-10-10 10:51:54 -06:00
Gerda Shank
4391dc1a63 Type aliasing for model contract column data_type (#8592) 2023-10-10 11:43:26 -04:00
Michelle Ark
964e0e4e8a [Fix] respect project root when loading seeds (#8762) 2023-10-10 11:17:41 -04:00
Chenyu Li
549dbf3390 Deps lock by justbldwn (#8408)
*  adding installed_packages.json functionality

*  update test_simple_dependency_deps test

* 📝 adding changelog for deps feature via changie

*  restructure deps command, include lock/add

*  add new deps event types to sample_values

*  fix test_simple_dependency_deps test

* 🐛 attempting to fix cli commands

* 🐛 convert dbt deps to dbt deps install

also leave dbt deps as just a new click group

*  update test_command_mutually_exclusive_option

change deps command to deps install

*  update functional tests from deps > deps install

*  change missing deps to deps install

*  convert adapter tests to deps install from deps

* move back to deps and merge more with main

* fix-unittest

* add hash

* foramt yml and update command structure

* nits

* add new param

* nits

* nits

* nits

* fix_tests

* pr_feedback

* nits

* nits

* move_check

* Update Features-20230125-165933.yaml

---------

Co-authored-by: Justin Baldwin <91483530+justbldwn@users.noreply.github.com>
2023-10-09 21:05:00 -07:00
Quigley Malcolm
70b2e15a25 Add semantic model test to test_contracts_graph_parsed.py (#8654)
* Add semantic model test to `test_contracts_graph_parsed.py`

The tests in `test_contracts_graph_parsed.py` are meant to ensure
that we can go from objects to dictionaries and back without any
changes. We've had a desire to simplify these tests. Most tests in
this file have three to four fixtures, this test only has one. What
a test of this format ensures is that parsing a SemanticModel from
a dictionary doesn't add/drop any keys from the dictionary and that
when going back to the dictionary no keys are dropped. This style of
test will still break whenever the semantic model (or sub objects)
change. However now when that happens, only one fixture will have to
be updated (whereas previously we had to update 3-4 fixtures).

* Begin using hypothesis package for symmetry testing

Hypothesis is a python package for doing property testing. The `@given`
parameterizes a test, with it generating the arguements it has following
`strategies`. The main strategies we use is `builds` this takes in a callable
passes any sub strategies for named arguements, and will try to infer any
other arguments if the callable is typed. I found that even though the
test was run many many times, some of the `SemanticModel` properties
weren't being changed. For instance `dimensions`, `entities`, and `measures`
were always empty lists. Because of this I defined sub strategies for
some attributes of `SemanticModel`s.

* Update unittest readme to have details on test_contracts_graph_parsed methodology
2023-10-09 14:55:26 -07:00
Mark Scannell
bb249d612c Generate static index html documentation (#8615)
* Include option to generate static index.html

* Added changie

* Using DBT's system load / write file methods for better cross platform
support

* Updated docs tests with dbt.client.systems calls for file reading

* Writing out static_index.html as binary file to prevent line-ending
conversions on Windows. (similar behaviour as index.html)
2023-10-06 10:37:13 -07:00
Emily Rockman
17773bdb94 pin types-requests in dev-requirements (#8788)
* pin types-requests

* changelog
2023-10-06 09:53:08 -05:00
Gerda Shank
f30293359c Selectors in docs generate limits catalog generation (#8772) 2023-10-05 16:00:12 -04:00
Emily Rockman
0c85e6149f remove guild as codeowner (#8778) 2023-10-05 10:16:32 -05:00
Emily Rockman
ec57d7af94 add gha for dependabot checks (#8777) 2023-10-05 10:16:20 -05:00
Emily Rockman
df791f729c support doc blocks (#8771) 2023-10-05 08:12:28 -05:00
Emily Rockman
c6ff3abecd remove top level meta attribute (#8766) 2023-10-04 13:23:09 -05:00
Emily Rockman
eac13e3bd3 Add meta to SemanticModels (#8754)
* WIP

* changelog
2023-10-03 13:08:37 -05:00
Emily Rockman
46ee3f3d9c rebuild manifest missed fields (#8755)
* rebuild manifest missed fields

* changelogs
2023-10-02 09:38:50 -05:00
Peter Webb
5e1f0c5fbc Report Resource Usage Statistics When a dbt Command Finishes (#8671)
* Add performance metrics to the CommandCompleted event.

* Add changelog entry.

* Add flag for controling the log level of ResourceReport.

* Update changelog entry to reflect changes

* Remove outdated attributes

* Work around missing resource module on windows

* Fix corner case where flags are not set
2023-09-29 18:37:37 -04:00
Peter Webb
c4f09b160a Add new get_catalog_relations macro, Supporting Changes (#8648)
* Add new get_catalog_relations macro, allowing dbt to specify which relations in a schema the adapter should return data about

* Implement postgres adapter support for relation filtering on catalog queries

* Code review changes adding feature flag for catalog-by-relation-list support

* Use profile specified in --profile with dbt init (#7450)

* Use profile specified in --profile with dbt init

* Update .changes/unreleased/Fixes-20230424-161642.yaml

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Refactor run() method into functions, replace exit() calls with exceptions

* Update help text for profile option

---------

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* add TestLargeEphemeralCompilation (#8376)

* Fix a couple of issues in the postgres implementation of get_catalog_relations

* Add relation count limit at which to fall back to batch retrieval

* Better feature detection mechanism for adapters.

* Code review changes to get_catalog_relations and adapter feature checking

* Add changelog entry

---------

Co-authored-by: ezraerb <ezraerb@alum.mit.edu>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-09-29 16:13:23 -04:00
Jeremy Cohen
48c97e86dd Fix tag: selection for projects with semantic models (#8750)
* Add unit test to repro regression

* Add defensive code for tag: selection

* Add changelog entry
2023-09-29 09:49:32 -07:00
Emily Rockman
416bc845ad fix duplication of docs issues (#8747)
* fix duplication of docs issues

* update conditional to only run on merged PRs
2023-09-29 09:40:34 -05:00
Michelle Ark
408a78985a Fix: avoid double-rendering sql_header in dbt show (#8740) 2023-09-28 19:35:14 +01:00
Emily Rockman
0c965c8115 update changelog kind (#8737) 2023-09-28 08:20:57 -07:00
Philippe Boyd
f65e4b6940 feat: resolve packages with same git repo and unique subdirectory (#8322) 2023-09-27 15:31:37 -04:00
Michelle Ark
a2d4424f92 Inline limit in SQL sent from dbt show (#8641) 2023-09-27 15:56:34 +01:00
Kshitij Aranke
997f839cd6 Disallow cleaning paths outside current working directory (#8469) 2023-09-27 15:47:00 +01:00
Emily Rockman
556fad50df Model contracts: raise warning for numeric types without specified scale (#8721)
* add warning when contracting fields don't have precision

* rename files

* changelog

* move tests out of adapter zone

* Update core/dbt/include/global_project/macros/adapters/columns.sql

Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>

* Apply suggestions from code review

---------

Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
2023-09-27 08:00:12 -05:00
dave-connors-3
bb4214b5c2 Dc/8546 semantic models in graph selection (#8589) 2023-09-26 23:23:03 +01:00
Kshitij Aranke
f17c1f3fe7 Fix #6497: Support global flags passed in after subcommands (#8670) 2023-09-26 16:16:36 +01:00
Kshitij Aranke
d4fe9a8ad4 Fix #8509: Support doc blocks in nested semantic model YAML (#8709) 2023-09-26 16:02:21 +01:00
Emily Rockman
2910aa29e4 Automated Repo Cleanup (#8686)
* add workflow

* cleanup and rename workflow

* rename workflow in actions

* add more context
2023-09-26 09:35:29 -05:00
Renan Leme
89cc073ea8 CT-3144 Fix test edges type filter on Graph (#8696)
* CT-3144 Fix test edges filter

* CT-3144 Add changelog

* CT-3144 Remove duplicated line

* CT-3144 Remove duplicated line

* CT-3144 Rename vars

* CT-3144 Update filter to use get_edge_data

* Trigger cla
2023-09-26 09:55:45 -04:00
Quigley Malcolm
aa86fdfe71 Add date spine macros to core (#8616)
* Add `date_spine` macro (and macros it depends on) from dbt-utils to core

The macros added are
- date_spine
- get_intervals_between
- generate_series
- get_powers_of_two

We're adding these to core because they are becoming more prevalently used
with the increase usage in the semantic layer. Basically if you are
using the semantic layer currently, then it is almost a requirement
to use dbt-utils, which is undesireable given the SL is supported directly
in core. The primary focus of this was to just add `date_spine`. However,
because `date_spine` depends on other macros, these other macros were
also moved.

* Add adapter tests for `get_powers_of_two` macro

* Add adapter tests for `generate_series` macro

* Add adapter tests for `get_intervals_between` macro

* Add adapter tests for `date_spine` macro

* Improve test fixture for `date_spine` macro to work with multiple adapters

* Cast to types to date in fixture_date_spine when targeting redshift

* Improve test fixture for `get_intervals_between` macro to work with multiple adapters

* changie doc for adding date_spine macro
2023-09-25 12:20:05 -07:00
Quigley Malcolm
48e9ced781 Support null coalescing properties for metric nodes (#8700)
* Include 'join_to_timespine` and `fill_nulls_with` in metric fixture

* Support `join_to_timespine` and `fill_nulls_with` properties on measure inputs to metrics

* Assert new `fill_nulls_with` and `join_to_timespine` properties don't break associated DSI protocol

* Add doc for metric null coalescing improvements

* Fix unit test for unparsed metric objects

The `assert_symmetric` function asserts that dictionaries are mostly
equivalent. I say mostly equivalent because it drops keys that are
`None`. The issue is that that `join_to_timespine` gets defaulted
to `False`, so we have to specify it in the `get_ok_dict` so that
they match.
2023-09-25 11:02:47 -07:00
Doug Beatty
7b02bd1f02 Lower bound of 8.0.2 for click (#8684)
* Lower bound of `8.0.2` for `click`

* Changelog entry
2023-09-24 13:46:54 -06:00
Emily Rockman
417fc2a735 Support quoted parameter list for MultiOption cli options (#8665)
* allow multioption to be quoted

* changelog

* fix test

* remove list format

* fix tests

* fix list object

* review arg change

* fix quotes

* Update .changes/unreleased/Features-20230918-150855.yaml

* add types

* convert list to set in test

* make mypy happy

* mroe mypy happiness

* more mypy happiness

* last mypy change

* add node to test
2023-09-22 14:34:36 -05:00
Quigley Malcolm
317128f790 Update pull request template to include type annotations as part of checklist (#8687)
* Update pull request template to include type annotations as part of checklist

* Add to checklist item a link to python typing information
2023-09-22 11:30:28 -07:00
Emily Rockman
e3dfb09b10 Support labels for semantic_models, dimensions, measures and entities (#8646)
* first pass

* changelog

* changelog

* Delete .changes/unreleased/Features-20230913-155802.yaml

* Update .changes/unreleased/Features-20230914-074429.yaml
2023-09-22 10:56:57 -05:00
colin-rogers-dbt
d912654110 Allow adapters to include python package logging in dbt logs (#8643)
* add set_package_log_level functionality

* set package handler

* set package handler

* add logging about stting up logging

* test event log handler

* add event log handler

* add event log level

* rename package and add unit tests

* revert logfile config change

* cleanup and add code comments

* add changie

* swap function for dict

* add additional unit tests

* fix unit test
2023-09-20 09:27:30 -07:00
Peter Webb
34ab4cf9be More Type Annotations (#8536)
* Extend use of type annotations in the events module.

* Add return type of None to more __init__ definitions.

* Still more type annotations adding -> None to __init__

* Tweak per review
2023-09-20 11:35:22 -04:00
Michelle Ark
d597b80486 add TestLargeEphemeralCompilation (#8376) 2023-09-18 15:00:10 +01:00
ezraerb
3f5ebe81b9 Use profile specified in --profile with dbt init (#7450)
* Use profile specified in --profile with dbt init

* Update .changes/unreleased/Fixes-20230424-161642.yaml

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

* Refactor run() method into functions, replace exit() calls with exceptions

* Update help text for profile option

---------

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-09-15 10:14:53 -05:00
Kshitij Aranke
f52bd9287b Fix #8160: Warn when --state == --target (#8638) 2023-09-14 17:04:32 +01:00
Gerda Shank
f5baeeea1c Allow setting access in config in addition to properties (#8635) 2023-09-14 11:09:41 -04:00
Mike Alfare
3cc7044fb3 Loosen type on replaceable_relation and renameable_relation and provide guidance in docstrings (#8647)
* loosen type on replaceable_relation and renameable_relation and provide guidance in docstrings
2023-09-13 20:17:33 -04:00
Gerda Shank
26c7675c28 Fix test_numeric_values of the show test (#8644) 2023-09-13 15:38:18 -04:00
colin-rogers-dbt
8aaed0e29f replace is_replaceable function with can_be_replaced call (#8637) 2023-09-13 10:19:09 -04:00
Emily Rockman
5182e3c40c split up test class (#8610) 2023-09-12 10:59:56 -05:00
Kshitij Aranke
1e252c7664 move codecov failure threshold to 0.1% (#8625) 2023-09-12 16:45:36 +01:00
Kshitij Aranke
05ef3b6e44 Audit potential circular dependencies (#8489) 2023-09-12 14:00:16 +01:00
leahwicz
ad04012b63 Revert "Merge branch 'main' into main" (#8622)
This reverts commit c93cba4603, reversing
changes made to 971669016f.
2023-09-11 22:18:12 -04:00
leahwicz
c93cba4603 Merge branch 'main' into main 2023-09-11 21:40:10 -04:00
Mike Alfare
971669016f ADAP-869: Support atomic replace in replace macro (#8539)
* move config changes into alter.sql in alignment with other adapters
* move shared relations macros to relations root
* move single models files to models root
* add table to replace
* move create file into relation directory
* implement replace for postgres
* move column specific macros into column directory
* add unit test for can_be_replaced
* update renameable_relations and replaceable_relations to frozensets to set defaults
* fixed tests for new defaults
2023-09-11 16:23:25 -04:00
Emily Rockman
6c6f245914 update PR template (#8613) 2023-09-11 13:48:25 -05:00
Quigley Malcolm
b39eeb328c Unskip and rename test_expression_metric (#8578)
* Add docstrings to `contracts/graph/metrics.py` functions to document what they do

Used [dbt-labs/dbt-core#5607](https://github.com/dbt-labs/dbt-core/pull/5607)
for context on what the functions should do.

* Add typing to `reverse_dag_parsing` and update function to work on 1.6+ metrics

* Add typing to `parent_metrics` and `parent_metrics_names`

* Add typing to `base_metric_dependency` and `derived_metric_dependency` and update functions to work on 1.6+ metrics

* Simplify implementations of `basic_metric_dependency` and `derived_metric_dependnecy`

* Add typing to `ResolvedMetricReference` initialization

* Add typing to `derived_metric_dependency_graph`

* Simplify conditional controls in `ResolvedMetricReference` functions

The functions in `ResolvedMetricReference` use `manifest.metric.get(...)`
which will only return either a `Metric` or `None`, never a different
node type. Thus we don't need to check that the returned metric is
a metric.

* Don't recurse on over `depends_on` for non-derived metrics in `reverse_dag_parsing`

The function `reverse_dag_parsing` only cares about derived metrics,
that is metrics that depend on other metrics. Metrics only depend on
other metrics if they are one of the `DERIVED_METRICS` types. Thus
doing a recursive call to `reverse_dag_parsing` for non `DERIVED_METRICS`
types is unnecessary. Previously we were iterating over a metric's
`depends_on` property regardless of whether the metric was a `DERIVED_METRICS`
type. Now we only do this work if the metric is of a `DERIVED_METRICS`
type.

* Simplify `parent_metrics_names` by having it call `parent_metrics`

* Unskip `TestMetricHelperFunctions.test_derived_metric` and update fixture setup

* Add changie doc for metric helper function updates

* Get manifest in `test_derived_metric` from the parse dbt_run invocation

* Remove `Relation` a intiatlization attribute for `ResolvedMetricReference`

* Add return typing to class `__` functions of `ResolvedMetricReference`

* Move from `manifest.metrics.get` to `manifest.expect` in metric helpers

Previously with `manifest.metrics.get` we were just skipping when `None`
was returned. Getting `None` back was expected in that `parent_unique_id`s
that didn't belong to metrics should return `None` when calling
`manifest.metrics.get`, and these are fine to skip. However, there's
an edgecase where a `parent_unique_id` is supposed to be a metric, but
isn't found, thus returning `None`. How likely this edge case could
get hit, I'm not sure, but it's a possible edge case. Using `manifest.metrics.get`
it we can't actually tell if we're in the edge case or not. By moving
to `manifest.expect` we get the error handling built in, and the only
trade off is that we need to change our conditional to skip returned
nodes that aren't metrics.
2023-09-07 15:28:36 -07:00
Emily Rockman
be94bf1f3c Preserve decimal places for dbt show (#8561)
* update `Number` class to handle integer values (#8306)

* add show test for json data

* oh changie my changie

* revert unecessary cahnge to fixture

* keep decimal class for precision methods, but return __int__ value

* jerco updates

* update integer type

* update other tests

* Update .changes/unreleased/Fixes-20230803-093502.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* account for integer vs number on table merges

* add tests for combining number with integer.

* add unit test when nulls are added

* cant none as an Integer

* fix null tests

---------

Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Dave Connors <dave.connors@fishtownanalytics.com>
2023-09-07 09:15:11 -05:00
Ben Mosher
e24a952e98 compile --no-inject-ephemeral-ctes flag (#8482) 2023-09-07 10:13:40 -04:00
Michelle Ark
89f20d12cf make UnparsedVersion.__lt__ order-agnostic (#8559) 2023-09-06 17:46:59 -04:00
Michelle Ark
ebeb0f1154 test advanced ref override (#8552) 2023-09-06 17:45:16 -04:00
Kshitij Aranke
d66fe214d9 Fix #8544: Parse the correct schema version from manifest (#8551)
* Fix #8544: Parse the correct schema version from manifest

* add changie

* add comment
2023-09-06 20:38:37 +01:00
Michelle Ark
75781503b8 add typing to partially typed methods in runnable.py (#8569) 2023-09-06 13:57:55 -04:00
Kshitij Aranke
9aff3ca274 Fix #8398: Add typing to __init__ in base.py (#8568) 2023-09-06 18:13:05 +01:00
Emily Rockman
7e2a08f3a5 remove label (#8557) 2023-09-05 19:12:16 -05:00
Emily Rockman
a0e13561b1 Support dbt-cloud config dict in dbt_project.yml (#8527)
* first pass at adding dbt-cloud config

* changelog

* fix test, add direct validation
2023-09-05 09:48:37 -05:00
FishtownBuildBot
7eedfcd274 [Automated] Merged prep-release/1.7.0b2_6049623160 into target main during release process 2023-09-01 08:48:37 -05:00
Github Build Bot
da779ac77c Bumping version to 1.7.0b2 and generate changelog 2023-09-01 12:51:37 +00:00
Mike Alfare
adfa3226e3 ADAP-814: Add support for replacing materialized views with tables/views and vice versa (#8449)
* first draft of adding in table - materialized view swap
* table/view/materialized view can all replace each other
* update renameable relations to a config
* migrate relations macros from `macros/adapters/relations` to `macros/relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar
* move drop macro to drop macro file
* align the behavior of get_drop_sql and drop_relation, adopt existing default from drop_relation
* add explicit ddl for drop statements instead of inheriting the default from dbt-core
* update replace macro dependent macros to align with naming standards
* update type for mashumaro, update related test
2023-08-31 20:41:45 -04:00
Quigley Malcolm
e5e1a272ff Fix untyped functions in core/dbt/context/base.py (#8525)
* Improve typing of `ContextMember` functions

* Improve typing of `Var` functions

* Improve typing of `ContextMeta.__new__`

* Improve typing `BaseContext` and functions

In addition to just adding parameter typing and return typing to
`BaseContext` functions. We also declared `_context_members_` and
`_context_attrs_` as properites of `BaseContext` this was necessary
because they're being accessed in the classes functions. However,
because they were being indirectly instantiated by the metaclass
`ContextMeta`, the properties weren't actually known to exist. By
adding declaring the properties on the `BaseContext`, we let mypy
know they exist.

* Remove bare `invocations` of `@contextmember` and `@contextproperty`, and add typing to them

Previously `contextmember` and `contextproperty` were 2-in-1 decorators.
This meant they could be invoked either as `@contextmember` or
`@contextmember('some_string')`. This was fine until we wanted to return
typing to the functions. In the instance where the bare decorator was used
(i.e. no `(...)` were present) an object was expected to be returned. However
in the instance where parameters were passed on the invocation, a callable
was expected to be returned. Putting a union of both in the return type
made the invocations complain about each others' return type. To get around this
we've dropped the bare invocation as acceptable. The parenthesis are now always
required, but passing a string in them is optional.
2023-08-31 13:34:57 -07:00
Gerda Shank
d8e8a78368 Fix snapshot success message to display "INSERT 0 1" (for example) instead of success (#8524) 2023-08-31 13:09:15 -04:00
Emily Rockman
7ae3de1fa0 Semantic model configs - enable/disable + groups (#8502)
* WIP

* WIP

* get group and enabled added

* changelog

* cleanup

* getting measure lookup working

* missed file

* get project level working

* fix last test

* add groups to config tests

* more group tests

* fix path

* clean up manifest.py

* update error message

* fix test assert

* remove extra check

* resolve conflicts in manaifest

* update manifest

* resolve conflict

* add alias
2023-08-30 20:18:22 -05:00
Gerda Shank
72898c7211 Add return value to AdapterContainer.__init__ and AdapterMeta.__new__ (#8523) 2023-08-30 17:37:32 -04:00
Peter Webb
fc1a14a0e3 Include Compiled Node Attributes in run_results.json (#8492)
* Add compiled node properties to run_results.json

* Include compiled-node attributes in run_results.json

* Fix typo

* Bump schema version of run_results

* Fix test assertions

* Update expected run_results to reflect new attributes

* Code review changes

* Fix mypy warnings for ManifestLoader.load() (#8443)

* revert python version for docker images (#8445)

* revert python version for docker images

* add comment to not update python version, update changelog

* Bumping version to 1.7.0b1 and generate changelog

* [CT-3013]  Fix parsing of `window_groupings` (#8454)

* Update semantic model parsing tests to check measure non_additive_dimension spec

* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`

* Add changie doc for `window_groupings`  parsing fix

* update `Number` class to handle integer values (#8306)

* add show test for json data

* oh changie my changie

* revert unecessary cahnge to fixture

* keep decimal class for precision methods, but return __int__ value

* jerco updates

* update integer type

* update other tests

* Update .changes/unreleased/Fixes-20230803-093502.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Improve docker image README (#8212)

* Improve docker image README

- Fix unnecessary/missing newline escapes
- Remove double whitespace between parameters
- 2-space indent for extra lines in image build commands

* Add changelog entry for #8212

* ADAP-814: Refactor prep for MV updates (#8459)

* apply reformatting changes only for #8449
* add logging back to get_create_materialized_view_as_sql
* changie

* swap trigger (#8463)

* update the implementation template (#8466)

* update the implementation template

* add colon

* Split tests into classes (#8474)

* add flaky decorator

* split up tests into classes

* revert update agate for int (#8478)

* updated typing and methods to meet mypy standards (#8485)

* Convert error to conditional warning for unversioned contracted model, fix msg format (#8451)

* first pass, tests need updates

* update proto defn

* fixing tests

* more test fixes

* finish fixing test file

* reformat the message

* formatting messages

* changelog

* add event to unit test

* feedback on message structure

* WIP

* fix up event to take in all fields

* fix test

* Fix ambiguous reference error for duplicate model names across packages with tests (#8488)

* Safely remove external nodes from manifest (#8495)

* [CT-2840] Improved semantic layer protocol satisfaction tests (#8456)

* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified

* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols

* Add test asserting Defaults obj satisfies protocol

* Add test asserting SemanticModel with optionals specified satisfies protocol

* Split dimension protocol satisfaction tests into with and without optionals

* Simplify DSI Protocol import strategy in protocol satisfaction tests

* Add test asserting DimensionValidtyParams satisfies protocol

* Add test asserting DimensionTypeParams satisfies protocol

* Split entity protocol satisfaction tests into with and without optionals

* Split measure protocol satisfication tests and add measure aggregation params satisficaition test

* Split metric protocol satisfaction test into optional specified an unspecified

Additionally, create where_filter pytest fixture

* Improve protocol satisfaction tests for MetricTypeParams and sub protocols

Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow

* Convert to using mashumaro jsonschema with acceptable performance (#8437)

* Regenerate run_results schema after merging in changes from main.

---------

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Jaime Martínez Rincón <jaime@jamezrin.name>
Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-08-30 17:28:49 -04:00
Gerda Shank
f063e4e01c Convert to using mashumaro jsonschema with acceptable performance (#8437) 2023-08-30 14:06:59 -04:00
Quigley Malcolm
07372db906 [CT-2840] Improved semantic layer protocol satisfaction tests (#8456)
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified

* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols

* Add test asserting Defaults obj satisfies protocol

* Add test asserting SemanticModel with optionals specified satisfies protocol

* Split dimension protocol satisfaction tests into with and without optionals

* Simplify DSI Protocol import strategy in protocol satisfaction tests

* Add test asserting DimensionValidtyParams satisfies protocol

* Add test asserting DimensionTypeParams satisfies protocol

* Split entity protocol satisfaction tests into with and without optionals

* Split measure protocol satisfication tests and add measure aggregation params satisficaition test

* Split metric protocol satisfaction test into optional specified an unspecified

Additionally, create where_filter pytest fixture

* Improve protocol satisfaction tests for MetricTypeParams and sub protocols

Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
2023-08-29 09:40:30 -07:00
Michelle Ark
48d04e8141 Safely remove external nodes from manifest (#8495) 2023-08-28 14:10:34 -04:00
Michelle Ark
6234267242 Fix ambiguous reference error for duplicate model names across packages with tests (#8488) 2023-08-25 09:15:07 -04:00
Emily Rockman
1afbb87e99 Convert error to conditional warning for unversioned contracted model, fix msg format (#8451)
* first pass, tests need updates

* update proto defn

* fixing tests

* more test fixes

* finish fixing test file

* reformat the message

* formatting messages

* changelog

* add event to unit test

* feedback on message structure

* WIP

* fix up event to take in all fields

* fix test
2023-08-24 19:29:31 -05:00
Mike Alfare
d18a74ddb7 updated typing and methods to meet mypy standards (#8485) 2023-08-24 16:25:42 -04:00
Michelle Ark
4d3c6d9c7c revert update agate for int (#8478) 2023-08-23 14:06:52 -04:00
Emily Rockman
10f9724827 Split tests into classes (#8474)
* add flaky decorator

* split up tests into classes
2023-08-23 11:27:34 -05:00
Emily Rockman
582faa129e update the implementation template (#8466)
* update the implementation template

* add colon
2023-08-22 13:58:25 -05:00
Emily Rockman
4ec87a01e0 swap trigger (#8463) 2023-08-21 15:48:37 -05:00
Mike Alfare
ff98685dd6 ADAP-814: Refactor prep for MV updates (#8459)
* apply reformatting changes only for #8449
* add logging back to get_create_materialized_view_as_sql
* changie
2023-08-21 14:44:47 -04:00
Jaime Martínez Rincón
424f3d218a Improve docker image README (#8212)
* Improve docker image README

- Fix unnecessary/missing newline escapes
- Remove double whitespace between parameters
- 2-space indent for extra lines in image build commands

* Add changelog entry for #8212
2023-08-18 14:14:59 -05:00
dave-connors-3
661623f9f7 update Number class to handle integer values (#8306)
* add show test for json data

* oh changie my changie

* revert unecessary cahnge to fixture

* keep decimal class for precision methods, but return __int__ value

* jerco updates

* update integer type

* update other tests

* Update .changes/unreleased/Fixes-20230803-093502.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-08-18 14:01:08 -05:00
Quigley Malcolm
49397b4d7b [CT-3013] Fix parsing of window_groupings (#8454)
* Update semantic model parsing tests to check measure non_additive_dimension spec

* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`

* Add changie doc for `window_groupings`  parsing fix
2023-08-18 10:57:29 -07:00
FishtownBuildBot
0553fd817c [Automated] Merged prep-release/1.7.0b1_5895067219 into target main during release process 2023-08-17 15:04:21 -05:00
Github Build Bot
7ad971f720 Bumping version to 1.7.0b1 and generate changelog 2023-08-17 19:25:44 +00:00
Matthew McKnight
f485c13035 revert python version for docker images (#8445)
* revert python version for docker images

* add comment to not update python version, update changelog
2023-08-17 14:18:28 -05:00
Gerda Shank
c30b691164 Fix mypy warnings for ManifestLoader.load() (#8443) 2023-08-17 14:34:22 -04:00
Quigley Malcolm
d088d4493e Add doc string context to Identifier validion regex rule (#8440) 2023-08-17 10:11:03 -07:00
Emily Rockman
770f804325 Fix test failures (#8432)
* fail job when anything fails in previous matrix

* tweak wording

* PR feedback
2023-08-17 07:20:07 -05:00
Emily Rockman
37a29073de change trigger (#8418) 2023-08-16 19:49:59 -05:00
Peter Webb
17cd145f09 Temporarily disable test. (#8434) 2023-08-16 16:42:48 -04:00
Kshitij Aranke
ac539fd5cf Ignore .github and .changes directories for code coverage (#8424)
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-08-16 16:19:12 +01:00
Gerda Shank
048553ddc3 Fix using project-dir with list command and path selector (#8388) 2023-08-16 10:39:22 -04:00
Gerda Shank
dfe6b71fd9 Add return values to functions to fix mypy warnings (#8416) 2023-08-16 10:37:46 -04:00
Peter Webb
18ee93ca3a Fix run_results.json Performance Regression (#8413)
* Remedy performance regression by only writing run_results.json once.

* Write results before cleaning up connections.
2023-08-15 16:41:14 -04:00
Emily Rockman
cb4bc2d6e9 Automate opening docs issues (#8373)
* first pass

* WIP

* update issue body

* fix triggering label

* fix docs

* add better run name

* reduce complexity

* update description

* fix PR title

* point at workflow on main

* fix wording

* add label
2023-08-15 14:20:46 -05:00
Quigley Malcolm
b0451806ef [CT-2526] Add ability to automatically create metrics from semantic model measures (#8310)
* Update semantic model parsing test to check `create_metric = true` functionality

* Add `create_metric` boolean property to unparsed measure objects

* Begin creating metrics from measures with `create_metric = True`

* Add test ensuring partial parsing handles metrics generated from measures

* Ensure partial parsing appropriately deletes metrics generated from semantic models

* Add changie doc for  addition

* Separate generated metrics from parsed metrics for partial parsing

I was doing a demo earlier today of this branch (minus this commit)
and noticed something odd. When I changes a semantic model, metrics
that should have been technically uneffected would get dropped. Basically
if I made a change to a semantic model which had metrics in the same
file, and then ran parse, those metrics defined in the same file
would get dropped. Then with no other changes, if I ran parse again
they would come back. What was happening was that parsed metrics
and generated metrics were getting tracked the same way on the file
objects for partial parsing. In 0787a7c7b6
we began dropping all metrics tracked in a file objects when changes
to semantic models were detected. Since parsed metrics and generated
metrics were being tracked together on the file object, the parsed
metrics were getting dropped as well. In this commit we begin separating
out the tracking of generated metrics and parsed metrics on the
file object, and now only drop the generated metrics when semantic
models have a detected change.

* Assert in test that  semantic model partial parsing doesn't clobber regular metrics
2023-08-14 12:42:11 -07:00
Kshitij Aranke
b514e4c249 Fix #8350: add connection status into list of statuses for dbt debug (#8351) 2023-08-14 18:13:10 +01:00
Michelle Ark
8350dfead3 add --no-partial-parse-file-diff flag (#8361)
* add --no-partial-parse-file-diff flag

* changelog entry
2023-08-14 13:24:16 +02:00
Michelle Ark
34e6edbb13 Fix: deleting models that depend on external nodes (#8330) 2023-08-14 10:31:58 +02:00
FishtownBuildBot
27be92903e Add new index.html and changelog yaml files from dbt-docs (#8346) 2023-08-14 13:12:37 +08:00
Michelle Ark
9388030182 fix ModelNodeArgs.fqn (#8364) 2023-08-11 23:21:44 -04:00
Emily Rockman
b7aee3f5a4 add env vars to tox.ini (#8365)
* add env vars to tox.ini

* revert test
2023-08-11 10:49:41 -05:00
Michelle Ark
83ff38ab24 track plugin.get_nodes (#8336) 2023-08-10 11:33:11 -04:00
Michelle Ark
6603a44151 Detect changes to model access, deprecation_date, and latest_version in state:modified (#8264) 2023-08-10 11:29:02 -04:00
Kshitij Aranke
e69d4e7f14 Fix #8245: Add flag to codecov report (#8341) 2023-08-09 18:14:52 +01:00
d-kaneshiro
506f65e880 fixed comment util.py (#8222)
* fixed comments util.py

* add CHANGELOG entries
2023-08-09 09:24:38 -04:00
Gerda Shank
41bb52762b Bump manifest jsonschema to v11, update v10 schema (#8335) 2023-08-08 22:55:47 -04:00
Anju
8c98ef3e70 Copy dir if symlink fails (#7447) 2023-08-08 21:37:36 -04:00
Emily Rockman
44d1e73b4f Fix missing quote in fixtures.py (#8324) 2023-08-07 15:46:44 -05:00
Emily Rockman
53794fbaba Update implementation-ticket.yml (#8332) 2023-08-07 14:09:37 -06:00
Emily Rockman
556b4043e9 Update implementation-ticket.yml to reference adapters (#8329) 2023-08-07 12:39:29 -05:00
Grant Murray
424c636533 [CT-2776] [Feature] Enable-post-parse-population-of-dbt-custom-env (#7998)
* patch(events/functions): enable-repopulation-of-metadata-vars

* changie new
2023-08-04 12:59:47 -07:00
Emily Rockman
f63709260e add formatting events into json logs (#8308)
* add formatting events into json logs

* changelog

* Delete Under the Hood-20230803-100811.yaml
2023-08-03 15:56:01 -05:00
Michelle Ark
991618dfc1 capitalize integration-report name (#8265) 2023-08-02 18:06:05 -04:00
Chenyu Li
1af489b1cd fix constructing param with 0 value (#8298)
* fix constructing param with 0 value

* Update core/dbt/cli/flags.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

---------

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-08-02 14:57:55 -07:00
Kshitij Aranke
a433c31d6e Fix #7179 (#8279) 2023-08-02 16:56:16 +01:00
Peter Webb
5814928e38 Issue One Event Per Node Failure (#8210)
* Replaced the FirstRunResultError and AfterFirstRunResultError events with RunResultError.

* Attempts at reasonable unit tests.

* Restore event manager after unit test.
2023-08-02 10:24:05 -04:00
Ramon Vermeulen
6130a6e1d0 Support configurable delimiter for seed files, default to comma (#3990) (#7186)
* Support configurable delimiter for seed files, default to comma (#3990)

* Update Features-20230317-144957.yaml

* Moved "delimiter" to seed config instead of node config

* Update core/dbt/clients/agate_helper.py

Co-authored-by: Cor <jczuurmond@protonmail.com>

* Update test_contracts_graph_parsed.py

* fixed integration tests

* Added functional tests for seed files with a unique delimiter

* Added docstrings

* Added a test for an empty string configured delimiter value

* whitespace

* ran black

* updated changie entry

* Update Features-20230317-144957.yaml

---------

Co-authored-by: Cor <jczuurmond@protonmail.com>
2023-08-01 09:15:43 -07:00
Quigley Malcolm
7872f6a670 Add tests for specifcally checking the population of SemanticModel.depends_on (#8226) 2023-07-31 14:15:09 -07:00
Emily Rockman
f230e418aa hard code test splits (#8258)
* change trigger

* add logic for different targets

* fix comment

* hard code test splits
2023-07-31 13:26:17 -05:00
Quigley Malcolm
518eb73f88 [CT-2888] Support dbt-semantic-interfaces 0.2.0 (#8250)
* Upgrade DSI dependency to ~=0.2.0

* Allow users to specify `primary_entity` on semantic models

* Add `primary_entity` and `primary_entity_reference` to SemanticModel node

* Plumb primary_entity from unparsed to parsed semantic nodes

* Fix metric filter specifications in existing tests

* Add changie doc about supporting DSI 0.2.0
2023-07-28 14:36:51 -07:00
Emily Rockman
5b6d21d7da loosen the click pin (#8232)
* loosen the click pin

* changelog

* separate out sqlparse pin

* remove changelog

* add ignores
2023-07-28 15:53:27 -05:00
Michelle Ark
410506f448 [Fix] raise execution errors for runnable tasks (#8237) 2023-07-28 13:18:51 -04:00
dependabot[bot]
3cb44d37c0 Bump mypy from 1.4.0 to 1.4.1 (#8219)
* Bump mypy from 1.4.0 to 1.4.1

Bumps [mypy](https://github.com/python/mypy) from 1.4.0 to 1.4.1.
- [Commits](https://github.com/python/mypy/compare/v1.4.0...v1.4.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

* update pre-commit

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-07-28 07:33:16 -05:00
Quigley Malcolm
f977ed7471 [CT-2879] Fix unbound variable error in checked_agg_time_dimension_for_measure (#8235)
* Fix unbound variable error in `checked_agg_time_dimension_for_measure`

* Improve assertion error message in `checked_agg_time_dimension_for_measure`

* Add changie doc for checked_agg_time_dimension_for_measure unbound variable fix

* Add unit tests for checking functionality of `checked_agg_time_dimension_for_measure`
2023-07-27 14:58:59 -07:00
Emily Rockman
3f5617b569 pin upper bound for sqlparse (#8236)
* pin upper bound for sqlparse

* changelog
2023-07-27 16:29:42 -05:00
Gerda Shank
fe9c875d32 Ensure that target_schema from snapshot config is promoted to node level (#8117) 2023-07-27 13:40:52 -04:00
Michelle Ark
23b16ad6d2 Split integration tests into parallel groups / jobs (#6346) 2023-07-27 11:27:34 -04:00
Gerda Shank
fdeccfaf24 Initialize sqlparse lexer and tweak order of setting compilation fields (#8215) 2023-07-26 17:29:51 -04:00
dependabot[bot]
fecde23da5 Bump mypy from 1.3.0 to 1.4.0 (#7912)
* Bump mypy from 1.3.0 to 1.4.0

Bumps [mypy](https://github.com/python/mypy) from 1.3.0 to 1.4.0.
- [Commits](https://github.com/python/mypy/compare/v1.3.0...v1.4.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

* add to pre-commit config

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-07-26 15:15:40 -05:00
Emily Rockman
b1d931337e up timeout (#8218) 2023-07-26 13:50:03 -05:00
Michelle Ark
39542336b8 Update implementation-ticket.yml (#8208) 2023-07-25 18:50:13 -04:00
Peter Webb
799588cada Update Core for Latest dbt-extractor with Version Parsing (#8206)
* Update dbt-extractor requirement, and adjust ref handling accordingly

* Add changelog entry.
2023-07-25 15:47:29 -04:00
Chenyu Li
f392add4b8 add param to control maxBytes for single dbt.log file (#8200)
* add param to control maxBytes for single dbt.log file

* nits

* nits

* Update core/dbt/cli/params.py

Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>

---------

Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
2023-07-25 12:30:55 -07:00
Gerda Shank
49560bf2a2 Check for existing_relation immediately prior to renaming (#8193) 2023-07-25 12:59:18 -04:00
Quigley Malcolm
44b3ed5ae9 [CT-2594] Fix serialization of warn_error_options on Contexts (#8180)
* Add test ensuring `warn_error_options` is dictified in `invocation_args_dict` of contexts

* Add dictification specific to `warn_error_options` in `args_to_dict`

* Changie doc for serialization changes of warn_error_options
2023-07-24 13:35:27 -07:00
Quigley Malcolm
6235145641 [CT-1483] Let macro names include word materialization (#8181)
* Add test asserting that a macro with the work materializtion doesn't cause issues

* Let macro names include the word `materialization`

Previously we were checking if a macro included a materialization
based on if the macro name included the word `materialization`. However,
a macro name included the word `materialization` isn't guarnteed to
actually have a materialization, and a macro that doesn't have
`materialization` in the name isn't guaranteed to not have a materialization.
This change is to detect macros with materializations based on the
detected block type of the macro.

* Add changie doc materialization in macro detection
2023-07-24 13:10:42 -07:00
lllong33
ff5cb7ba51 Fixed double underline (#7944)
* Fixed double-underline
* backward compatibility postgres_get_relations
* Remove invalid comments
* compatibility adapter and get_relation
* fix generic for call
* fix adapter dispatch grammar issue
2023-07-21 17:28:27 -04:00
Quigley Malcolm
1e2b9ae962 [CT-1849] _connection_exception_retry handles EOFError exceptions (#8182)
* Add test for checking that `_connection_exception_retry` handles `EOFError`s

* Update `_connection_exception_retry` to handle `EOFError` exceptions

* Add changie docs for `_connection_exception_retry` handling `EOFError` exceptions
2023-07-21 11:34:06 -07:00
Michelle Ark
8cab58d248 Add implementation issue template (#8176) 2023-07-21 13:50:53 -04:00
Chenyu Li
0d645c227f add a node status (#8174) 2023-07-20 14:38:01 -07:00
Pádraic Slattery
fb6c349677 Correcting spelling of partition (#8101)
* Correcting spelling of partition

* Changie entry
2023-07-20 11:05:11 -07:00
Kshitij Aranke
eeb057085c Hotfix for 372: Use JSONEncoder in json.dumps (#8151) 2023-07-19 18:32:47 -05:00
Michelle Ark
121371f4a4 format exception from dbtPlugin.initialize (#8143) 2023-07-19 17:39:26 -04:00
Gerda Shank
a32713198b Rearrange pp_versioned_models test (#8150) 2023-07-19 17:05:09 -04:00
Jeremy Cohen
a1b067c683 Rm docs changelog entry (#8147) 2023-07-19 12:53:14 +02:00
leahwicz
22c40a4766 Update release-docker.yml 2023-05-03 09:33:31 -04:00
leahwicz
bcf140b3c1 Update release-docker.yml 2023-05-02 23:21:08 -04:00
leahwicz
e3692a6a3d Update release.yml 2023-05-02 23:11:40 -04:00
leahwicz
e7489383a2 Update release-docker.yml 2023-05-02 23:11:27 -04:00
leahwicz
70246c3f86 Update release.yml 2023-05-02 22:50:35 -04:00
leahwicz
0796c84da5 Update release-docker.yml 2023-05-02 22:50:18 -04:00
leahwicz
718482fb02 Update release.yml 2023-05-02 22:46:13 -04:00
leahwicz
a3fb66daa4 Update release-docker.yml 2023-05-02 22:45:41 -04:00
leahwicz
da34b80c26 Update release-docker.yml 2023-05-02 22:43:59 -04:00
leahwicz
ba5ab21140 Update release-docker.yml 2023-05-02 22:27:57 -04:00
leahwicz
65f41a1e36 Update testing.yml 2023-05-02 22:12:39 -04:00
leahwicz
0930c9c059 Update release.yml 2023-05-02 22:08:11 -04:00
leahwicz
1d193a9ab9 Update release-docker.yml 2023-05-02 22:07:53 -04:00
leahwicz
3adc6dca61 Update release-docker.yml 2023-05-02 22:03:15 -04:00
leahwicz
36d9f841d6 Update release-docker.yml 2023-05-02 22:01:27 -04:00
leahwicz
48ad13de00 Update release.yml 2023-05-02 21:57:19 -04:00
leahwicz
42935cce05 Update release.yml 2023-05-02 21:52:56 -04:00
leahwicz
e77f1c3b0f Update release.yml 2023-05-02 19:24:59 -04:00
leahwicz
388838aa99 Update testing.yml 2023-05-02 19:20:06 -04:00
leahwicz
d4d0990072 Update release.yml 2023-05-02 19:04:40 -04:00
leahwicz
4210d17f14 Update testing.yml 2023-05-02 19:03:15 -04:00
leahwicz
fbd12e78c9 Update testing.yml 2023-05-02 18:40:41 -04:00
leahwicz
83d3421e72 Update testing.yml 2023-05-02 17:55:56 -04:00
leahwicz
8bcbf73aaa Update release-docker.yml 2023-05-02 17:31:20 -04:00
leahwicz
cc5f15885d Update release.yml 2023-05-02 17:29:33 -04:00
leahwicz
20fdf55bf6 Update testing.yml 2023-05-02 17:15:07 -04:00
leahwicz
955dcec68b Update testing.yml 2023-05-02 17:03:23 -04:00
leahwicz
2b8564b16f Update testing.yml 2023-05-02 16:55:21 -04:00
leahwicz
57da3e51cd Update testing.yml 2023-05-02 13:14:50 -04:00
leahwicz
dede0e9747 Update testing.yml 2023-05-02 13:11:57 -04:00
leahwicz
35d2fc1158 Update testing.yml 2023-05-02 13:10:04 -04:00
leahwicz
c5267335a3 Update testing.yml 2023-05-02 13:08:00 -04:00
leahwicz
15c7b589c2 Update testing.yml 2023-05-02 13:06:35 -04:00
leahwicz
0ada5e8bf7 Create testing.yml 2023-05-02 12:17:26 -04:00
leahwicz
412ac8d1b9 Update release.yml 2023-04-19 09:08:39 -04:00
leahwicz
5df501a281 Update release.yml 2023-04-18 21:48:49 -04:00
leahwicz
3e4c61d020 Update release.yml 2023-04-18 21:46:26 -04:00
leahwicz
cc39fe51b3 Update release-docker.yml 2023-04-18 21:44:08 -04:00
leahwicz
89cd24388d Update release.yml 2023-04-18 21:40:55 -04:00
leahwicz
d5da0a8093 Update release.yml 2023-04-18 21:36:29 -04:00
leahwicz
88ae1f8871 Update release-docker.yml 2023-04-18 09:15:19 -04:00
leahwicz
50b3d1deaa Update release-docker.yml 2023-04-17 21:22:55 -04:00
leahwicz
3b3def5b8a Update release-docker.yml 2023-04-17 21:11:34 -04:00
leahwicz
4f068a45ff Update release-docker.yml 2023-04-17 21:06:40 -04:00
leahwicz
23a9504a51 Update release-docker.yml 2023-04-17 21:03:29 -04:00
leahwicz
d0d4eba477 Update release-docker.yml 2023-04-17 21:02:10 -04:00
leahwicz
a3fab0b5a9 Update release-docker.yml 2023-04-17 20:57:20 -04:00
647 changed files with 50227 additions and 11459 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.7.0a1
current_version = 1.8.0a1
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

View File

@@ -3,6 +3,7 @@
For information on prior major and minor releases, see their changelogs:
* [1.7](https://github.com/dbt-labs/dbt-core/blob/1.7.latest/CHANGELOG.md)
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Remove adapter.get_compiler interface
time: 2023-11-27T11:47:57.443202-05:00
custom:
Author: michelleark
Issue: "9148"

View File

@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Move AdapterLogger to adapters folder
time: 2023-11-28T13:43:56.853925-08:00
custom:
Author: colin-rogers-dbt
Issue: "9151"

View File

@@ -0,0 +1,7 @@
kind: Breaking Changes
body: move event manager setup back to core, remove ref to global EVENT_MANAGER and
clean up event manager functions
time: 2023-11-30T13:53:48.645192-08:00
custom:
Author: colin-rogers-dbt
Issue: "9150"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Begin using DSI 0.4.x
time: 2023-10-31T13:19:54.750009-07:00
custom:
Author: QMalcolm peterallenwebb
PR: "8892"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Update typing-extensions version to >=4.4
time: 2023-11-06T13:00:51.062386-08:00
custom:
Author: tlento
PR: "9012"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Fix for column tests not rendering on quoted columns
time: 2023-05-31T11:54:19.687363-04:00
custom:
Author: drewbanin
Issue: "201"

View File

@@ -1,6 +0,0 @@
kind: Docs
body: Remove static SQL codeblock for metrics
time: 2023-07-18T19:24:22.155323+02:00
custom:
Author: marcodamore
Issue: "436"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: fix get_custom_database docstring
time: 2023-11-06T12:31:57.525711Z
custom:
Author: LeoTheGriff
Issue: "9003"

View File

@@ -0,0 +1,6 @@
kind: Features
body: 'Allow adapters to include package logs in dbt standard logging '
time: 2023-09-15T12:37:33.862862-07:00
custom:
Author: colin-rogers-dbt
Issue: "7859"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add drop_schema_named macro
time: 2023-10-17T14:36:20.612289-07:00
custom:
Author: colin-rogers-dbt
Issue: "8025"

View File

@@ -0,0 +1,6 @@
kind: Features
body: migrate utils to common and adapters folders
time: 2023-10-26T11:08:21.458709-07:00
custom:
Author: colin-rogers-dbt
Issue: "8924"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Move Agate helper client into common
time: 2023-10-26T12:35:56.538587-07:00
custom:
Author: MichelleArk
Issue: "8926"

View File

@@ -0,0 +1,6 @@
kind: Features
body: remove usage of dbt.config.PartialProject from dbt/adapters
time: 2023-10-26T12:39:13.904116-07:00
custom:
Author: MichelleArk
Issue: "8928"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add exports to SavedQuery spec
time: 2023-10-31T13:20:22.448158-07:00
custom:
Author: QMalcolm peterallenwebb
Issue: "8892"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Remove legacy logger
time: 2023-11-07T13:56:35.186648-08:00
custom:
Author: colin-rogers-dbt
Issue: "8027"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support setting export configs hierarchically via saved query and project configs
time: 2023-11-10T15:42:55.042317-08:00
custom:
Author: QMalcolm
Issue: "8956"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support --empty flag for schema-only dry runs
time: 2023-11-16T23:40:49.96651-05:00
custom:
Author: michelleark
Issue: "8971"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Enable converting deprecation warnings to errors
time: 2023-07-18T12:55:18.03914-04:00
custom:
Author: michelleark
Issue: "8130"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: For packages installed with tarball method, fetch metadata to resolve nested dependencies
time: 2023-10-13T13:09:43.188308-04:00
custom:
Author: adamlopez
Issue: "8621"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix partial parsing not working for semantic model change
time: 2023-10-16T16:39:53.05058-07:00
custom:
Author: ChenyuLInx
Issue: "8859"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Handle unknown `type_code` for model contracts
time: 2023-10-24T11:01:51.980781-06:00
custom:
Author: dbeatty10
Issue: 8877 8353

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add back contract enforcement for temporary tables on postgres
time: 2023-10-24T14:55:04.051683-05:00
custom:
Author: emmyoop
Issue: "8857"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Rework get_catalog implementation to retain previous adapter interface semantics
time: 2023-10-24T15:54:00.628086-04:00
custom:
Author: peterallenwebb
Issue: "8846"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add version to fqn when version==0
time: 2023-10-26T00:25:36.259356-05:00
custom:
Author: aranke
Issue: "8836"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix cased comparison in catalog-retrieval function.
time: 2023-10-30T09:37:34.258612-04:00
custom:
Author: peterallenwebb
Issue: "8939"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Catalog queries now assign the correct type to materialized views
time: 2023-10-31T00:53:45.486203-04:00
custom:
Author: mikealfare
Issue: "8864"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix compilation exception running empty seed file and support new Integer agate data_type
time: 2023-10-31T14:48:37.774871-04:00
custom:
Author: gshank
Issue: "8895"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Make relation filtering None-tolerant for maximal flexibility across adapters.
time: 2023-11-01T15:58:24.552054-04:00
custom:
Author: peterallenwebb
Issue: "8974"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Update run_results.json from previous versions of dbt to support deferral and
rerun from failure
time: 2023-11-06T15:59:33.677915-05:00
custom:
Author: jtcohen6 peterallenwebb
Issue: "9010"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix git repository with subdirectory for Deps
time: 2023-11-07T09:23:58.214271-08:00
custom:
Author: ChenyuLInx
Issue: "9000"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Use MANIFEST.in to recursively include all jinja templates; fixes issue where
some templates were not included in the distribution
time: 2023-11-07T09:41:30.121733-05:00
custom:
Author: mikealfare
Issue: "9016"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix formatting of tarball information in packages-lock.yml
time: 2023-11-13T11:49:56.437007-08:00
custom:
Author: ChenyuLInx QMalcolm
Issue: "9062"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'deps: Lock git packages to commit SHA during resolution'
time: 2023-11-27T15:43:10.122069+01:00
custom:
Author: jtcohen6
Issue: "9050"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'deps: Use PackageRenderer to read package-lock.json'
time: 2023-11-27T15:43:47.842423+01:00
custom:
Author: jtcohen6
Issue: "9127"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Ensure we produce valid jsonschema schemas for manifest, catalog, run-results,
and sources
time: 2023-11-27T16:52:44.590313-08:00
custom:
Author: QMalcolm
Issue: "8991"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: 'Get sources working again in dbt docs generate'
time: 2023-11-28T15:52:25.738256Z
custom:
Author: aranke
Issue: "9119"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Added more type annotations.
time: 2023-08-31T16:44:35.737954-04:00
custom:
Author: peterallenwebb
Issue: "8537"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.include.global_project in dbt/adapters
time: 2023-10-26T18:49:53.36449-04:00
custom:
Author: michelleark
Issue: "8925"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add a no-op runner for Saved Qeury
time: 2023-10-27T14:00:48.4755-07:00
custom:
Author: ChenyuLInx
Issue: "8893"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: remove dbt.flags.MP_CONTEXT usage in dbt/adapters
time: 2023-11-01T10:27:58.790153-04:00
custom:
Author: michelleark
Issue: "8967"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: 'Remove usage of dbt.flags.LOG_CACHE_EVENTS in dbt/adapters'
time: 2023-11-01T17:31:24.974093-04:00
custom:
Author: michelleark
Issue: "8969"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move CatalogRelationTypes test case to the shared test suite to be reused by
adapter maintainers
time: 2023-11-03T19:52:22.694394-04:00
custom:
Author: mikealfare
Issue: "8952"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Treat SystemExit as an interrupt if raised during node execution.
time: 2023-11-06T08:04:22.022179-05:00
custom:
Author: benmosher
Issue: n/a

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Removing unused 'documentable'
time: 2023-11-06T10:57:30.694056-08:00
custom:
Author: QMalcolm
Issue: "8871"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove use of dbt/core exceptions in dbt/adapter
time: 2023-11-07T13:57:28.683727-08:00
custom:
Author: colin-rogers-dbt MichelleArk
Issue: "8920"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Cache dbt plugin modules to improve integration test performance
time: 2023-11-07T19:15:46.170151-05:00
custom:
Author: peterallenwebb
Issue: "9029"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Fix test_current_timestamp_matches_utc test; allow for MacOS runner system clock
variance
time: 2023-11-11T17:53:50.098843-05:00
custom:
Author: mikealfare
Issue: "9057"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific
event types and protos
time: 2023-11-16T17:42:51.005023-05:00
custom:
Author: michelleark
Issue: 8927 8918

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Clean up unused adaptor folders
time: 2023-11-20T13:47:35.923794-08:00
custom:
Author: ChenyuLInx
Issue: "9123"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move column constraints into common/contracts, removing another dependency of
adapters on core.
time: 2023-11-20T18:32:14.859503-05:00
custom:
Author: peterallenwebb
Issue: "9024"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move dbt.semver to dbt.common.semver and update references.
time: 2023-11-28T17:07:32.172421-08:00
custom:
Author: versusfacit
Issue: "9039"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move lowercase utils method to common
time: 2023-11-30T13:54:32.561673-08:00
custom:
Author: colin-rogers-dbt
Issue: "9180"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usages of dbt.clients.jinja in dbt/adapters
time: 2023-12-05T09:35:44.845352+09:00
custom:
Author: michelleark
Issue: "9205"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.contracts in dbt/adapters
time: 2023-12-05T12:05:59.936775+09:00
custom:
Author: michelleark
Issue: "9208"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.contracts.graph.nodes.ResultNode in dbt/adapters
time: 2023-12-05T16:58:12.932172+09:00
custom:
Author: michelleark
Issue: "9214"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Introduce RelationConfig Protocol, consolidate Relation.create_from
time: 2023-12-05T17:07:25.33861+09:00
custom:
Author: michelleark
Issue: "9215"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move BaseConfig, Metadata and various other contract classes from model_config
to common/contracts/config
time: 2023-12-05T18:50:22.321229-08:00
custom:
Author: colin-rorgers-dbt
Issue: "8919"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: remove manifest from adapter.set_relations_cache signature
time: 2023-12-05T23:58:30.920144+09:00
custom:
Author: michelleark
Issue: "9217"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: ' remove manifest from adapter catalog method signatures'
time: 2023-12-06T00:03:43.824252+09:00
custom:
Author: michelleark
Issue: "9218"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add MacroResolverProtocol, remove lazy loading of manifest in adapter.execute_macro
time: 2023-12-07T11:15:54.427818+09:00
custom:
Author: michelleark
Issue: "9244"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: add macro_context_generator on adapter
time: 2023-12-08T00:48:54.506911+09:00
custom:
Author: michelleark
Issue: "9247"

View File

@@ -10,3 +10,5 @@ ignore =
E741
E501 # long line checking is done in black
exclude = test/
per-file-ignores =
*/__init__.py: F401

2
.gitattributes vendored
View File

@@ -1,4 +1,4 @@
core/dbt/include/index.html binary
core/dbt/task/docs/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary
core/dbt/docs/build/html/searchindex.js binary
core/dbt/docs/build/html/index.html binary

19
.github/CODEOWNERS vendored
View File

@@ -13,23 +13,6 @@
# the core team as a whole will be assigned
* @dbt-labs/core-team
### OSS Tooling Guild
/.github/ @dbt-labs/guild-oss-tooling
.bumpversion.cfg @dbt-labs/guild-oss-tooling
.changie.yaml @dbt-labs/guild-oss-tooling
pre-commit-config.yaml @dbt-labs/guild-oss-tooling
pytest.ini @dbt-labs/guild-oss-tooling
tox.ini @dbt-labs/guild-oss-tooling
pyproject.toml @dbt-labs/guild-oss-tooling
requirements.txt @dbt-labs/guild-oss-tooling
dev_requirements.txt @dbt-labs/guild-oss-tooling
/core/setup.py @dbt-labs/guild-oss-tooling
/core/MANIFEST.in @dbt-labs/guild-oss-tooling
### ADAPTERS
# Adapter interface ("base" + "sql" adapter defaults, cache)
@@ -40,7 +23,7 @@ dev_requirements.txt @dbt-labs/guild-oss-tooling
# Postgres plugin
/plugins/ @dbt-labs/core-adapters
/plugins/postgres/setup.py @dbt-labs/core-adapters @dbt-labs/guild-oss-tooling
/plugins/postgres/setup.py @dbt-labs/core-adapters
# Functional tests for adapter plugins
/tests/adapter @dbt-labs/core-adapters

View File

@@ -0,0 +1,58 @@
name: 🛠️ Implementation
description: This is an implementation ticket intended for use by the maintainers of dbt-core
title: "[<project>] <title>"
labels: ["user docs"]
body:
- type: markdown
attributes:
value: This is an implementation ticket intended for use by the maintainers of dbt-core
- type: checkboxes
attributes:
label: Housekeeping
description: >
A couple friendly reminders:
1. Remove the `user docs` label if the scope of this work does not require changes to https://docs.getdbt.com/docs: no end-user interface (e.g. yml spec, CLI, error messages, etc) or functional changes
2. Link any blocking issues in the "Blocked on" field under the "Core devs & maintainers" project.
options:
- label: I am a maintainer of dbt-core
required: true
- type: textarea
attributes:
label: Short description
description: |
Describe the scope of the ticket, a high-level implementation approach and any tradeoffs to consider
validations:
required: true
- type: textarea
attributes:
label: Acceptance criteria
description: |
What is the definition of done for this ticket? Include any relevant edge cases and/or test cases
validations:
required: true
- type: textarea
attributes:
label: Impact to Other Teams
description: |
Will this change impact other teams? Include details of the kinds of changes required (new tests, code changes, related tickets) and _add the relevant `Impact:[team]` label_.
placeholder: |
Example: This change impacts `dbt-redshift` because the tests will need to be modified. The `Impact:[Adapter]` label has been added.
validations:
required: true
- type: textarea
attributes:
label: Will backports be required?
description: |
Will this change need to be backported to previous versions? Add details, possible blockers to backporting and _add the relevant backport labels `backport 1.x.latest`_
placeholder: |
Example: Backport to 1.6.latest, 1.5.latest and 1.4.latest. Since 1.4 isn't using click, the backport may be complicated. The `backport 1.6.latest`, `backport 1.5.latest` and `backport 1.4.latest` labels have been added.
validations:
required: true
- type: textarea
attributes:
label: Context
description: |
Provide the "why", motivation, and alternative approaches considered -- linking to previous refinement issues, spikes, Notion docs as appropriate
validations:
validations:
required: false

View File

@@ -28,3 +28,10 @@ updates:
schedule:
interval: "weekly"
rebase-strategy: "disabled"
# github dependencies
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
rebase-strategy: "disabled"

View File

@@ -1,15 +1,12 @@
resolves #
[docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose) dbt-labs/docs.getdbt.com/#
resolves #
<!---
Include the number of the issue addressed by this PR above if applicable.
PRs for code changes without an associated issue *will not be merged*.
See CONTRIBUTING.md for more information.
Include the number of the docs issue that was opened for this PR. If
this change has no user-facing implications, "N/A" suffices instead. New
docs tickets can be created by clicking the link above or by going to
https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose.
Add the `user docs` label to this PR if it will need docs changes. An
issue will get opened in docs.getdbt.com upon successful merge of this PR.
-->
### Problem
@@ -33,3 +30,4 @@ resolves #
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [ ] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions

View File

@@ -2,10 +2,8 @@
# Checks that a file has been committed under the /.changes directory
# as a new CHANGELOG entry. Cannot check for a specific filename as
# it is dynamically generated by change type and timestamp.
# This workflow should not require any secrets since it runs for PRs
# from forked repos.
# By default, secrets are not passed to workflows running from
# a forked repo.
# This workflow runs on pull_request_target because it requires
# secrets to post comments.
# **why?**
# Ensure code change gets reflected in the CHANGELOG.
@@ -19,7 +17,7 @@
name: Check Changelog Entry
on:
pull_request:
pull_request_target:
types: [opened, reopened, labeled, unlabeled, synchronize]
workflow_dispatch:

43
.github/workflows/docs-issue.yml vendored Normal file
View File

@@ -0,0 +1,43 @@
# **what?**
# Open an issue in docs.getdbt.com when a PR is labeled `user docs`
# **why?**
# To reduce barriers for keeping docs up to date
# **when?**
# When a PR is labeled `user docs` and is merged. Runs on pull_request_target to run off the workflow already merged,
# not the workflow that existed on the PR branch. This allows old PRs to get comments.
name: Open issues in docs.getdbt.com repo when a PR is labeled
run-name: "Open an issue in docs.getdbt.com for PR #${{ github.event.pull_request.number }}"
on:
pull_request_target:
types: [labeled, closed]
defaults:
run:
shell: bash
permissions:
issues: write # opens new issues
pull-requests: write # comments on PRs
jobs:
open_issues:
# we only want to run this when the PR has been merged or the label in the labeled event is `user docs`. Otherwise it runs the
# risk of duplicaton of issues being created due to merge and label both triggering this workflow to run and neither having
# generating the comment before the other runs. This lives here instead of the shared workflow because this is where we
# decide if it should run or not.
if: |
(github.event.pull_request.merged == true) &&
((github.event.action == 'closed' && contains( github.event.pull_request.labels.*.name, 'user docs')) ||
(github.event.action == 'labeled' && github.event.label.name == 'user docs'))
uses: dbt-labs/actions/.github/workflows/open-issue-in-repo.yml@main
with:
issue_repository: "dbt-labs/docs.getdbt.com"
issue_title: "Docs Changes Needed from ${{ github.event.repository.name }} PR #${{ github.event.pull_request.number }}"
issue_body: "At a minimum, update body to include a link to the page on docs.getdbt.com requiring updates and what part(s) of the page you would like to see updated."
secrets: inherit

View File

@@ -33,6 +33,11 @@ defaults:
run:
shell: bash
# top-level adjustments can be made here
env:
# number of parallel processes to spawn for python integration testing
PYTHON_INTEGRATION_TEST_WORKERS: 5
jobs:
code-quality:
name: code-quality
@@ -103,26 +108,59 @@ jobs:
- name: Upload Unit Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: unit
integration-metadata:
name: integration test metadata generation
runs-on: ubuntu-latest
outputs:
split-groups: ${{ steps.generate-split-groups.outputs.split-groups }}
include: ${{ steps.generate-include.outputs.include }}
steps:
- name: generate split-groups
id: generate-split-groups
run: |
MATRIX_JSON="["
for B in $(seq 1 ${{ env.PYTHON_INTEGRATION_TEST_WORKERS }}); do
MATRIX_JSON+=$(sed 's/^/"/;s/$/"/' <<< "${B}")
done
MATRIX_JSON="${MATRIX_JSON//\"\"/\", \"}"
MATRIX_JSON+="]"
echo "split-groups=${MATRIX_JSON}"
echo "split-groups=${MATRIX_JSON}" >> $GITHUB_OUTPUT
- name: generate include
id: generate-include
run: |
INCLUDE=('"python-version":"3.8","os":"windows-latest"' '"python-version":"3.8","os":"macos-latest"' )
INCLUDE_GROUPS="["
for include in ${INCLUDE[@]}; do
for group in $(seq 1 ${{ env.PYTHON_INTEGRATION_TEST_WORKERS }}); do
INCLUDE_GROUPS+=$(sed 's/$/, /' <<< "{\"split-group\":\"${group}\",${include}}")
done
done
INCLUDE_GROUPS=$(echo $INCLUDE_GROUPS | sed 's/,*$//g')
INCLUDE_GROUPS+="]"
echo "include=${INCLUDE_GROUPS}"
echo "include=${INCLUDE_GROUPS}" >> $GITHUB_OUTPUT
integration:
name: integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
name: (${{ matrix.split-group }}) integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 60
timeout-minutes: 30
needs:
- integration-metadata
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-20.04]
include:
- python-version: 3.8
os: windows-latest
- python-version: 3.8
os: macos-latest
split-group: ${{ fromJson(needs.integration-metadata.outputs.split-groups) }}
include: ${{ fromJson(needs.integration-metadata.outputs.include) }}
env:
TOXENV: integration
DBT_INVOCATION_ENV: github-actions
@@ -165,6 +203,8 @@ jobs:
- name: Run tests
run: tox -- --ddtrace
env:
PYTEST_ADDOPTS: ${{ format('--splits {0} --group {1}', env.PYTHON_INTEGRATION_TEST_WORKERS, matrix.split-group) }}
- name: Get current date
if: always()
@@ -182,8 +222,26 @@ jobs:
- name: Upload Integration Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: integration
integration-report:
if: ${{ always() }}
name: Integration Test Suite
runs-on: ubuntu-latest
needs: integration
steps:
- name: "Integration Tests Failed"
if: ${{ contains(needs.integration.result, 'failure') || contains(needs.integration.result, 'cancelled') }}
# when this is true the next step won't execute
run: |
echo "::notice title='Integration test suite failed'"
exit 1
- name: "Integration Tests Passed"
run: |
echo "::notice title='Integration test suite passed'"
build:
name: build packages

View File

@@ -83,7 +83,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push MAJOR.MINOR.PATCH tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v5
with:
file: docker/Dockerfile
push: True
@@ -94,7 +94,7 @@ jobs:
ghcr.io/dbt-labs/${{ github.event.inputs.package }}:${{ github.event.inputs.version_number }}
- name: Build and push MINOR.latest tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v5
if: ${{ needs.get_version_meta.outputs.minor_latest == 'True' }}
with:
file: docker/Dockerfile
@@ -106,7 +106,7 @@ jobs:
ghcr.io/dbt-labs/${{ github.event.inputs.package }}:${{ needs.get_version_meta.outputs.major }}.${{ needs.get_version_meta.outputs.minor }}.latest
- name: Build and push latest tag
uses: docker/build-push-action@v4
uses: docker/build-push-action@v5
if: ${{ needs.get_version_meta.outputs.latest == 'True' }}
with:
file: docker/Dockerfile

View File

@@ -0,0 +1,30 @@
# **what?**
# Cleanup branches left over from automation and testing. Also cleanup
# draft releases from release testing.
# **why?**
# The automations are leaving behind branches and releases that clutter
# the repository. Sometimes we need them to debug processes so we don't
# want them immediately deleted. Running on Saturday to avoid running
# at the same time as an actual release to prevent breaking a release
# mid-release.
# **when?**
# Mainly on a schedule of 12:00 Saturday.
# Manual trigger can also run on demand
name: Repository Cleanup
on:
schedule:
- cron: '0 12 * * SAT' # At 12:00 on Saturday - details in `why` above
workflow_dispatch: # for manual triggering
permissions:
contents: write
jobs:
cleanup-repo:
uses: dbt-labs/actions/.github/workflows/repository-cleanup.yml@main
secrets: inherit

View File

@@ -18,11 +18,41 @@ on:
permissions: read-all
# top-level adjustments can be made here
env:
# number of parallel processes to spawn for python testing
PYTHON_INTEGRATION_TEST_WORKERS: 5
jobs:
integration-metadata:
name: integration test metadata generation
runs-on: ubuntu-latest
outputs:
split-groups: ${{ steps.generate-split-groups.outputs.split-groups }}
steps:
- name: generate split-groups
id: generate-split-groups
run: |
MATRIX_JSON="["
for B in $(seq 1 ${{ env.PYTHON_INTEGRATION_TEST_WORKERS }}); do
MATRIX_JSON+=$(sed 's/^/"/;s/$/"/' <<< "${B}")
done
MATRIX_JSON="${MATRIX_JSON//\"\"/\", \"}"
MATRIX_JSON+="]"
echo "split-groups=${MATRIX_JSON}" >> $GITHUB_OUTPUT
# run the performance measurements on the current or default branch
test-schema:
name: Test Log Schema
runs-on: ubuntu-20.04
timeout-minutes: 30
needs:
- integration-metadata
strategy:
fail-fast: false
matrix:
split-group: ${{ fromJson(needs.integration-metadata.outputs.split-groups) }}
env:
# turns warnings into errors
RUSTFLAGS: "-D warnings"
@@ -65,3 +95,14 @@ jobs:
# we actually care if these pass, because the normal test run doesn't usually include many json log outputs
- name: Run integration tests
run: tox -e integration -- -nauto
env:
PYTEST_ADDOPTS: ${{ format('--splits {0} --group {1}', env.PYTHON_INTEGRATION_TEST_WORKERS, matrix.split-group) }}
test-schema-report:
name: Log Schema Test Suite
runs-on: ubuntu-latest
needs: test-schema
steps:
- name: "[Notification] Log test suite passes"
run: |
echo "::notice title="Log test suite passes""

View File

@@ -1,7 +1,7 @@
# Configuration for pre-commit hooks (see https://pre-commit.com/).
# Eventually the hooks described here will be run as tests before merging each PR.
exclude: ^(core/dbt/docs/build/|core/dbt/events/types_pb2.py)
exclude: ^(core/dbt/docs/build/|core/dbt/common/events/types_pb2.py|core/dbt/events/core_types_pb2.py|core/dbt/adapters/events/adapter_types_pb2.py)
# Force all unspecified python hooks to run python 3.8
default_language_version:
@@ -37,7 +37,7 @@ repos:
alias: flake8-check
stages: [manual]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.3.0
rev: v1.4.1
hooks:
- id: mypy
# N.B.: Mypy is... a bit fragile.

View File

@@ -26,7 +26,7 @@ Legacy tests are found in the 'test' directory:
The "tasks" map to top-level dbt commands. So `dbt run` => task.run.RunTask, etc. Some are more like abstract base classes (GraphRunnableTask, for example) but all the concrete types outside of task should map to tasks. Currently one executes at a time. The tasks kick off their “Runners” and those do execute in parallel. The parallelism is managed via a thread pool, in GraphRunnableTask.
core/dbt/include/index.html
core/dbt/task/docs/index.html
This is the docs website code. It comes from the dbt-docs repository, and is generated when a release is packaged.
## Adapters

View File

@@ -10,6 +10,7 @@
For information on prior major and minor releases, see their changelogs:
* [1.7](https://github.com/dbt-labs/dbt-core/blob/1.7.latest/CHANGELOG.md)
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
* [1.4](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)

View File

@@ -40,7 +40,16 @@ dev: dev_req ## Installs dbt-* packages in develop mode along with development d
.PHONY: proto_types
proto_types: ## generates google protobuf python file from types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/types.proto
protoc -I=./core/dbt/common/events --python_out=./core/dbt/common/events ./core/dbt/common/events/types.proto
.PHONY: core_proto_types
core_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/core_types.proto
.PHONY: adapter_proto_types
adapter_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/adapters/events --python_out=./core/dbt/adapters/events ./core/dbt/adapters/events/adapter_types.proto
.PHONY: mypy
mypy: .env ## Runs mypy against staged changes for static type checking.

View File

@@ -0,0 +1,13 @@
ignore:
- ".github"
- ".changes"
coverage:
status:
project:
default:
target: auto
threshold: 0.1% # Reduce noise by ignoring rounding errors in coverage drops
patch:
default:
target: auto
threshold: 80%

View File

@@ -1,7 +1,7 @@
# these are all just exports, #noqa them so flake8 will be happy
# TODO: Should we still include this in the `adapters` namespace?
from dbt.contracts.connection import Credentials # noqa: F401
from dbt.adapters.contracts.connection import Credentials # noqa: F401
from dbt.adapters.base.meta import available # noqa: F401
from dbt.adapters.base.connections import BaseConnectionManager # noqa: F401
from dbt.adapters.base.relation import ( # noqa: F401

View File

@@ -2,17 +2,17 @@ from dataclasses import dataclass
import re
from typing import Dict, ClassVar, Any, Optional
from dbt.exceptions import DbtRuntimeError
from dbt.common.exceptions import DbtRuntimeError
@dataclass
class Column:
# Note: This is automatically used by contract code
# No-op conversions (INTEGER => INT) have been removed.
# Any adapter that wants to take advantage of "translate_type"
# should create a ClassVar with the appropriate conversions.
TYPE_LABELS: ClassVar[Dict[str, str]] = {
"STRING": "TEXT",
"TIMESTAMP": "TIMESTAMP",
"FLOAT": "FLOAT",
"INTEGER": "INT",
"BOOLEAN": "BOOLEAN",
}
column: str
dtype: str

View File

@@ -6,6 +6,7 @@ import traceback
# multiprocessing.RLock is a function returning this type
from multiprocessing.synchronize import RLock
from multiprocessing.context import SpawnContext
from threading import get_ident
from typing import (
Any,
@@ -23,8 +24,9 @@ from typing import (
import agate
import dbt.exceptions
from dbt.contracts.connection import (
import dbt.adapters.exceptions
import dbt.common.exceptions.base
from dbt.adapters.contracts.connection import (
Connection,
Identifier,
ConnectionState,
@@ -36,9 +38,9 @@ from dbt.contracts.graph.manifest import Manifest
from dbt.adapters.base.query_headers import (
MacroQueryStringSetter,
)
from dbt.events import AdapterLogger
from dbt.events.functions import fire_event
from dbt.events.types import (
from dbt.adapters.events.logging import AdapterLogger
from dbt.common.events.functions import fire_event
from dbt.adapters.events.types import (
NewConnection,
ConnectionReused,
ConnectionLeftOpenInCleanup,
@@ -48,9 +50,8 @@ from dbt.events.types import (
Rollback,
RollbackFailed,
)
from dbt.events.contextvars import get_node_info
from dbt import flags
from dbt.utils import cast_to_str
from dbt.common.events.contextvars import get_node_info
from dbt.common.utils import cast_to_str
SleepTime = Union[int, float] # As taken by time.sleep.
AdapterHandle = Any # Adapter connection handle objects can be any class.
@@ -72,10 +73,10 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
TYPE: str = NotImplemented
def __init__(self, profile: AdapterRequiredConfig):
def __init__(self, profile: AdapterRequiredConfig, mp_context: SpawnContext) -> None:
self.profile = profile
self.thread_connections: Dict[Hashable, Connection] = {}
self.lock: RLock = flags.MP_CONTEXT.RLock()
self.lock: RLock = mp_context.RLock()
self.query_header: Optional[MacroQueryStringSetter] = None
def set_query_header(self, manifest: Manifest) -> None:
@@ -91,13 +92,15 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
key = self.get_thread_identifier()
with self.lock:
if key not in self.thread_connections:
raise dbt.exceptions.InvalidConnectionError(key, list(self.thread_connections))
raise dbt.adapters.exceptions.InvalidConnectionError(
key, list(self.thread_connections)
)
return self.thread_connections[key]
def set_thread_connection(self, conn: Connection) -> None:
key = self.get_thread_identifier()
if key in self.thread_connections:
raise dbt.exceptions.DbtInternalError(
raise dbt.common.exceptions.DbtInternalError(
"In set_thread_connection, existing connection exists for {}"
)
self.thread_connections[key] = conn
@@ -137,13 +140,13 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:return: A context manager that handles exceptions raised by the
underlying database.
"""
raise dbt.exceptions.NotImplementedError(
raise dbt.common.exceptions.base.NotImplementedError(
"`exception_handler` is not implemented for this adapter!"
)
def set_connection_name(self, name: Optional[str] = None) -> Connection:
"""Called by 'acquire_connection' in BaseAdapter, which is called by
'connection_named', called by 'connection_for(node)'.
'connection_named'.
Creates a connection for this thread if one doesn't already
exist, and will rename an existing connection."""
@@ -220,14 +223,14 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:param int _attempts: Parameter used to keep track of the number of attempts in calling the
connect function across recursive calls. Passed as an argument to retry_timeout if it
is a Callable. This parameter should not be set by the initial caller.
:raises dbt.exceptions.FailedToConnectError: Upon exhausting all retry attempts without
:raises dbt.adapters.exceptions.FailedToConnectError: Upon exhausting all retry attempts without
successfully acquiring a handle.
:return: The given connection with its appropriate state and handle attributes set
depending on whether we successfully acquired a handle or not.
"""
timeout = retry_timeout(_attempts) if callable(retry_timeout) else retry_timeout
if timeout < 0:
raise dbt.exceptions.FailedToConnectError(
raise dbt.adapters.exceptions.FailedToConnectError(
"retry_timeout cannot be negative or return a negative time."
)
@@ -235,7 +238,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
# This guard is not perfect others may add to the recursion limit (e.g. built-ins).
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError("retry_limit cannot be negative")
raise dbt.adapters.exceptions.FailedToConnectError("retry_limit cannot be negative")
try:
connection.handle = connect()
@@ -246,7 +249,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
if retry_limit <= 0:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError(str(e))
raise dbt.adapters.exceptions.FailedToConnectError(str(e))
logger.debug(
f"Got a retryable error when attempting to open a {cls.TYPE} connection.\n"
@@ -268,12 +271,12 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
except Exception as e:
connection.handle = None
connection.state = ConnectionState.FAIL
raise dbt.exceptions.FailedToConnectError(str(e))
raise dbt.adapters.exceptions.FailedToConnectError(str(e))
@abc.abstractmethod
def cancel_open(self) -> Optional[List[str]]:
"""Cancel all open connections on the adapter. (passable)"""
raise dbt.exceptions.NotImplementedError(
raise dbt.common.exceptions.base.NotImplementedError(
"`cancel_open` is not implemented for this adapter!"
)
@@ -288,7 +291,9 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
This should be thread-safe, or hold the lock if necessary. The given
connection should not be in either in_use or available.
"""
raise dbt.exceptions.NotImplementedError("`open` is not implemented for this adapter!")
raise dbt.common.exceptions.base.NotImplementedError(
"`open` is not implemented for this adapter!"
)
def release(self) -> None:
with self.lock:
@@ -320,12 +325,16 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
@abc.abstractmethod
def begin(self) -> None:
"""Begin a transaction. (passable)"""
raise dbt.exceptions.NotImplementedError("`begin` is not implemented for this adapter!")
raise dbt.common.exceptions.base.NotImplementedError(
"`begin` is not implemented for this adapter!"
)
@abc.abstractmethod
def commit(self) -> None:
"""Commit a transaction. (passable)"""
raise dbt.exceptions.NotImplementedError("`commit` is not implemented for this adapter!")
raise dbt.common.exceptions.base.NotImplementedError(
"`commit` is not implemented for this adapter!"
)
@classmethod
def _rollback_handle(cls, connection: Connection) -> None:
@@ -361,7 +370,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
def _rollback(cls, connection: Connection) -> None:
"""Roll back the given connection."""
if connection.transaction_open is False:
raise dbt.exceptions.DbtInternalError(
raise dbt.common.exceptions.DbtInternalError(
f"Tried to rollback transaction on connection "
f'"{connection.name}", but it does not have one open!'
)
@@ -400,7 +409,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
@abc.abstractmethod
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False
self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
) -> Tuple[AdapterResponse, agate.Table]:
"""Execute the given SQL.
@@ -408,7 +417,30 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:param bool auto_begin: If set, and dbt is not currently inside a
transaction, automatically begin one.
:param bool fetch: If set, fetch results.
:param int limit: If set, limits the result set
:return: A tuple of the query status and results (empty if fetch=False).
:rtype: Tuple[AdapterResponse, agate.Table]
"""
raise dbt.exceptions.NotImplementedError("`execute` is not implemented for this adapter!")
raise dbt.common.exceptions.base.NotImplementedError(
"`execute` is not implemented for this adapter!"
)
def add_select_query(self, sql: str) -> Tuple[Connection, Any]:
"""
This was added here because base.impl.BaseAdapter.get_column_schema_from_query expects it to be here.
That method wouldn't work unless the adapter used sql.impl.SQLAdapter, sql.connections.SQLConnectionManager
or defined this method on <Adapter>ConnectionManager before passing it in to <Adapter>Adapter.
See https://github.com/dbt-labs/dbt-core/issues/8396 for more information.
"""
raise dbt.common.exceptions.base.NotImplementedError(
"`add_select_query` is not implemented for this adapter!"
)
@classmethod
def data_type_code_to_name(cls, type_code: Union[int, str]) -> str:
"""Get the string representation of the data type from the type_code."""
# https://peps.python.org/pep-0249/#type-objects
raise dbt.common.exceptions.base.NotImplementedError(
"`data_type_code_to_name` is not implemented for this adapter!"
)

View File

@@ -4,12 +4,10 @@ from contextlib import contextmanager
from datetime import datetime
from enum import Enum
import time
from itertools import chain
from typing import (
Any,
Callable,
Dict,
Iterable,
Iterator,
List,
Mapping,
@@ -17,39 +15,56 @@ from typing import (
Set,
Tuple,
Type,
TypedDict,
Union,
FrozenSet,
Iterable,
)
from multiprocessing.context import SpawnContext
from dbt.contracts.graph.nodes import ColumnLevelConstraint, ConstraintType, ModelLevelConstraint
from dbt.adapters.capability import Capability, CapabilityDict
from dbt.common.contracts.constraints import (
ColumnLevelConstraint,
ConstraintType,
ModelLevelConstraint,
)
from dbt.adapters.contracts.macros import MacroClient
import agate
import pytz
from dbt.exceptions import (
from dbt.adapters.exceptions import (
SnapshotTargetIncompleteError,
SnapshotTargetNotSnapshotTableError,
NullRelationDropAttemptedError,
NullRelationCacheAttemptedError,
RelationReturnedMultipleResultsError,
UnexpectedNonTimestampError,
RenameToNoneAttemptedError,
QuoteConfigTypeError,
)
from dbt.common.exceptions import (
NotImplementedError,
DbtInternalError,
DbtRuntimeError,
DbtValidationError,
UnexpectedNullError,
MacroArgTypeError,
MacroResultError,
NotImplementedError,
NullRelationCacheAttemptedError,
NullRelationDropAttemptedError,
QuoteConfigTypeError,
RelationReturnedMultipleResultsError,
RenameToNoneAttemptedError,
SnapshotTargetIncompleteError,
SnapshotTargetNotSnapshotTableError,
UnexpectedNonTimestampError,
UnexpectedNullError,
)
from dbt.adapters.protocol import AdapterConfig, ConnectionManagerProtocol
from dbt.clients.agate_helper import empty_table, merge_tables, table_from_rows
from dbt.clients.jinja import MacroGenerator
from dbt.contracts.graph.manifest import Manifest, MacroManifest
from dbt.contracts.graph.nodes import ResultNode
from dbt.events.functions import fire_event, warn_or_error
from dbt.events.types import (
from dbt.adapters.protocol import AdapterConfig
from dbt.common.clients.agate_helper import (
empty_table,
get_column_value_uncased,
merge_tables,
table_from_rows,
Integer,
)
from dbt.common.clients.jinja import CallableMacroGenerator
from dbt.common.events.functions import fire_event, warn_or_error
from dbt.adapters.events.types import (
CacheMiss,
ListRelations,
CodeExecution,
@@ -58,9 +73,14 @@ from dbt.events.types import (
ConstraintNotSupported,
ConstraintNotEnforced,
)
from dbt.utils import filter_null_values, executor, cast_to_str, AttrDict
from dbt.common.utils import filter_null_values, executor, cast_to_str, AttrDict
from dbt.adapters.base.connections import Connection, AdapterResponse
from dbt.adapters.contracts.relation import RelationConfig
from dbt.adapters.base.connections import (
Connection,
AdapterResponse,
BaseConnectionManager,
)
from dbt.adapters.base.meta import AdapterMeta, available
from dbt.adapters.base.relation import (
ComponentName,
@@ -71,10 +91,13 @@ from dbt.adapters.base.relation import (
from dbt.adapters.base import Column as BaseColumn
from dbt.adapters.base import Credentials
from dbt.adapters.cache import RelationsCache, _make_ref_key_dict
from dbt import deprecations
from dbt.adapters.events.types import CollectFreshnessReturnSignature
GET_CATALOG_MACRO_NAME = "get_catalog"
GET_CATALOG_RELATIONS_MACRO_NAME = "get_catalog_relations"
FRESHNESS_MACRO_NAME = "collect_freshness"
GET_RELATION_LAST_MODIFIED_MACRO_NAME = "get_relation_last_modified"
class ConstraintSupport(str, Enum):
@@ -91,11 +114,13 @@ def _expect_row_value(key: str, row: agate.Row):
return row[key]
def _catalog_filter_schemas(manifest: Manifest) -> Callable[[agate.Row], bool]:
def _catalog_filter_schemas(
used_schemas: FrozenSet[Tuple[str, str]]
) -> Callable[[agate.Row], bool]:
"""Return a function that takes a row and decides if the row should be
included in the catalog output.
"""
schemas = frozenset((d.lower(), s.lower()) for d, s in manifest.get_used_schemas())
schemas = frozenset((d.lower(), s.lower()) for d, s in used_schemas)
def test(row: agate.Row) -> bool:
table_database = _expect_row_value("table_database", row)
@@ -109,7 +134,7 @@ def _catalog_filter_schemas(manifest: Manifest) -> Callable[[agate.Row], bool]:
return test
def _utc(dt: Optional[datetime], source: BaseRelation, field_name: str) -> datetime:
def _utc(dt: Optional[datetime], source: Optional[BaseRelation], field_name: str) -> datetime:
"""If dt has a timezone, return a new datetime that's in UTC. Otherwise,
assume the datetime is already for UTC and add the timezone.
"""
@@ -161,6 +186,12 @@ class PythonJobHelper:
raise NotImplementedError("PythonJobHelper submit function is not implemented yet")
class FreshnessResponse(TypedDict):
max_loaded_at: datetime
snapshotted_at: datetime
age: float # age in seconds
class BaseAdapter(metaclass=AdapterMeta):
"""The BaseAdapter provides an abstract base class for adapters.
@@ -208,7 +239,7 @@ class BaseAdapter(metaclass=AdapterMeta):
Relation: Type[BaseRelation] = BaseRelation
Column: Type[BaseColumn] = BaseColumn
ConnectionManager: Type[ConnectionManagerProtocol]
ConnectionManager: Type[BaseConnectionManager]
# A set of clobber config fields accepted by this adapter
# for use in materializations
@@ -222,11 +253,28 @@ class BaseAdapter(metaclass=AdapterMeta):
ConstraintType.foreign_key: ConstraintSupport.ENFORCED,
}
def __init__(self, config):
# This static member variable can be overriden in concrete adapter
# implementations to indicate adapter support for optional capabilities.
_capabilities = CapabilityDict({})
def __init__(self, config, mp_context: SpawnContext) -> None:
self.config = config
self.cache = RelationsCache()
self.connections = self.ConnectionManager(config)
self._macro_manifest_lazy: Optional[MacroManifest] = None
self.cache = RelationsCache(log_cache_events=config.log_cache_events)
self.connections = self.ConnectionManager(config, mp_context)
self._macro_client: Optional[MacroClient] = None
###
# Methods to set / access a macro client
###
def set_macro_client(self, macro_client: MacroClient) -> None:
self._macro_client = macro_client
def get_macro_client(self) -> Optional[MacroClient]:
return self._macro_client
def clear_macro_client(self) -> None:
if self._macro_client is not None:
self._macro_client = None
###
# Methods that pass through to the connection manager
@@ -256,10 +304,10 @@ class BaseAdapter(metaclass=AdapterMeta):
return conn.name
@contextmanager
def connection_named(self, name: str, node: Optional[ResultNode] = None) -> Iterator[None]:
def connection_named(self, name: str, query_header_context: Any = None) -> Iterator[None]:
try:
if self.connections.query_header is not None:
self.connections.query_header.set(name, node)
self.connections.query_header.set(name, query_header_context)
self.acquire_connection(name)
yield
finally:
@@ -267,11 +315,6 @@ class BaseAdapter(metaclass=AdapterMeta):
if self.connections.query_header is not None:
self.connections.query_header.reset()
@contextmanager
def connection_for(self, node: ResultNode) -> Iterator[None]:
with self.connection_named(node.unique_id, node):
yield
@available.parse(lambda *a, **k: ("", empty_table()))
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
@@ -315,14 +358,21 @@ class BaseAdapter(metaclass=AdapterMeta):
@available.parse(lambda *a, **k: ("", empty_table()))
def get_partitions_metadata(self, table: str) -> Tuple[agate.Table]:
"""Obtain partitions metadata for a BigQuery partitioned table.
"""
TODO: Can we move this to dbt-bigquery?
Obtain partitions metadata for a BigQuery partitioned table.
:param str table_id: a partitioned table id, in standard SQL format.
:param str table: a partitioned table id, in standard SQL format.
:return: a partition metadata tuple, as described in
https://cloud.google.com/bigquery/docs/creating-partitioned-tables#getting_partition_metadata_using_meta_tables.
:rtype: agate.Table
"""
return self.connections.get_partitions_metadata(table=table)
if hasattr(self.connections, "get_partitions_metadata"):
return self.connections.get_partitions_metadata(table=table)
else:
raise NotImplementedError(
"`get_partitions_metadata` is not implemented for this adapter!"
)
###
# Methods that should never be overridden
@@ -337,39 +387,6 @@ class BaseAdapter(metaclass=AdapterMeta):
"""
return cls.ConnectionManager.TYPE
@property
def _macro_manifest(self) -> MacroManifest:
if self._macro_manifest_lazy is None:
return self.load_macro_manifest()
return self._macro_manifest_lazy
def check_macro_manifest(self) -> Optional[MacroManifest]:
"""Return the internal manifest (used for executing macros) if it's
been initialized, otherwise return None.
"""
return self._macro_manifest_lazy
def load_macro_manifest(self, base_macros_only=False) -> MacroManifest:
# base_macros_only is for the test framework
if self._macro_manifest_lazy is None:
# avoid a circular import
from dbt.parser.manifest import ManifestLoader
manifest = ManifestLoader.load_macros(
self.config,
self.connections.set_query_header,
base_macros_only=base_macros_only,
)
# TODO CT-211
self._macro_manifest_lazy = manifest # type: ignore[assignment]
# TODO CT-211
return self._macro_manifest_lazy # type: ignore[return-value]
def clear_macro_manifest(self):
if self._macro_manifest_lazy is not None:
self._macro_manifest_lazy = None
###
# Caching methods
###
def _schema_is_cached(self, database: Optional[str], schema: str) -> bool:
@@ -387,18 +404,16 @@ class BaseAdapter(metaclass=AdapterMeta):
else:
return True
def _get_cache_schemas(self, manifest: Manifest) -> Set[BaseRelation]:
def _get_cache_schemas(self, relation_configs: Iterable[RelationConfig]) -> Set[BaseRelation]:
"""Get the set of schema relations that the cache logic needs to
populate. This means only executable nodes are included.
populate.
"""
# the cache only cares about executable nodes
return {
self.Relation.create_from(self.config, node).without_identifier()
for node in manifest.nodes.values()
if (node.is_relational and not node.is_ephemeral_model and not node.is_external_node)
self.Relation.create_from(quoting=self.config, relation_config=relation_config)
for relation_config in relation_configs
}
def _get_catalog_schemas(self, manifest: Manifest) -> SchemaSearchMap:
def _get_catalog_schemas(self, relation_configs: Iterable[RelationConfig]) -> SchemaSearchMap:
"""Get a mapping of each node's "information_schema" relations to a
set of all schemas expected in that information_schema.
@@ -408,16 +423,8 @@ class BaseAdapter(metaclass=AdapterMeta):
lowercase strings.
"""
info_schema_name_map = SchemaSearchMap()
nodes: Iterator[ResultNode] = chain(
[
node
for node in manifest.nodes.values()
if (node.is_relational and not node.is_ephemeral_model)
],
manifest.sources.values(),
)
for node in nodes:
relation = self.Relation.create_from(self.config, node)
relations = self._get_catalog_relations(relation_configs)
for relation in relations:
info_schema_name_map.add(relation)
# result is a map whose keys are information_schema Relations without
# identifiers that have appropriate database prefixes, and whose values
@@ -425,14 +432,37 @@ class BaseAdapter(metaclass=AdapterMeta):
# databases
return info_schema_name_map
def _get_catalog_relations_by_info_schema(
self, relations
) -> Dict[InformationSchema, List[BaseRelation]]:
relations_by_info_schema: Dict[InformationSchema, List[BaseRelation]] = dict()
for relation in relations:
info_schema = relation.information_schema_only()
if info_schema not in relations_by_info_schema:
relations_by_info_schema[info_schema] = []
relations_by_info_schema[info_schema].append(relation)
return relations_by_info_schema
def _get_catalog_relations(
self, relation_configs: Iterable[RelationConfig]
) -> List[BaseRelation]:
relations = [
self.Relation.create_from(quoting=self.config, relation_config=relation_config)
for relation_config in relation_configs
]
return relations
def _relations_cache_for_schemas(
self, manifest: Manifest, cache_schemas: Optional[Set[BaseRelation]] = None
self,
relation_configs: Iterable[RelationConfig],
cache_schemas: Optional[Set[BaseRelation]] = None,
) -> None:
"""Populate the relations cache for the given schemas. Returns an
iterable of the schemas populated, as strings.
"""
if not cache_schemas:
cache_schemas = self._get_cache_schemas(manifest)
cache_schemas = self._get_cache_schemas(relation_configs)
with executor(self.config) as tpe:
futures: List[Future[List[BaseRelation]]] = []
for cache_schema in cache_schemas:
@@ -453,14 +483,15 @@ class BaseAdapter(metaclass=AdapterMeta):
# it's possible that there were no relations in some schemas. We want
# to insert the schemas we query into the cache's `.schemas` attribute
# so we can check it later
cache_update: Set[Tuple[Optional[str], Optional[str]]] = set()
cache_update: Set[Tuple[Optional[str], str]] = set()
for relation in cache_schemas:
cache_update.add((relation.database, relation.schema))
if relation.schema:
cache_update.add((relation.database, relation.schema))
self.cache.update_schemas(cache_update)
def set_relations_cache(
self,
manifest: Manifest,
relation_configs: Iterable[RelationConfig],
clear: bool = False,
required_schemas: Optional[Set[BaseRelation]] = None,
) -> None:
@@ -470,7 +501,7 @@ class BaseAdapter(metaclass=AdapterMeta):
with self.cache.lock:
if clear:
self.cache.clear()
self._relations_cache_for_schemas(manifest, required_schemas)
self._relations_cache_for_schemas(relation_configs, required_schemas)
@available
def cache_added(self, relation: Optional[BaseRelation]) -> str:
@@ -917,6 +948,17 @@ class BaseAdapter(metaclass=AdapterMeta):
"""
raise NotImplementedError("`convert_number_type` is not implemented for this adapter!")
@classmethod
def convert_integer_type(cls, agate_table: agate.Table, col_idx: int) -> str:
"""Return the type in the database that best maps to the agate.Number
type for the given agate table and column index.
:param agate_table: The table
:param col_idx: The index into the agate table for the column.
:return: The name of the type in the database
"""
return "integer"
@classmethod
@abc.abstractmethod
def convert_boolean_type(cls, agate_table: agate.Table, col_idx: int) -> str:
@@ -974,6 +1016,7 @@ class BaseAdapter(metaclass=AdapterMeta):
def convert_agate_type(cls, agate_table: agate.Table, col_idx: int) -> Optional[str]:
agate_type: Type = agate_table.column_types[col_idx]
conversions: List[Tuple[Type, Callable[..., str]]] = [
(Integer, cls.convert_integer_type),
(agate.Text, cls.convert_text_type),
(agate.Number, cls.convert_number_type),
(agate.Boolean, cls.convert_boolean_type),
@@ -993,11 +1036,10 @@ class BaseAdapter(metaclass=AdapterMeta):
def execute_macro(
self,
macro_name: str,
manifest: Optional[Manifest] = None,
macro_client: Optional[MacroClient] = None,
project: Optional[str] = None,
context_override: Optional[Dict[str, Any]] = None,
kwargs: Optional[Dict[str, Any]] = None,
text_only_columns: Optional[Iterable[str]] = None,
) -> AttrDict:
"""Look macro_name up in the manifest and execute its results.
@@ -1017,13 +1059,11 @@ class BaseAdapter(metaclass=AdapterMeta):
if context_override is None:
context_override = {}
if manifest is None:
# TODO CT-211
manifest = self._macro_manifest # type: ignore[assignment]
# TODO CT-211
macro = manifest.find_macro_by_name( # type: ignore[union-attr]
macro_name, self.config.project_name, project
)
client = self._macro_client or macro_client
if client is None:
raise DbtInternalError("Macro client was None when calling execute_macro!")
macro = client.find_macro_by_name(macro_name, self.config.project_name, project)
if macro is None:
if project is None:
package_name = "any package"
@@ -1035,27 +1075,20 @@ class BaseAdapter(metaclass=AdapterMeta):
macro_name, package_name
)
)
# This causes a reference cycle, as generate_runtime_macro_context()
# ends up calling get_adapter, so the import has to be here.
from dbt.context.providers import generate_runtime_macro_context
macro_context = generate_runtime_macro_context(
# TODO CT-211
macro=macro,
config=self.config,
manifest=manifest, # type: ignore[arg-type]
package_name=project,
)
macro_context = client.generate_context_for_macro(self.config, macro, project)
macro_context.update(context_override)
macro_function = MacroGenerator(macro, macro_context)
macro_function = CallableMacroGenerator(macro, macro_context)
with self.connections.exception_handler(f"macro {macro_name}"):
result = macro_function(**kwargs)
return result
@classmethod
def _catalog_filter_table(cls, table: agate.Table, manifest: Manifest) -> agate.Table:
def _catalog_filter_table(
cls, table: agate.Table, used_schemas: FrozenSet[Tuple[str, str]]
) -> agate.Table:
"""Filter the table as appropriate for catalog entries. Subclasses can
override this to change filtering rules on a per-adapter basis.
"""
@@ -1065,44 +1098,122 @@ class BaseAdapter(metaclass=AdapterMeta):
table.column_names,
text_only_columns=["table_database", "table_schema", "table_name"],
)
return table.where(_catalog_filter_schemas(manifest))
return table.where(_catalog_filter_schemas(used_schemas))
def _get_one_catalog(
self,
information_schema: InformationSchema,
schemas: Set[str],
manifest: Manifest,
used_schemas: FrozenSet[Tuple[str, str]],
) -> agate.Table:
kwargs = {"information_schema": information_schema, "schemas": schemas}
table = self.execute_macro(
GET_CATALOG_MACRO_NAME,
kwargs=kwargs,
# pass in the full manifest so we get any local project
# overrides
manifest=manifest,
)
table = self.execute_macro(GET_CATALOG_MACRO_NAME, kwargs=kwargs)
results = self._catalog_filter_table(table, manifest) # type: ignore[arg-type]
results = self._catalog_filter_table(table, used_schemas) # type: ignore[arg-type]
return results
def get_catalog(self, manifest: Manifest) -> Tuple[agate.Table, List[Exception]]:
schema_map = self._get_catalog_schemas(manifest)
def _get_one_catalog_by_relations(
self,
information_schema: InformationSchema,
relations: List[BaseRelation],
used_schemas: FrozenSet[Tuple[str, str]],
) -> agate.Table:
kwargs = {
"information_schema": information_schema,
"relations": relations,
}
table = self.execute_macro(GET_CATALOG_RELATIONS_MACRO_NAME, kwargs=kwargs)
results = self._catalog_filter_table(table, used_schemas) # type: ignore[arg-type]
return results
def get_filtered_catalog(
self,
relation_configs: Iterable[RelationConfig],
used_schemas: FrozenSet[Tuple[str, str]],
relations: Optional[Set[BaseRelation]] = None,
):
catalogs: agate.Table
if (
relations is None
or len(relations) > 100
or not self.supports(Capability.SchemaMetadataByRelations)
):
# Do it the traditional way. We get the full catalog.
catalogs, exceptions = self.get_catalog(relation_configs, used_schemas)
else:
# Do it the new way. We try to save time by selecting information
# only for the exact set of relations we are interested in.
catalogs, exceptions = self.get_catalog_by_relations(used_schemas, relations)
if relations and catalogs:
relation_map = {
(
r.database.casefold() if r.database else None,
r.schema.casefold() if r.schema else None,
r.identifier.casefold() if r.identifier else None,
)
for r in relations
}
def in_map(row: agate.Row):
d = _expect_row_value("table_database", row)
s = _expect_row_value("table_schema", row)
i = _expect_row_value("table_name", row)
d = d.casefold() if d is not None else None
s = s.casefold() if s is not None else None
i = i.casefold() if i is not None else None
return (d, s, i) in relation_map
catalogs = catalogs.where(in_map)
return catalogs, exceptions
def row_matches_relation(self, row: agate.Row, relations: Set[BaseRelation]):
pass
def get_catalog(
self,
relation_configs: Iterable[RelationConfig],
used_schemas: FrozenSet[Tuple[str, str]],
) -> Tuple[agate.Table, List[Exception]]:
with executor(self.config) as tpe:
futures: List[Future[agate.Table]] = []
schema_map: SchemaSearchMap = self._get_catalog_schemas(relation_configs)
for info, schemas in schema_map.items():
if len(schemas) == 0:
continue
name = ".".join([str(info.database), "information_schema"])
fut = tpe.submit_connected(
self, name, self._get_one_catalog, info, schemas, manifest
self, name, self._get_one_catalog, info, schemas, used_schemas
)
futures.append(fut)
catalogs, exceptions = catch_as_completed(futures)
return catalogs, exceptions
def get_catalog_by_relations(
self, used_schemas: FrozenSet[Tuple[str, str]], relations: Set[BaseRelation]
) -> Tuple[agate.Table, List[Exception]]:
with executor(self.config) as tpe:
futures: List[Future[agate.Table]] = []
relations_by_schema = self._get_catalog_relations_by_info_schema(relations)
for info_schema in relations_by_schema:
name = ".".join([str(info_schema.database), "information_schema"])
relations = set(relations_by_schema[info_schema])
fut = tpe.submit_connected(
self,
name,
self._get_one_catalog_by_relations,
info_schema,
relations,
used_schemas,
)
futures.append(fut)
catalogs, exceptions = catch_as_completed(futures)
return catalogs, exceptions
return catalogs, exceptions
def cancel_open_connections(self):
"""Cancel all open connections."""
@@ -1113,8 +1224,8 @@ class BaseAdapter(metaclass=AdapterMeta):
source: BaseRelation,
loaded_at_field: str,
filter: Optional[str],
manifest: Optional[Manifest] = None,
) -> Tuple[Optional[AdapterResponse], Dict[str, Any]]:
macro_client: Optional[MacroClient] = None,
) -> Tuple[Optional[AdapterResponse], FreshnessResponse]:
"""Calculate the freshness of sources in dbt, and return it"""
kwargs: Dict[str, Any] = {
"source": source,
@@ -1129,9 +1240,9 @@ class BaseAdapter(metaclass=AdapterMeta):
AttrDict, # current: contains AdapterResponse + agate.Table
agate.Table, # previous: just table
]
result = self.execute_macro(FRESHNESS_MACRO_NAME, kwargs=kwargs, manifest=manifest)
result = self.execute_macro(FRESHNESS_MACRO_NAME, kwargs=kwargs, macro_client=macro_client)
if isinstance(result, agate.Table):
deprecations.warn("collect-freshness-return-signature")
warn_or_error(CollectFreshnessReturnSignature())
adapter_response = None
table = result
else:
@@ -1149,13 +1260,52 @@ class BaseAdapter(metaclass=AdapterMeta):
snapshotted_at = _utc(table[0][1], source, loaded_at_field)
age = (snapshotted_at - max_loaded_at).total_seconds()
freshness = {
freshness: FreshnessResponse = {
"max_loaded_at": max_loaded_at,
"snapshotted_at": snapshotted_at,
"age": age,
}
return adapter_response, freshness
def calculate_freshness_from_metadata(
self,
source: BaseRelation,
macro_client: Optional[MacroClient] = None,
) -> Tuple[Optional[AdapterResponse], FreshnessResponse]:
kwargs: Dict[str, Any] = {
"information_schema": source.information_schema_only(),
"relations": [source],
}
result = self.execute_macro(
GET_RELATION_LAST_MODIFIED_MACRO_NAME, kwargs=kwargs, macro_client=macro_client
)
adapter_response, table = result.response, result.table # type: ignore[attr-defined]
try:
row = table[0]
last_modified_val = get_column_value_uncased("last_modified", row)
snapshotted_at_val = get_column_value_uncased("snapshotted_at", row)
except Exception:
raise MacroResultError(GET_RELATION_LAST_MODIFIED_MACRO_NAME, table)
if last_modified_val is None:
# Interpret missing value as "infinitely long ago"
max_loaded_at = datetime(1, 1, 1, 0, 0, 0, tzinfo=pytz.UTC)
else:
max_loaded_at = _utc(last_modified_val, None, "last_modified")
snapshotted_at = _utc(snapshotted_at_val, None, "snapshotted_at")
age = (snapshotted_at - max_loaded_at).total_seconds()
freshness: FreshnessResponse = {
"max_loaded_at": max_loaded_at,
"snapshotted_at": snapshotted_at,
"age": age,
}
return adapter_response, freshness
def pre_model_hook(self, config: Mapping[str, Any]) -> Any:
"""A hook for running some operation before the model materialization
runs. The hook can assume it has a connection available.
@@ -1181,11 +1331,6 @@ class BaseAdapter(metaclass=AdapterMeta):
"""
pass
def get_compiler(self):
from dbt.compilation import Compiler
return Compiler(self.config)
# Methods used in adapter tests
def update_column_sql(
self,
@@ -1305,7 +1450,7 @@ class BaseAdapter(metaclass=AdapterMeta):
strategy = strategy.replace("+", "_")
macro_name = f"get_incremental_{strategy}_sql"
# The model_context should have MacroGenerator callable objects for all macros
# The model_context should have callable objects for all macros
if macro_name not in model_context:
raise DbtRuntimeError(
'dbt could not find an incremental strategy macro with the name "{}" in {}'.format(
@@ -1429,6 +1574,14 @@ class BaseAdapter(metaclass=AdapterMeta):
else:
return None
@classmethod
def capabilities(cls) -> CapabilityDict:
return cls._capabilities
@classmethod
def supports(cls, capability: Capability) -> bool:
return bool(cls.capabilities()[capability])
COLUMNS_EQUAL_SQL = """
with diff_count as (

View File

@@ -1,9 +1,8 @@
import abc
from functools import wraps
from typing import Callable, Optional, Any, FrozenSet, Dict, Set
from dbt.deprecations import warn, renamed_method
from dbt.common.events.functions import warn_or_error
from dbt.adapters.events.types import AdapterDeprecationWarning
Decorator = Callable[[Any], Callable]
@@ -62,11 +61,12 @@ class _Available:
def wrapper(func):
func_name = func.__name__
renamed_method(func_name, supported_name)
@wraps(func)
def inner(*args, **kwargs):
warn("adapter:{}".format(func_name))
warn_or_error(
AdapterDeprecationWarning(old_name=func_name, new_name=supported_name)
)
return func(*args, **kwargs)
if parse_replacement:
@@ -93,7 +93,7 @@ class AdapterMeta(abc.ABCMeta):
_available_: FrozenSet[str]
_parse_replacements_: Dict[str, Callable]
def __new__(mcls, name, bases, namespace, **kwargs):
def __new__(mcls, name, bases, namespace, **kwargs) -> "AdapterMeta":
# mypy does not like the `**kwargs`. But `ABCMeta` itself takes
# `**kwargs` in its argspec here (and passes them to `type.__new__`.
# I'm not sure there is any benefit to it after poking around a bit,

View File

@@ -1,20 +1,10 @@
from typing import List, Optional, Type
from pathlib import Path
from dbt.adapters.base import Credentials
from dbt.exceptions import CompilationError
from dbt.adapters.protocol import AdapterProtocol
def project_name_from_path(include_path: str) -> str:
# avoid an import cycle
from dbt.config.project import PartialProject
partial = PartialProject.from_project_root(include_path)
if partial.project_name is None:
raise CompilationError(f"Invalid project at {include_path}: name not set!")
return partial.project_name
class AdapterPlugin:
"""Defines the basic requirements for a dbt adapter plugin.
@@ -29,12 +19,13 @@ class AdapterPlugin:
credentials: Type[Credentials],
include_path: str,
dependencies: Optional[List[str]] = None,
):
project_name: Optional[str] = None,
) -> None:
self.adapter: Type[AdapterProtocol] = adapter
self.credentials: Type[Credentials] = credentials
self.include_path: str = include_path
self.project_name: str = project_name_from_path(include_path)
self.project_name: str = project_name or f"dbt_{Path(include_path).name}"
self.dependencies: List[str]
if dependencies is None:
self.dependencies = []

View File

@@ -1,21 +1,20 @@
from threading import local
from typing import Optional, Callable, Dict, Any
from dbt.clients.jinja import QueryStringGenerator
from dbt.adapters.clients.jinja import QueryStringGenerator
from dbt.context.manifest import generate_query_header_context
from dbt.contracts.connection import AdapterRequiredConfig, QueryComment
from dbt.contracts.graph.nodes import ResultNode
from dbt.adapters.contracts.connection import AdapterRequiredConfig, QueryComment
from dbt.contracts.graph.manifest import Manifest
from dbt.exceptions import DbtRuntimeError
from dbt.common.exceptions import DbtRuntimeError
class NodeWrapper:
def __init__(self, node):
self._inner_node = node
class QueryHeaderContextWrapper:
def __init__(self, context) -> None:
self._inner_context = context
def __getattr__(self, name):
return getattr(self._inner_node, name, "")
return getattr(self._inner_context, name, "")
class _QueryComment(local):
@@ -25,9 +24,9 @@ class _QueryComment(local):
- a source_name indicating what set the current thread's query comment
"""
def __init__(self, initial):
def __init__(self, initial) -> None:
self.query_comment: Optional[str] = initial
self.append = False
self.append: bool = False
def add(self, sql: str) -> str:
if not self.query_comment:
@@ -53,11 +52,11 @@ class _QueryComment(local):
self.append = append
QueryStringFunc = Callable[[str, Optional[NodeWrapper]], str]
QueryStringFunc = Callable[[str, Optional[QueryHeaderContextWrapper]], str]
class MacroQueryStringSetter:
def __init__(self, config: AdapterRequiredConfig, manifest: Manifest):
def __init__(self, config: AdapterRequiredConfig, manifest: Manifest) -> None:
self.manifest = manifest
self.config = config
@@ -90,10 +89,10 @@ class MacroQueryStringSetter:
def reset(self):
self.set("master", None)
def set(self, name: str, node: Optional[ResultNode]):
wrapped: Optional[NodeWrapper] = None
if node is not None:
wrapped = NodeWrapper(node)
def set(self, name: str, query_header_context: Any):
wrapped: Optional[QueryHeaderContextWrapper] = None
if query_header_context is not None:
wrapped = QueryHeaderContextWrapper(query_header_context)
comment_str = self.generator(name, wrapped)
append = False

View File

@@ -1,9 +1,9 @@
from collections.abc import Hashable
from dataclasses import dataclass, field
from typing import Optional, TypeVar, Any, Type, Dict, Iterator, Tuple, Set
from typing import Optional, TypeVar, Any, Type, Dict, Iterator, Tuple, Set, Union, FrozenSet
from dbt.contracts.graph.nodes import SourceDefinition, ManifestNode, ResultNode, ParsedNode
from dbt.contracts.relation import (
from dbt.adapters.contracts.relation import (
RelationConfig,
RelationType,
ComponentName,
HasQuoting,
@@ -11,18 +11,15 @@ from dbt.contracts.relation import (
Policy,
Path,
)
from dbt.exceptions import (
ApproximateMatchError,
DbtInternalError,
MultipleDatabasesNotAllowedError,
)
from dbt.node_types import NodeType
from dbt.utils import filter_null_values, deep_merge, classproperty
from dbt.adapters.exceptions import MultipleDatabasesNotAllowedError, ApproximateMatchError
from dbt.common.utils import filter_null_values, deep_merge
from dbt.adapters.utils import classproperty
import dbt.exceptions
import dbt.common.exceptions
Self = TypeVar("Self", bound="BaseRelation")
SerializableIterable = Union[Tuple, FrozenSet]
@dataclass(frozen=True, eq=False, repr=False)
@@ -35,6 +32,19 @@ class BaseRelation(FakeAPIObject, Hashable):
include_policy: Policy = field(default_factory=lambda: Policy())
quote_policy: Policy = field(default_factory=lambda: Policy())
dbt_created: bool = False
limit: Optional[int] = None
# register relation types that can be renamed for the purpose of replacing relations using stages and backups
# adding a relation type here also requires defining the associated rename macro
# e.g. adding RelationType.View in dbt-postgres requires that you define:
# include/postgres/macros/relations/view/rename.sql::postgres__get_rename_view_sql()
renameable_relations: SerializableIterable = ()
# register relation types that are atomically replaceable, e.g. they have "create or replace" syntax
# adding a relation type here also requires defining the associated replace macro
# e.g. adding RelationType.View in dbt-postgres requires that you define:
# include/postgres/macros/relations/view/replace.sql::postgres__get_replace_view_sql()
replaceable_relations: SerializableIterable = ()
def _is_exactish_match(self, field: ComponentName, value: str) -> bool:
if self.dbt_created and self.quote_policy.get_part(field) is False:
@@ -87,7 +97,7 @@ class BaseRelation(FakeAPIObject, Hashable):
if not search:
# nothing was passed in
raise dbt.exceptions.DbtRuntimeError(
raise dbt.common.exceptions.DbtRuntimeError(
"Tried to match relation, but no search path was passed!"
)
@@ -169,7 +179,6 @@ class BaseRelation(FakeAPIObject, Hashable):
return self.include(identifier=False).replace_path(identifier=None)
def _render_iterator(self) -> Iterator[Tuple[Optional[ComponentName], Optional[str]]]:
for key in ComponentName:
path_part: Optional[str] = None
if self.include_policy.get_part(key):
@@ -182,88 +191,65 @@ class BaseRelation(FakeAPIObject, Hashable):
# if there is nothing set, this will return the empty string.
return ".".join(part for _, part in self._render_iterator() if part is not None)
def render_limited(self) -> str:
rendered = self.render()
if self.limit is None:
return rendered
elif self.limit == 0:
return f"(select * from {rendered} where false limit 0) _dbt_limit_subq"
else:
return f"(select * from {rendered} limit {self.limit}) _dbt_limit_subq"
def quoted(self, identifier):
return "{quote_char}{identifier}{quote_char}".format(
quote_char=self.quote_character,
identifier=identifier,
)
@classmethod
def create_from_source(cls: Type[Self], source: SourceDefinition, **kwargs: Any) -> Self:
source_quoting = source.quoting.to_dict(omit_none=True)
source_quoting.pop("column", None)
quote_policy = deep_merge(
cls.get_default_quote_policy().to_dict(omit_none=True),
source_quoting,
kwargs.get("quote_policy", {}),
)
return cls.create(
database=source.database,
schema=source.schema,
identifier=source.identifier,
quote_policy=quote_policy,
**kwargs,
)
@staticmethod
def add_ephemeral_prefix(name: str):
return f"__dbt__cte__{name}"
@classmethod
def create_ephemeral_from_node(
def create_ephemeral_from(
cls: Type[Self],
config: HasQuoting,
node: ManifestNode,
relation_config: RelationConfig,
limit: Optional[int],
) -> Self:
# Note that ephemeral models are based on the name.
identifier = cls.add_ephemeral_prefix(node.name)
identifier = cls.add_ephemeral_prefix(relation_config.name)
return cls.create(
type=cls.CTE,
identifier=identifier,
limit=limit,
).quote(identifier=False)
@classmethod
def create_from_node(
cls: Type[Self],
config: HasQuoting,
node,
quote_policy: Optional[Dict[str, bool]] = None,
**kwargs: Any,
) -> Self:
if quote_policy is None:
quote_policy = {}
quote_policy = dbt.utils.merge(config.quoting, quote_policy)
return cls.create(
database=node.database,
schema=node.schema,
identifier=node.alias,
quote_policy=quote_policy,
**kwargs,
)
@classmethod
def create_from(
cls: Type[Self],
config: HasQuoting,
node: ResultNode,
quoting: HasQuoting,
relation_config: RelationConfig,
**kwargs: Any,
) -> Self:
if node.resource_type == NodeType.Source:
if not isinstance(node, SourceDefinition):
raise DbtInternalError(
"type mismatch, expected SourceDefinition but got {}".format(type(node))
)
return cls.create_from_source(node, **kwargs)
else:
# Can't use ManifestNode here because of parameterized generics
if not isinstance(node, (ParsedNode)):
raise DbtInternalError(
f"type mismatch, expected ManifestNode but got {type(node)}"
)
return cls.create_from_node(config, node, **kwargs)
quote_policy = kwargs.pop("quote_policy", {})
config_quoting = relation_config.quoting_dict
config_quoting.pop("column", None)
# precedence: kwargs quoting > relation config quoting > base quoting > default quoting
quote_policy = deep_merge(
cls.get_default_quote_policy().to_dict(omit_none=True),
quoting.quoting,
config_quoting,
quote_policy,
)
return cls.create(
database=relation_config.database,
schema=relation_config.schema,
identifier=relation_config.identifier,
quote_policy=quote_policy,
**kwargs,
)
@classmethod
def create(
@@ -286,6 +272,14 @@ class BaseRelation(FakeAPIObject, Hashable):
)
return cls.from_dict(kwargs)
@property
def can_be_renamed(self) -> bool:
return self.type in self.renameable_relations
@property
def can_be_replaced(self) -> bool:
return self.type in self.replaceable_relations
def __repr__(self) -> str:
return "<{} {}>".format(self.__class__.__name__, self.render())
@@ -293,7 +287,7 @@ class BaseRelation(FakeAPIObject, Hashable):
return hash(self.render())
def __str__(self) -> str:
return self.render()
return self.render() if self.limit is None else self.render_limited()
@property
def database(self) -> Optional[str]:
@@ -366,7 +360,7 @@ class InformationSchema(BaseRelation):
def __post_init__(self):
if not isinstance(self.information_schema_view, (type(None), str)):
raise dbt.exceptions.CompilationError(
raise dbt.common.exceptions.CompilationError(
"Got an invalid name: {}".format(self.information_schema_view)
)
@@ -439,11 +433,11 @@ class SchemaSearchMap(Dict[InformationSchema, Set[Optional[str]]]):
self[key].add(schema)
def search(self) -> Iterator[Tuple[InformationSchema, Optional[str]]]:
for information_schema_name, schemas in self.items():
for information_schema, schemas in self.items():
for schema in schemas:
yield information_schema_name, schema
yield information_schema, schema
def flatten(self, allow_multiple_databases: bool = False):
def flatten(self, allow_multiple_databases: bool = False) -> "SchemaSearchMap":
new = self.__class__()
# make sure we don't have multiple databases if allow_multiple_databases is set to False

View File

@@ -7,17 +7,16 @@ from dbt.adapters.reference_keys import (
_make_ref_key_dict,
_ReferenceKey,
)
from dbt.exceptions import (
DependentLinkNotCachedError,
from dbt.common.exceptions.cache import (
NewNameAlreadyInCacheError,
NoneRelationFoundError,
ReferencedLinkNotCachedError,
DependentLinkNotCachedError,
TruncatedModelNameCausedCollisionError,
NoneRelationFoundError,
)
from dbt.events.functions import fire_event, fire_event_if
from dbt.events.types import CacheAction, CacheDumpGraph
from dbt.flags import get_flags
from dbt.utils import lowercase
from dbt.common.events.functions import fire_event, fire_event_if
from dbt.adapters.events.types import CacheAction, CacheDumpGraph
from dbt.common.utils.formatting import lowercase
def dot_separated(key: _ReferenceKey) -> str:
@@ -38,8 +37,8 @@ class _CachedRelation:
:attr BaseRelation inner: The underlying dbt relation.
"""
def __init__(self, inner):
self.referenced_by = {}
def __init__(self, inner) -> None:
self.referenced_by: Dict[_ReferenceKey, _CachedRelation] = {}
self.inner = inner
def __str__(self) -> str:
@@ -165,10 +164,11 @@ class RelationsCache:
:attr Set[str] schemas: The set of known/cached schemas, all lowercased.
"""
def __init__(self) -> None:
def __init__(self, log_cache_events: bool = False) -> None:
self.relations: Dict[_ReferenceKey, _CachedRelation] = {}
self.lock = threading.RLock()
self.schemas: Set[Tuple[Optional[str], Optional[str]]] = set()
self.log_cache_events = log_cache_events
def add_schema(
self,
@@ -318,10 +318,9 @@ class RelationsCache:
:param BaseRelation relation: The underlying relation.
"""
flags = get_flags()
cached = _CachedRelation(relation)
fire_event_if(
flags.LOG_CACHE_EVENTS,
self.log_cache_events,
lambda: CacheDumpGraph(before_after="before", action="adding", dump=self.dump_graph()),
)
fire_event(CacheAction(action="add_relation", ref_key=_make_ref_key_dict(cached)))
@@ -329,7 +328,7 @@ class RelationsCache:
with self.lock:
self._setdefault(cached)
fire_event_if(
flags.LOG_CACHE_EVENTS,
self.log_cache_events,
lambda: CacheDumpGraph(before_after="after", action="adding", dump=self.dump_graph()),
)
@@ -454,9 +453,8 @@ class RelationsCache:
ref_key_2=new_key._asdict(),
)
)
flags = get_flags()
fire_event_if(
flags.LOG_CACHE_EVENTS,
self.log_cache_events,
lambda: CacheDumpGraph(before_after="before", action="rename", dump=self.dump_graph()),
)
@@ -467,7 +465,7 @@ class RelationsCache:
self._setdefault(_CachedRelation(new))
fire_event_if(
flags.LOG_CACHE_EVENTS,
self.log_cache_events,
lambda: CacheDumpGraph(before_after="after", action="rename", dump=self.dump_graph()),
)

View File

@@ -0,0 +1,52 @@
from dataclasses import dataclass
from enum import Enum
from typing import Optional, DefaultDict, Mapping
class Capability(str, Enum):
"""Enumeration of optional adapter features which can be probed using BaseAdapter.capabilities()"""
SchemaMetadataByRelations = "SchemaMetadataByRelations"
"""Indicates efficient support for retrieving schema metadata for a list of relations, rather than always retrieving
all the relations in a schema."""
TableLastModifiedMetadata = "TableLastModifiedMetadata"
"""Indicates support for determining the time of the last table modification by querying database metadata."""
class Support(str, Enum):
Unknown = "Unknown"
"""The adapter has not declared whether this capability is a feature of the underlying DBMS."""
Unsupported = "Unsupported"
"""This capability is not possible with the underlying DBMS, so the adapter does not implement related macros."""
NotImplemented = "NotImplemented"
"""This capability is available in the underlying DBMS, but support has not yet been implemented in the adapter."""
Versioned = "Versioned"
"""Some versions of the DBMS supported by the adapter support this capability and the adapter has implemented any
macros needed to use it."""
Full = "Full"
"""All versions of the DBMS supported by the adapter support this capability and the adapter has implemented any
macros needed to use it."""
@dataclass
class CapabilitySupport:
support: Support
first_version: Optional[str] = None
def __bool__(self):
return self.support == Support.Versioned or self.support == Support.Full
class CapabilityDict(DefaultDict[Capability, CapabilitySupport]):
def __init__(self, vals: Mapping[Capability, CapabilitySupport]):
super().__init__(self._default)
self.update(vals)
@staticmethod
def _default():
return CapabilitySupport(support=Support.Unknown)

View File

@@ -0,0 +1,23 @@
from typing import Dict, Any
from dbt.common.clients.jinja import BaseMacroGenerator, get_environment
class QueryStringGenerator(BaseMacroGenerator):
def __init__(self, template_str: str, context: Dict[str, Any]) -> None:
super().__init__(context)
self.template_str: str = template_str
env = get_environment()
self.template = env.from_string(
self.template_str,
globals=self.context,
)
def get_name(self) -> str:
return "query_comment_macro"
def get_template(self):
"""Don't use the template cache, we don't have a node"""
return self.template
def __call__(self, connection_name: str, node) -> str:
return str(self.call_macro(connection_name, node))

View File

View File

@@ -11,31 +11,32 @@ from typing import (
List,
Callable,
)
from dbt.exceptions import DbtInternalError
from dbt.utils import translate_aliases, md5
from dbt.events.functions import fire_event
from dbt.events.types import NewConnectionOpening
from dbt.events.contextvars import get_node_info
from typing_extensions import Protocol
from dbt.dataclass_schema import (
from typing_extensions import Protocol, Annotated
from mashumaro.jsonschema.annotations import Pattern
from dbt.adapters.utils import translate_aliases
from dbt.common.exceptions import DbtInternalError
from dbt.common.dataclass_schema import (
dbtClassMixin,
StrEnum,
ExtensibleDbtClassMixin,
HyphenatedDbtClassMixin,
ValidatedStringMixin,
register_pattern,
)
from dbt.contracts.util import Replaceable
from dbt.common.contracts.util import Replaceable
from dbt.common.utils import md5
from dbt.common.events.functions import fire_event
from dbt.adapters.events.types import NewConnectionOpening
# TODO: this is a very bad dependency - shared global state
from dbt.common.events.contextvars import get_node_info
class Identifier(ValidatedStringMixin):
ValidationRegex = r"^[A-Za-z_][A-Za-z0-9_]+$"
# we need register_pattern for jsonschema validation
register_pattern(Identifier, r"^[A-Za-z_][A-Za-z0-9_]+$")
@dataclass
class AdapterResponse(dbtClassMixin):
_message: str
@@ -55,7 +56,8 @@ class ConnectionState(StrEnum):
@dataclass(init=False)
class Connection(ExtensibleDbtClassMixin, Replaceable):
type: Identifier
# Annotated is used by mashumaro for jsonschema generation
type: Annotated[Identifier, Pattern(r"^[A-Za-z_][A-Za-z0-9_]+$")]
name: Optional[str] = None
state: ConnectionState = ConnectionState.INIT
transaction_open: bool = False
@@ -108,7 +110,7 @@ class LazyHandle:
connection, updating the handle on the Connection.
"""
def __init__(self, opener: Callable[[Connection], Connection]):
def __init__(self, opener: Callable[[Connection], Connection]) -> None:
self.opener = opener
def resolve(self, connection: Connection) -> Connection:
@@ -161,6 +163,7 @@ class Credentials(ExtensibleDbtClassMixin, Replaceable, metaclass=abc.ABCMeta):
@classmethod
def __pre_deserialize__(cls, data):
data = super().__pre_deserialize__(data)
# Need to fixup dbname => database, pass => password
data = cls.translate_aliases(data)
return data
@@ -220,10 +223,10 @@ DEFAULT_QUERY_COMMENT = """
@dataclass
class QueryComment(HyphenatedDbtClassMixin):
class QueryComment(dbtClassMixin):
comment: str = DEFAULT_QUERY_COMMENT
append: bool = False
job_label: bool = False
job_label: bool = field(default=False, metadata={"alias": "job-label"})
class AdapterRequiredConfig(HasCredentials, Protocol):
@@ -231,3 +234,4 @@ class AdapterRequiredConfig(HasCredentials, Protocol):
query_comment: QueryComment
cli_vars: Dict[str, Any]
target_path: str
log_cache_events: bool

View File

@@ -0,0 +1,16 @@
from typing import Optional, Dict, Any
from typing_extensions import Protocol
from dbt.common.clients.jinja import MacroProtocol
class MacroClient(Protocol):
def find_macro_by_name(
self, name: str, root_project_name: str, package: Optional[str]
) -> Optional[MacroProtocol]:
raise NotImplementedError("find_macro_by_name not implemented")
def generate_context_for_macro(
self, config, macro: MacroProtocol, package: Optional[str]
) -> Dict[str, Any]:
raise NotImplementedError("generate_context_for_macro not implemented")

View File

@@ -6,11 +6,11 @@ from typing import (
)
from typing_extensions import Protocol
from dbt.dataclass_schema import dbtClassMixin, StrEnum
from dbt.common.dataclass_schema import dbtClassMixin, StrEnum
from dbt.contracts.util import Replaceable
from dbt.exceptions import CompilationError, DataclassNotDictError
from dbt.utils import deep_merge
from dbt.common.contracts.util import Replaceable
from dbt.common.exceptions import CompilationError, DataclassNotDictError
from dbt.common.utils import deep_merge
class RelationType(StrEnum):
@@ -19,6 +19,15 @@ class RelationType(StrEnum):
CTE = "cte"
MaterializedView = "materialized_view"
External = "external"
Ephemeral = "ephemeral"
class RelationConfig(Protocol):
name: str
database: str
schema: str
identifier: str
quoting_dict: Dict[str, bool]
class ComponentName(StrEnum):

View File

@@ -0,0 +1,57 @@
# Events Module
The Events module is responsible for communicating internal dbt structures into a consumable interface. Because the "event" classes are based entirely on protobuf definitions, the interface is really clearly defined, whether or not protobufs are used to consume it. We use Betterproto for compiling the protobuf message definitions into Python classes.
# Using the Events Module
The event module provides types that represent what is happening in dbt in `events.types`. These types are intended to represent an exhaustive list of all things happening within dbt that will need to be logged, streamed, or printed. To fire an event, `common.events.functions::fire_event` is the entry point to the module from everywhere in dbt.
# Logging
When events are processed via `fire_event`, nearly everything is logged. Whether or not the user has enabled the debug flag, all debug messages are still logged to the file. However, some events are particularly time consuming to construct because they return a huge amount of data. Today, the only messages in this category are cache events and are only logged if the `--log-cache-events` flag is on. This is important because these messages should not be created unless they are going to be logged, because they cause a noticable performance degredation. These events use a "fire_event_if" functions.
# Adding a New Event
* Add a new message in types.proto, and a second message with the same name + "Msg". The "Msg" message should have two fields, an "info" field of EventInfo, and a "data" field referring to the message name without "Msg"
* run the protoc compiler to update adapter_types_pb2.py: make adapter_proto_types
* Add a wrapping class in core/dbt/adapters/event/types.py with a Level superclass plus code and message methods
We have switched from using betterproto to using google protobuf, because of a lack of support for Struct fields in betterproto.
The google protobuf interface is janky and very much non-Pythonic. The "generated" classes in types_pb2.py do not resemble regular Python classes. They do not have normal constructors; they can only be constructed empty. They can be "filled" by setting fields individually or using a json_format method like ParseDict. We have wrapped the logging events with a class (in types.py) which allows using a constructor -- keywords only, no positional parameters.
## Required for Every Event
- a method `code`, that's unique across events
- assign a log level by using the Level mixin: `DebugLevel`, `InfoLevel`, `WarnLevel`, or `ErrorLevel`
- a message()
Example
```
class PartialParsingDeletedExposure(DebugLevel):
def code(self):
return "I049"
def message(self) -> str:
return f"Partial parsing: deleted exposure {self.unique_id}"
```
# Adapter Maintainers
To integrate existing log messages from adapters, you likely have a line of code like this in your adapter already:
```python
from dbt.logger import GLOBAL_LOGGER as logger
```
Simply change it to these two lines with your adapter's database name, and all your existing call sites will now use the new system for v1.0:
```python
from dbt.adapter.events.logging import AdapterLogger
logger = AdapterLogger("<database name>")
# e.g. AdapterLogger("Snowflake")
```
## Compiling types.proto
After adding a new message in `adapter_types.proto`, either:
- In the repository root directory: `make adapter_proto_types`
- In the `core/dbt/adapters/events` directory: `protoc -I=. --python_out=. types.proto`

View File

View File

@@ -0,0 +1,517 @@
syntax = "proto3";
package proto_types;
import "google/protobuf/timestamp.proto";
import "google/protobuf/struct.proto";
// Common event info
message AdapterCommonEventInfo {
string name = 1;
string code = 2;
string msg = 3;
string level = 4;
string invocation_id = 5;
int32 pid = 6;
string thread = 7;
google.protobuf.Timestamp ts = 8;
map<string, string> extra = 9;
string category = 10;
}
// AdapterNodeRelation
message AdapterNodeRelation {
string database = 10;
string schema = 11;
string alias = 12;
string relation_name = 13;
}
// NodeInfo
message AdapterNodeInfo {
string node_path = 1;
string node_name = 2;
string unique_id = 3;
string resource_type = 4;
string materialized = 5;
string node_status = 6;
string node_started_at = 7;
string node_finished_at = 8;
google.protobuf.Struct meta = 9;
AdapterNodeRelation node_relation = 10;
}
// ReferenceKey
message ReferenceKeyMsg {
string database = 1;
string schema = 2;
string identifier = 3;
}
// D - Deprecations
// D005
message AdapterDeprecationWarning {
string old_name = 1;
string new_name = 2;
}
message AdapterDeprecationWarningMsg {
AdapterCommonEventInfo info = 1;
AdapterDeprecationWarning data = 2;
}
// D012
message CollectFreshnessReturnSignature {
}
message CollectFreshnessReturnSignatureMsg {
AdapterCommonEventInfo info = 1;
CollectFreshnessReturnSignature data = 2;
}
// E - DB Adapter
// E001
message AdapterEventDebug {
AdapterNodeInfo node_info = 1;
string name = 2;
string base_msg = 3;
google.protobuf.ListValue args = 4;
}
message AdapterEventDebugMsg {
AdapterCommonEventInfo info = 1;
AdapterEventDebug data = 2;
}
// E002
message AdapterEventInfo {
AdapterNodeInfo node_info = 1;
string name = 2;
string base_msg = 3;
google.protobuf.ListValue args = 4;
}
message AdapterEventInfoMsg {
AdapterCommonEventInfo info = 1;
AdapterEventInfo data = 2;
}
// E003
message AdapterEventWarning {
AdapterNodeInfo node_info = 1;
string name = 2;
string base_msg = 3;
google.protobuf.ListValue args = 4;
}
message AdapterEventWarningMsg {
AdapterCommonEventInfo info = 1;
AdapterEventWarning data = 2;
}
// E004
message AdapterEventError {
AdapterNodeInfo node_info = 1;
string name = 2;
string base_msg = 3;
google.protobuf.ListValue args = 4;
string exc_info = 5;
}
message AdapterEventErrorMsg {
AdapterCommonEventInfo info = 1;
AdapterEventError data = 2;
}
// E005
message NewConnection {
AdapterNodeInfo node_info = 1;
string conn_type = 2;
string conn_name = 3;
}
message NewConnectionMsg {
AdapterCommonEventInfo info = 1;
NewConnection data = 2;
}
// E006
message ConnectionReused {
string conn_name = 1;
string orig_conn_name = 2;
}
message ConnectionReusedMsg {
AdapterCommonEventInfo info = 1;
ConnectionReused data = 2;
}
// E007
message ConnectionLeftOpenInCleanup {
string conn_name = 1;
}
message ConnectionLeftOpenInCleanupMsg {
AdapterCommonEventInfo info = 1;
ConnectionLeftOpenInCleanup data = 2;
}
// E008
message ConnectionClosedInCleanup {
string conn_name = 1;
}
message ConnectionClosedInCleanupMsg {
AdapterCommonEventInfo info = 1;
ConnectionClosedInCleanup data = 2;
}
// E009
message RollbackFailed {
AdapterNodeInfo node_info = 1;
string conn_name = 2;
string exc_info = 3;
}
message RollbackFailedMsg {
AdapterCommonEventInfo info = 1;
RollbackFailed data = 2;
}
// E010
message ConnectionClosed {
AdapterNodeInfo node_info = 1;
string conn_name = 2;
}
message ConnectionClosedMsg {
AdapterCommonEventInfo info = 1;
ConnectionClosed data = 2;
}
// E011
message ConnectionLeftOpen {
AdapterNodeInfo node_info = 1;
string conn_name = 2;
}
message ConnectionLeftOpenMsg {
AdapterCommonEventInfo info = 1;
ConnectionLeftOpen data = 2;
}
// E012
message Rollback {
AdapterNodeInfo node_info = 1;
string conn_name = 2;
}
message RollbackMsg {
AdapterCommonEventInfo info = 1;
Rollback data = 2;
}
// E013
message CacheMiss {
string conn_name = 1;
string database = 2;
string schema = 3;
}
message CacheMissMsg {
AdapterCommonEventInfo info = 1;
CacheMiss data = 2;
}
// E014
message ListRelations {
string database = 1;
string schema = 2;
repeated ReferenceKeyMsg relations = 3;
}
message ListRelationsMsg {
AdapterCommonEventInfo info = 1;
ListRelations data = 2;
}
// E015
message ConnectionUsed {
AdapterNodeInfo node_info = 1;
string conn_type = 2;
string conn_name = 3;
}
message ConnectionUsedMsg {
AdapterCommonEventInfo info = 1;
ConnectionUsed data = 2;
}
// E016
message SQLQuery {
AdapterNodeInfo node_info = 1;
string conn_name = 2;
string sql = 3;
}
message SQLQueryMsg {
AdapterCommonEventInfo info = 1;
SQLQuery data = 2;
}
// E017
message SQLQueryStatus {
AdapterNodeInfo node_info = 1;
string status = 2;
float elapsed = 3;
}
message SQLQueryStatusMsg {
AdapterCommonEventInfo info = 1;
SQLQueryStatus data = 2;
}
// E018
message SQLCommit {
AdapterNodeInfo node_info = 1;
string conn_name = 2;
}
message SQLCommitMsg {
AdapterCommonEventInfo info = 1;
SQLCommit data = 2;
}
// E019
message ColTypeChange {
string orig_type = 1;
string new_type = 2;
ReferenceKeyMsg table = 3;
}
message ColTypeChangeMsg {
AdapterCommonEventInfo info = 1;
ColTypeChange data = 2;
}
// E020
message SchemaCreation {
ReferenceKeyMsg relation = 1;
}
message SchemaCreationMsg {
AdapterCommonEventInfo info = 1;
SchemaCreation data = 2;
}
// E021
message SchemaDrop {
ReferenceKeyMsg relation = 1;
}
message SchemaDropMsg {
AdapterCommonEventInfo info = 1;
SchemaDrop data = 2;
}
// E022
message CacheAction {
string action = 1;
ReferenceKeyMsg ref_key = 2;
ReferenceKeyMsg ref_key_2 = 3;
ReferenceKeyMsg ref_key_3 = 4;
repeated ReferenceKeyMsg ref_list = 5;
}
message CacheActionMsg {
AdapterCommonEventInfo info = 1;
CacheAction data = 2;
}
// Skipping E023, E024, E025, E026, E027, E028, E029, E0230
// E031
message CacheDumpGraph {
map<string, string> dump = 1;
string before_after = 2;
string action = 3;
}
message CacheDumpGraphMsg {
AdapterCommonEventInfo info = 1;
CacheDumpGraph data = 2;
}
// Skipping E032, E033, E034
// E034
message AdapterRegistered {
string adapter_name = 1;
string adapter_version = 2;
}
message AdapterRegisteredMsg {
AdapterCommonEventInfo info = 1;
AdapterRegistered data = 2;
}
// E035
message AdapterImportError {
string exc = 1;
}
message AdapterImportErrorMsg {
AdapterCommonEventInfo info = 1;
AdapterImportError data = 2;
}
// E036
message PluginLoadError {
string exc_info = 1;
}
message PluginLoadErrorMsg {
AdapterCommonEventInfo info = 1;
PluginLoadError data = 2;
}
// E037
message NewConnectionOpening {
AdapterNodeInfo node_info = 1;
string connection_state = 2;
}
message NewConnectionOpeningMsg {
AdapterCommonEventInfo info = 1;
NewConnectionOpening data = 2;
}
// E038
message CodeExecution {
string conn_name = 1;
string code_content = 2;
}
message CodeExecutionMsg {
AdapterCommonEventInfo info = 1;
CodeExecution data = 2;
}
// E039
message CodeExecutionStatus {
string status = 1;
float elapsed = 2;
}
message CodeExecutionStatusMsg {
AdapterCommonEventInfo info = 1;
CodeExecutionStatus data = 2;
}
// E040
message CatalogGenerationError {
string exc = 1;
}
message CatalogGenerationErrorMsg {
AdapterCommonEventInfo info = 1;
CatalogGenerationError data = 2;
}
// E041
message WriteCatalogFailure {
int32 num_exceptions = 1;
}
message WriteCatalogFailureMsg {
AdapterCommonEventInfo info = 1;
WriteCatalogFailure data = 2;
}
// E042
message CatalogWritten {
string path = 1;
}
message CatalogWrittenMsg {
AdapterCommonEventInfo info = 1;
CatalogWritten data = 2;
}
// E043
message CannotGenerateDocs {
}
message CannotGenerateDocsMsg {
AdapterCommonEventInfo info = 1;
CannotGenerateDocs data = 2;
}
// E044
message BuildingCatalog {
}
message BuildingCatalogMsg {
AdapterCommonEventInfo info = 1;
BuildingCatalog data = 2;
}
// E045
message DatabaseErrorRunningHook {
string hook_type = 1;
}
message DatabaseErrorRunningHookMsg {
AdapterCommonEventInfo info = 1;
DatabaseErrorRunningHook data = 2;
}
// E046
message HooksRunning {
int32 num_hooks = 1;
string hook_type = 2;
}
message HooksRunningMsg {
AdapterCommonEventInfo info = 1;
HooksRunning data = 2;
}
// E047
message FinishedRunningStats {
string stat_line = 1;
string execution = 2;
float execution_time = 3;
}
message FinishedRunningStatsMsg {
AdapterCommonEventInfo info = 1;
FinishedRunningStats data = 2;
}
// E048
message ConstraintNotEnforced {
string constraint = 1;
string adapter = 2;
}
message ConstraintNotEnforcedMsg {
AdapterCommonEventInfo info = 1;
ConstraintNotEnforced data = 2;
}
// E049
message ConstraintNotSupported {
string constraint = 1;
string adapter = 2;
}
message ConstraintNotSupportedMsg {
AdapterCommonEventInfo info = 1;
ConstraintNotSupported data = 2;
}

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More