Compare commits

...

78 Commits

Author SHA1 Message Date
Emily Rockman
2ca8187c1f just testing 2023-09-08 09:57:03 -05:00
Quigley Malcolm
b39eeb328c Unskip and rename test_expression_metric (#8578)
* Add docstrings to `contracts/graph/metrics.py` functions to document what they do

Used [dbt-labs/dbt-core#5607](https://github.com/dbt-labs/dbt-core/pull/5607)
for context on what the functions should do.

* Add typing to `reverse_dag_parsing` and update function to work on 1.6+ metrics

* Add typing to `parent_metrics` and `parent_metrics_names`

* Add typing to `base_metric_dependency` and `derived_metric_dependency` and update functions to work on 1.6+ metrics

* Simplify implementations of `basic_metric_dependency` and `derived_metric_dependnecy`

* Add typing to `ResolvedMetricReference` initialization

* Add typing to `derived_metric_dependency_graph`

* Simplify conditional controls in `ResolvedMetricReference` functions

The functions in `ResolvedMetricReference` use `manifest.metric.get(...)`
which will only return either a `Metric` or `None`, never a different
node type. Thus we don't need to check that the returned metric is
a metric.

* Don't recurse on over `depends_on` for non-derived metrics in `reverse_dag_parsing`

The function `reverse_dag_parsing` only cares about derived metrics,
that is metrics that depend on other metrics. Metrics only depend on
other metrics if they are one of the `DERIVED_METRICS` types. Thus
doing a recursive call to `reverse_dag_parsing` for non `DERIVED_METRICS`
types is unnecessary. Previously we were iterating over a metric's
`depends_on` property regardless of whether the metric was a `DERIVED_METRICS`
type. Now we only do this work if the metric is of a `DERIVED_METRICS`
type.

* Simplify `parent_metrics_names` by having it call `parent_metrics`

* Unskip `TestMetricHelperFunctions.test_derived_metric` and update fixture setup

* Add changie doc for metric helper function updates

* Get manifest in `test_derived_metric` from the parse dbt_run invocation

* Remove `Relation` a intiatlization attribute for `ResolvedMetricReference`

* Add return typing to class `__` functions of `ResolvedMetricReference`

* Move from `manifest.metrics.get` to `manifest.expect` in metric helpers

Previously with `manifest.metrics.get` we were just skipping when `None`
was returned. Getting `None` back was expected in that `parent_unique_id`s
that didn't belong to metrics should return `None` when calling
`manifest.metrics.get`, and these are fine to skip. However, there's
an edgecase where a `parent_unique_id` is supposed to be a metric, but
isn't found, thus returning `None`. How likely this edge case could
get hit, I'm not sure, but it's a possible edge case. Using `manifest.metrics.get`
it we can't actually tell if we're in the edge case or not. By moving
to `manifest.expect` we get the error handling built in, and the only
trade off is that we need to change our conditional to skip returned
nodes that aren't metrics.
2023-09-07 15:28:36 -07:00
Emily Rockman
be94bf1f3c Preserve decimal places for dbt show (#8561)
* update `Number` class to handle integer values (#8306)

* add show test for json data

* oh changie my changie

* revert unecessary cahnge to fixture

* keep decimal class for precision methods, but return __int__ value

* jerco updates

* update integer type

* update other tests

* Update .changes/unreleased/Fixes-20230803-093502.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* account for integer vs number on table merges

* add tests for combining number with integer.

* add unit test when nulls are added

* cant none as an Integer

* fix null tests

---------

Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Dave Connors <dave.connors@fishtownanalytics.com>
2023-09-07 09:15:11 -05:00
Ben Mosher
e24a952e98 compile --no-inject-ephemeral-ctes flag (#8482) 2023-09-07 10:13:40 -04:00
Michelle Ark
89f20d12cf make UnparsedVersion.__lt__ order-agnostic (#8559) 2023-09-06 17:46:59 -04:00
Michelle Ark
ebeb0f1154 test advanced ref override (#8552) 2023-09-06 17:45:16 -04:00
Kshitij Aranke
d66fe214d9 Fix #8544: Parse the correct schema version from manifest (#8551)
* Fix #8544: Parse the correct schema version from manifest

* add changie

* add comment
2023-09-06 20:38:37 +01:00
Michelle Ark
75781503b8 add typing to partially typed methods in runnable.py (#8569) 2023-09-06 13:57:55 -04:00
Kshitij Aranke
9aff3ca274 Fix #8398: Add typing to __init__ in base.py (#8568) 2023-09-06 18:13:05 +01:00
Emily Rockman
7e2a08f3a5 remove label (#8557) 2023-09-05 19:12:16 -05:00
Emily Rockman
a0e13561b1 Support dbt-cloud config dict in dbt_project.yml (#8527)
* first pass at adding dbt-cloud config

* changelog

* fix test, add direct validation
2023-09-05 09:48:37 -05:00
FishtownBuildBot
7eedfcd274 [Automated] Merged prep-release/1.7.0b2_6049623160 into target main during release process 2023-09-01 08:48:37 -05:00
Github Build Bot
da779ac77c Bumping version to 1.7.0b2 and generate changelog 2023-09-01 12:51:37 +00:00
Mike Alfare
adfa3226e3 ADAP-814: Add support for replacing materialized views with tables/views and vice versa (#8449)
* first draft of adding in table - materialized view swap
* table/view/materialized view can all replace each other
* update renameable relations to a config
* migrate relations macros from `macros/adapters/relations` to `macros/relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar
* move drop macro to drop macro file
* align the behavior of get_drop_sql and drop_relation, adopt existing default from drop_relation
* add explicit ddl for drop statements instead of inheriting the default from dbt-core
* update replace macro dependent macros to align with naming standards
* update type for mashumaro, update related test
2023-08-31 20:41:45 -04:00
Quigley Malcolm
e5e1a272ff Fix untyped functions in core/dbt/context/base.py (#8525)
* Improve typing of `ContextMember` functions

* Improve typing of `Var` functions

* Improve typing of `ContextMeta.__new__`

* Improve typing `BaseContext` and functions

In addition to just adding parameter typing and return typing to
`BaseContext` functions. We also declared `_context_members_` and
`_context_attrs_` as properites of `BaseContext` this was necessary
because they're being accessed in the classes functions. However,
because they were being indirectly instantiated by the metaclass
`ContextMeta`, the properties weren't actually known to exist. By
adding declaring the properties on the `BaseContext`, we let mypy
know they exist.

* Remove bare `invocations` of `@contextmember` and `@contextproperty`, and add typing to them

Previously `contextmember` and `contextproperty` were 2-in-1 decorators.
This meant they could be invoked either as `@contextmember` or
`@contextmember('some_string')`. This was fine until we wanted to return
typing to the functions. In the instance where the bare decorator was used
(i.e. no `(...)` were present) an object was expected to be returned. However
in the instance where parameters were passed on the invocation, a callable
was expected to be returned. Putting a union of both in the return type
made the invocations complain about each others' return type. To get around this
we've dropped the bare invocation as acceptable. The parenthesis are now always
required, but passing a string in them is optional.
2023-08-31 13:34:57 -07:00
Gerda Shank
d8e8a78368 Fix snapshot success message to display "INSERT 0 1" (for example) instead of success (#8524) 2023-08-31 13:09:15 -04:00
Emily Rockman
7ae3de1fa0 Semantic model configs - enable/disable + groups (#8502)
* WIP

* WIP

* get group and enabled added

* changelog

* cleanup

* getting measure lookup working

* missed file

* get project level working

* fix last test

* add groups to config tests

* more group tests

* fix path

* clean up manifest.py

* update error message

* fix test assert

* remove extra check

* resolve conflicts in manaifest

* update manifest

* resolve conflict

* add alias
2023-08-30 20:18:22 -05:00
Gerda Shank
72898c7211 Add return value to AdapterContainer.__init__ and AdapterMeta.__new__ (#8523) 2023-08-30 17:37:32 -04:00
Peter Webb
fc1a14a0e3 Include Compiled Node Attributes in run_results.json (#8492)
* Add compiled node properties to run_results.json

* Include compiled-node attributes in run_results.json

* Fix typo

* Bump schema version of run_results

* Fix test assertions

* Update expected run_results to reflect new attributes

* Code review changes

* Fix mypy warnings for ManifestLoader.load() (#8443)

* revert python version for docker images (#8445)

* revert python version for docker images

* add comment to not update python version, update changelog

* Bumping version to 1.7.0b1 and generate changelog

* [CT-3013]  Fix parsing of `window_groupings` (#8454)

* Update semantic model parsing tests to check measure non_additive_dimension spec

* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`

* Add changie doc for `window_groupings`  parsing fix

* update `Number` class to handle integer values (#8306)

* add show test for json data

* oh changie my changie

* revert unecessary cahnge to fixture

* keep decimal class for precision methods, but return __int__ value

* jerco updates

* update integer type

* update other tests

* Update .changes/unreleased/Fixes-20230803-093502.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Improve docker image README (#8212)

* Improve docker image README

- Fix unnecessary/missing newline escapes
- Remove double whitespace between parameters
- 2-space indent for extra lines in image build commands

* Add changelog entry for #8212

* ADAP-814: Refactor prep for MV updates (#8459)

* apply reformatting changes only for #8449
* add logging back to get_create_materialized_view_as_sql
* changie

* swap trigger (#8463)

* update the implementation template (#8466)

* update the implementation template

* add colon

* Split tests into classes (#8474)

* add flaky decorator

* split up tests into classes

* revert update agate for int (#8478)

* updated typing and methods to meet mypy standards (#8485)

* Convert error to conditional warning for unversioned contracted model, fix msg format (#8451)

* first pass, tests need updates

* update proto defn

* fixing tests

* more test fixes

* finish fixing test file

* reformat the message

* formatting messages

* changelog

* add event to unit test

* feedback on message structure

* WIP

* fix up event to take in all fields

* fix test

* Fix ambiguous reference error for duplicate model names across packages with tests (#8488)

* Safely remove external nodes from manifest (#8495)

* [CT-2840] Improved semantic layer protocol satisfaction tests (#8456)

* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified

* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols

* Add test asserting Defaults obj satisfies protocol

* Add test asserting SemanticModel with optionals specified satisfies protocol

* Split dimension protocol satisfaction tests into with and without optionals

* Simplify DSI Protocol import strategy in protocol satisfaction tests

* Add test asserting DimensionValidtyParams satisfies protocol

* Add test asserting DimensionTypeParams satisfies protocol

* Split entity protocol satisfaction tests into with and without optionals

* Split measure protocol satisfication tests and add measure aggregation params satisficaition test

* Split metric protocol satisfaction test into optional specified an unspecified

Additionally, create where_filter pytest fixture

* Improve protocol satisfaction tests for MetricTypeParams and sub protocols

Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow

* Convert to using mashumaro jsonschema with acceptable performance (#8437)

* Regenerate run_results schema after merging in changes from main.

---------

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Jaime Martínez Rincón <jaime@jamezrin.name>
Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
2023-08-30 17:28:49 -04:00
Gerda Shank
f063e4e01c Convert to using mashumaro jsonschema with acceptable performance (#8437) 2023-08-30 14:06:59 -04:00
Quigley Malcolm
07372db906 [CT-2840] Improved semantic layer protocol satisfaction tests (#8456)
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified

* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols

* Add test asserting Defaults obj satisfies protocol

* Add test asserting SemanticModel with optionals specified satisfies protocol

* Split dimension protocol satisfaction tests into with and without optionals

* Simplify DSI Protocol import strategy in protocol satisfaction tests

* Add test asserting DimensionValidtyParams satisfies protocol

* Add test asserting DimensionTypeParams satisfies protocol

* Split entity protocol satisfaction tests into with and without optionals

* Split measure protocol satisfication tests and add measure aggregation params satisficaition test

* Split metric protocol satisfaction test into optional specified an unspecified

Additionally, create where_filter pytest fixture

* Improve protocol satisfaction tests for MetricTypeParams and sub protocols

Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
2023-08-29 09:40:30 -07:00
Michelle Ark
48d04e8141 Safely remove external nodes from manifest (#8495) 2023-08-28 14:10:34 -04:00
Michelle Ark
6234267242 Fix ambiguous reference error for duplicate model names across packages with tests (#8488) 2023-08-25 09:15:07 -04:00
Emily Rockman
1afbb87e99 Convert error to conditional warning for unversioned contracted model, fix msg format (#8451)
* first pass, tests need updates

* update proto defn

* fixing tests

* more test fixes

* finish fixing test file

* reformat the message

* formatting messages

* changelog

* add event to unit test

* feedback on message structure

* WIP

* fix up event to take in all fields

* fix test
2023-08-24 19:29:31 -05:00
Mike Alfare
d18a74ddb7 updated typing and methods to meet mypy standards (#8485) 2023-08-24 16:25:42 -04:00
Michelle Ark
4d3c6d9c7c revert update agate for int (#8478) 2023-08-23 14:06:52 -04:00
Emily Rockman
10f9724827 Split tests into classes (#8474)
* add flaky decorator

* split up tests into classes
2023-08-23 11:27:34 -05:00
Emily Rockman
582faa129e update the implementation template (#8466)
* update the implementation template

* add colon
2023-08-22 13:58:25 -05:00
Emily Rockman
4ec87a01e0 swap trigger (#8463) 2023-08-21 15:48:37 -05:00
Mike Alfare
ff98685dd6 ADAP-814: Refactor prep for MV updates (#8459)
* apply reformatting changes only for #8449
* add logging back to get_create_materialized_view_as_sql
* changie
2023-08-21 14:44:47 -04:00
Jaime Martínez Rincón
424f3d218a Improve docker image README (#8212)
* Improve docker image README

- Fix unnecessary/missing newline escapes
- Remove double whitespace between parameters
- 2-space indent for extra lines in image build commands

* Add changelog entry for #8212
2023-08-18 14:14:59 -05:00
dave-connors-3
661623f9f7 update Number class to handle integer values (#8306)
* add show test for json data

* oh changie my changie

* revert unecessary cahnge to fixture

* keep decimal class for precision methods, but return __int__ value

* jerco updates

* update integer type

* update other tests

* Update .changes/unreleased/Fixes-20230803-093502.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-08-18 14:01:08 -05:00
Quigley Malcolm
49397b4d7b [CT-3013] Fix parsing of window_groupings (#8454)
* Update semantic model parsing tests to check measure non_additive_dimension spec

* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`

* Add changie doc for `window_groupings`  parsing fix
2023-08-18 10:57:29 -07:00
FishtownBuildBot
0553fd817c [Automated] Merged prep-release/1.7.0b1_5895067219 into target main during release process 2023-08-17 15:04:21 -05:00
Github Build Bot
7ad971f720 Bumping version to 1.7.0b1 and generate changelog 2023-08-17 19:25:44 +00:00
Matthew McKnight
f485c13035 revert python version for docker images (#8445)
* revert python version for docker images

* add comment to not update python version, update changelog
2023-08-17 14:18:28 -05:00
Gerda Shank
c30b691164 Fix mypy warnings for ManifestLoader.load() (#8443) 2023-08-17 14:34:22 -04:00
Quigley Malcolm
d088d4493e Add doc string context to Identifier validion regex rule (#8440) 2023-08-17 10:11:03 -07:00
Emily Rockman
770f804325 Fix test failures (#8432)
* fail job when anything fails in previous matrix

* tweak wording

* PR feedback
2023-08-17 07:20:07 -05:00
Emily Rockman
37a29073de change trigger (#8418) 2023-08-16 19:49:59 -05:00
Peter Webb
17cd145f09 Temporarily disable test. (#8434) 2023-08-16 16:42:48 -04:00
Kshitij Aranke
ac539fd5cf Ignore .github and .changes directories for code coverage (#8424)
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-08-16 16:19:12 +01:00
Gerda Shank
048553ddc3 Fix using project-dir with list command and path selector (#8388) 2023-08-16 10:39:22 -04:00
Gerda Shank
dfe6b71fd9 Add return values to functions to fix mypy warnings (#8416) 2023-08-16 10:37:46 -04:00
Peter Webb
18ee93ca3a Fix run_results.json Performance Regression (#8413)
* Remedy performance regression by only writing run_results.json once.

* Write results before cleaning up connections.
2023-08-15 16:41:14 -04:00
Emily Rockman
cb4bc2d6e9 Automate opening docs issues (#8373)
* first pass

* WIP

* update issue body

* fix triggering label

* fix docs

* add better run name

* reduce complexity

* update description

* fix PR title

* point at workflow on main

* fix wording

* add label
2023-08-15 14:20:46 -05:00
Quigley Malcolm
b0451806ef [CT-2526] Add ability to automatically create metrics from semantic model measures (#8310)
* Update semantic model parsing test to check `create_metric = true` functionality

* Add `create_metric` boolean property to unparsed measure objects

* Begin creating metrics from measures with `create_metric = True`

* Add test ensuring partial parsing handles metrics generated from measures

* Ensure partial parsing appropriately deletes metrics generated from semantic models

* Add changie doc for  addition

* Separate generated metrics from parsed metrics for partial parsing

I was doing a demo earlier today of this branch (minus this commit)
and noticed something odd. When I changes a semantic model, metrics
that should have been technically uneffected would get dropped. Basically
if I made a change to a semantic model which had metrics in the same
file, and then ran parse, those metrics defined in the same file
would get dropped. Then with no other changes, if I ran parse again
they would come back. What was happening was that parsed metrics
and generated metrics were getting tracked the same way on the file
objects for partial parsing. In 0787a7c7b6
we began dropping all metrics tracked in a file objects when changes
to semantic models were detected. Since parsed metrics and generated
metrics were being tracked together on the file object, the parsed
metrics were getting dropped as well. In this commit we begin separating
out the tracking of generated metrics and parsed metrics on the
file object, and now only drop the generated metrics when semantic
models have a detected change.

* Assert in test that  semantic model partial parsing doesn't clobber regular metrics
2023-08-14 12:42:11 -07:00
Kshitij Aranke
b514e4c249 Fix #8350: add connection status into list of statuses for dbt debug (#8351) 2023-08-14 18:13:10 +01:00
Michelle Ark
8350dfead3 add --no-partial-parse-file-diff flag (#8361)
* add --no-partial-parse-file-diff flag

* changelog entry
2023-08-14 13:24:16 +02:00
Michelle Ark
34e6edbb13 Fix: deleting models that depend on external nodes (#8330) 2023-08-14 10:31:58 +02:00
FishtownBuildBot
27be92903e Add new index.html and changelog yaml files from dbt-docs (#8346) 2023-08-14 13:12:37 +08:00
Michelle Ark
9388030182 fix ModelNodeArgs.fqn (#8364) 2023-08-11 23:21:44 -04:00
Emily Rockman
b7aee3f5a4 add env vars to tox.ini (#8365)
* add env vars to tox.ini

* revert test
2023-08-11 10:49:41 -05:00
Michelle Ark
83ff38ab24 track plugin.get_nodes (#8336) 2023-08-10 11:33:11 -04:00
Michelle Ark
6603a44151 Detect changes to model access, deprecation_date, and latest_version in state:modified (#8264) 2023-08-10 11:29:02 -04:00
Kshitij Aranke
e69d4e7f14 Fix #8245: Add flag to codecov report (#8341) 2023-08-09 18:14:52 +01:00
d-kaneshiro
506f65e880 fixed comment util.py (#8222)
* fixed comments util.py

* add CHANGELOG entries
2023-08-09 09:24:38 -04:00
Gerda Shank
41bb52762b Bump manifest jsonschema to v11, update v10 schema (#8335) 2023-08-08 22:55:47 -04:00
Anju
8c98ef3e70 Copy dir if symlink fails (#7447) 2023-08-08 21:37:36 -04:00
Emily Rockman
44d1e73b4f Fix missing quote in fixtures.py (#8324) 2023-08-07 15:46:44 -05:00
Emily Rockman
53794fbaba Update implementation-ticket.yml (#8332) 2023-08-07 14:09:37 -06:00
Emily Rockman
556b4043e9 Update implementation-ticket.yml to reference adapters (#8329) 2023-08-07 12:39:29 -05:00
Grant Murray
424c636533 [CT-2776] [Feature] Enable-post-parse-population-of-dbt-custom-env (#7998)
* patch(events/functions): enable-repopulation-of-metadata-vars

* changie new
2023-08-04 12:59:47 -07:00
Emily Rockman
f63709260e add formatting events into json logs (#8308)
* add formatting events into json logs

* changelog

* Delete Under the Hood-20230803-100811.yaml
2023-08-03 15:56:01 -05:00
Michelle Ark
991618dfc1 capitalize integration-report name (#8265) 2023-08-02 18:06:05 -04:00
Chenyu Li
1af489b1cd fix constructing param with 0 value (#8298)
* fix constructing param with 0 value

* Update core/dbt/cli/flags.py

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

---------

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
2023-08-02 14:57:55 -07:00
Kshitij Aranke
a433c31d6e Fix #7179 (#8279) 2023-08-02 16:56:16 +01:00
Peter Webb
5814928e38 Issue One Event Per Node Failure (#8210)
* Replaced the FirstRunResultError and AfterFirstRunResultError events with RunResultError.

* Attempts at reasonable unit tests.

* Restore event manager after unit test.
2023-08-02 10:24:05 -04:00
Ramon Vermeulen
6130a6e1d0 Support configurable delimiter for seed files, default to comma (#3990) (#7186)
* Support configurable delimiter for seed files, default to comma (#3990)

* Update Features-20230317-144957.yaml

* Moved "delimiter" to seed config instead of node config

* Update core/dbt/clients/agate_helper.py

Co-authored-by: Cor <jczuurmond@protonmail.com>

* Update test_contracts_graph_parsed.py

* fixed integration tests

* Added functional tests for seed files with a unique delimiter

* Added docstrings

* Added a test for an empty string configured delimiter value

* whitespace

* ran black

* updated changie entry

* Update Features-20230317-144957.yaml

---------

Co-authored-by: Cor <jczuurmond@protonmail.com>
2023-08-01 09:15:43 -07:00
Quigley Malcolm
7872f6a670 Add tests for specifcally checking the population of SemanticModel.depends_on (#8226) 2023-07-31 14:15:09 -07:00
Emily Rockman
f230e418aa hard code test splits (#8258)
* change trigger

* add logic for different targets

* fix comment

* hard code test splits
2023-07-31 13:26:17 -05:00
Quigley Malcolm
518eb73f88 [CT-2888] Support dbt-semantic-interfaces 0.2.0 (#8250)
* Upgrade DSI dependency to ~=0.2.0

* Allow users to specify `primary_entity` on semantic models

* Add `primary_entity` and `primary_entity_reference` to SemanticModel node

* Plumb primary_entity from unparsed to parsed semantic nodes

* Fix metric filter specifications in existing tests

* Add changie doc about supporting DSI 0.2.0
2023-07-28 14:36:51 -07:00
Emily Rockman
5b6d21d7da loosen the click pin (#8232)
* loosen the click pin

* changelog

* separate out sqlparse pin

* remove changelog

* add ignores
2023-07-28 15:53:27 -05:00
Michelle Ark
410506f448 [Fix] raise execution errors for runnable tasks (#8237) 2023-07-28 13:18:51 -04:00
dependabot[bot]
3cb44d37c0 Bump mypy from 1.4.0 to 1.4.1 (#8219)
* Bump mypy from 1.4.0 to 1.4.1

Bumps [mypy](https://github.com/python/mypy) from 1.4.0 to 1.4.1.
- [Commits](https://github.com/python/mypy/compare/v1.4.0...v1.4.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

* update pre-commit

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
2023-07-28 07:33:16 -05:00
Quigley Malcolm
f977ed7471 [CT-2879] Fix unbound variable error in checked_agg_time_dimension_for_measure (#8235)
* Fix unbound variable error in `checked_agg_time_dimension_for_measure`

* Improve assertion error message in `checked_agg_time_dimension_for_measure`

* Add changie doc for checked_agg_time_dimension_for_measure unbound variable fix

* Add unit tests for checking functionality of `checked_agg_time_dimension_for_measure`
2023-07-27 14:58:59 -07:00
Emily Rockman
3f5617b569 pin upper bound for sqlparse (#8236)
* pin upper bound for sqlparse

* changelog
2023-07-27 16:29:42 -05:00
Gerda Shank
fe9c875d32 Ensure that target_schema from snapshot config is promoted to node level (#8117) 2023-07-27 13:40:52 -04:00
288 changed files with 13153 additions and 3133 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.7.0a1
current_version = 1.7.0b2
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

70
.changes/1.7.0-b1.md Normal file
View File

@@ -0,0 +1,70 @@
## dbt-core 1.7.0-b1 - August 17, 2023
### Breaking Changes
- Removed the FirstRunResultError and AfterFirstRunResultError event types, using the existing RunResultError in their place. ([#7963](https://github.com/dbt-labs/dbt-core/issues/7963))
### Features
- Enable re-population of metadata vars post-environment change during programmatic invocation ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- Added support to configure a delimiter for a seed file, defaults to comma ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- Allow specification of `create_metric: true` on measures ([#8125](https://github.com/dbt-labs/dbt-core/issues/8125))
### Fixes
- Copy dir during `dbt deps` if symlink fails ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- Fixed double-underline ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Enable converting deprecation warnings to errors ([#8130](https://github.com/dbt-labs/dbt-core/issues/8130))
- Add status to Parse Inline Error ([#8173](https://github.com/dbt-labs/dbt-core/issues/8173))
- Ensure `warn_error_options` get serialized in `invocation_args_dict` ([#7694](https://github.com/dbt-labs/dbt-core/issues/7694))
- Stop detecting materialization macros based on macro name ([#6231](https://github.com/dbt-labs/dbt-core/issues/6231))
- Update `dbt deps` download retry logic to handle `EOFError` exceptions ([#6653](https://github.com/dbt-labs/dbt-core/issues/6653))
- Improve handling of CTE injection with ephemeral models ([#8213](https://github.com/dbt-labs/dbt-core/issues/8213))
- Fix unbound local variable error in `checked_agg_time_dimension_for_measure` ([#8230](https://github.com/dbt-labs/dbt-core/issues/8230))
- Ensure runtime errors are raised for graph runnable tasks (compile, show, run, etc) ([#8166](https://github.com/dbt-labs/dbt-core/issues/8166))
- Fix retry not working with log-file-max-bytes ([#8297](https://github.com/dbt-labs/dbt-core/issues/8297))
- Detect changes to model access, version, or latest_version in state:modified ([#8189](https://github.com/dbt-labs/dbt-core/issues/8189))
- Add connection status into list of statuses for dbt debug ([#8350](https://github.com/dbt-labs/dbt-core/issues/8350))
- fix fqn-selection for external versioned models ([#8374](https://github.com/dbt-labs/dbt-core/issues/8374))
- Fix: DbtInternalError after model that previously ref'd external model is deleted ([#8375](https://github.com/dbt-labs/dbt-core/issues/8375))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
- Remedy performance regression by only writing run_results.json once. ([#8360](https://github.com/dbt-labs/dbt-core/issues/8360))
### Docs
- Corrected spelling of "Partiton" ([dbt-docs/#8100](https://github.com/dbt-labs/dbt-docs/issues/8100))
- Remove static SQL codeblock for metrics ([dbt-docs/#436](https://github.com/dbt-labs/dbt-docs/issues/436))
- fixed comment util.py ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Display contract and column constraints on the model page ([dbt-docs/#433](https://github.com/dbt-labs/dbt-docs/issues/433))
- Display semantic model details in docs ([dbt-docs/#431](https://github.com/dbt-labs/dbt-docs/issues/431))
### Under the Hood
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))
- format exception from dbtPlugin.initialize ([#8152](https://github.com/dbt-labs/dbt-core/issues/8152))
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
- Ref expressions with version can now be processed by the latest version of the high-performance dbt-extractor library. ([#7688](https://github.com/dbt-labs/dbt-core/issues/7688))
- Bump manifest schema version to v11, freeze manifest v10 ([#8333](https://github.com/dbt-labs/dbt-core/issues/8333))
- add tracking for plugin.get_nodes calls ([#8344](https://github.com/dbt-labs/dbt-core/issues/8344))
- add internal flag: --no-partial-parse-file-diff to inform whether to compute a file diff during partial parsing ([#8363](https://github.com/dbt-labs/dbt-core/issues/8363))
- Add return values to a number of functions for mypy ([#8389](https://github.com/dbt-labs/dbt-core/issues/8389))
- Fix mypy warnings for ManifestLoader.load() ([#8401](https://github.com/dbt-labs/dbt-core/issues/8401))
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))
### Dependencies
- Bump mypy from 1.3.0 to 1.4.0 ([#7912](https://github.com/dbt-labs/dbt-core/pull/7912))
- Bump mypy from 1.4.0 to 1.4.1 ([#8219](https://github.com/dbt-labs/dbt-core/pull/8219))
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))
- Support dbt-semantic-interfaces 0.2.0 ([#8250](https://github.com/dbt-labs/dbt-core/pull/8250))
### Contributors
- [@anjutiwari](https://github.com/anjutiwari) ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- [@d-kaneshiro](https://github.com/d-kaneshiro) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@gem7318](https://github.com/gem7318) ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- [@lllong33](https://github.com/lllong33) ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- [@marcodamore](https://github.com/marcodamore) ([#436](https://github.com/dbt-labs/dbt-core/issues/436))
- [@pgoslatara](https://github.com/pgoslatara) ([#8100](https://github.com/dbt-labs/dbt-core/issues/8100))
- [@ramonvermeulen](https://github.com/ramonvermeulen) ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))

29
.changes/1.7.0-b2.md Normal file
View File

@@ -0,0 +1,29 @@
## dbt-core 1.7.0-b2 - September 01, 2023
### Features
- Add node attributes related to compilation to run_results.json ([#7519](https://github.com/dbt-labs/dbt-core/issues/7519))
- Support configuration of semantic models with the addition of enable/disable and group enablement. ([#7968](https://github.com/dbt-labs/dbt-core/issues/7968))
### Fixes
- Add support for swapping materialized views with tables/views and vice versa ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Turn breaking changes to contracted models into warnings for unversioned models ([#8384](https://github.com/dbt-labs/dbt-core/issues/8384), [#8282](https://github.com/dbt-labs/dbt-core/issues/8282))
- Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension` ([#8453](https://github.com/dbt-labs/dbt-core/issues/8453))
- fix ambiguous reference error for tests and versions when model name is duplicated across packages ([#8327](https://github.com/dbt-labs/dbt-core/issues/8327), [#8493](https://github.com/dbt-labs/dbt-core/issues/8493))
- Fix "Internal Error: Expected node <unique-id> not found in manifest" when depends_on set on ModelNodeArgs ([#8506](https://github.com/dbt-labs/dbt-core/issues/8506))
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
### Docs
- Fix newline escapes and improve formatting in docker README ([dbt-docs/#8211](https://github.com/dbt-labs/dbt-docs/issues/8211))
### Under the Hood
- Switch from hologram to mashumaro jsonschema ([#8426](https://github.com/dbt-labs/dbt-core/issues/8426))
- Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>, relation agnostic in /macros/relations ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Update typing to meet mypy standards ([#8396](https://github.com/dbt-labs/dbt-core/issues/8396))
- Mypy errors - adapters/factory.py ([#8387](https://github.com/dbt-labs/dbt-core/issues/8387))
### Contributors
- [@jamezrin](https://github.com/jamezrin) ([#8211](https://github.com/dbt-labs/dbt-core/issues/8211))

View File

@@ -0,0 +1,7 @@
kind: Breaking Changes
body: Removed the FirstRunResultError and AfterFirstRunResultError event types, using
the existing RunResultError in their place.
time: 2023-07-25T17:13:59.441682-04:00
custom:
Author: peterallenwebb
Issue: "7963"

View File

@@ -0,0 +1,6 @@
kind: "Dependencies"
body: "Bump mypy from 1.4.0 to 1.4.1"
time: 2023-07-26T20:17:40.00000Z
custom:
Author: dependabot[bot]
PR: 8219

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Update pin for click<9
time: 2023-07-27T14:57:03.180458-05:00
custom:
Author: emmyoop
PR: "8232"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Add upper bound to sqlparse pin of <0.5
time: 2023-07-27T14:57:26.40416-05:00
custom:
Author: emmyoop
PR: "8236"

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Support dbt-semantic-interfaces 0.2.0
time: 2023-07-28T13:52:27.207241-07:00
custom:
Author: QMalcolm
PR: "8250"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: fixed comment util.py
time: 2023-07-27T17:09:00.089237+09:00
custom:
Author: d-kaneshiro
Issue: None

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Fix newline escapes and improve formatting in docker README
time: 2023-07-28T19:34:38.351042747+02:00
custom:
Author: jamezrin
Issue: "8211"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Display contract and column constraints on the model page
time: 2023-08-04T13:18:15.627005-05:00
custom:
Author: emmyoop
Issue: "433"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Display semantic model details in docs
time: 2023-08-07T15:25:48.711627-05:00
custom:
Author: emmyoop
Issue: "431"

View File

@@ -0,0 +1,7 @@
kind: Features
body: Enable re-population of metadata vars post-environment change during programmatic
invocation
time: 2023-07-02T12:28:13.416305-04:00
custom:
Author: gem7318
Issue: "8010"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Added support to configure a delimiter for a seed file, defaults to comma
time: 2023-07-14T20:24:45.513847165+02:00
custom:
Author: ramonvermeulen
Issue: "3990"

View File

@@ -0,0 +1,6 @@
kind: Features
body: 'Allow specification of `create_metric: true` on measures'
time: 2023-08-03T15:18:24.351003-07:00
custom:
Author: QMalcolm
Issue: "8125"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add node attributes related to compilation to run_results.json
time: 2023-08-21T10:33:57.200883-04:00
custom:
Author: peterallenwebb
Issue: "7519"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Support configuration of semantic models with the addition of enable/disable and group enablement.
time: 2023-08-28T09:21:00.551633-05:00
custom:
Author: emmyoop
Issue: "7968"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Copy dir during `dbt deps` if symlink fails
time: 2023-04-24T21:07:34.336797+05:30
custom:
Author: anjutiwari
Issue: "7428 8223"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Copy target_schema from config into snapshot node
time: 2023-07-17T16:06:52.957724-04:00
custom:
Author: gshank
Issue: "6745"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix unbound local variable error in `checked_agg_time_dimension_for_measure`
time: 2023-07-27T12:58:30.673803-07:00
custom:
Author: QMalcolm
Issue: "8230"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Ensure runtime errors are raised for graph runnable tasks (compile, show, run,
etc)
time: 2023-07-28T11:56:20.863718-04:00
custom:
Author: michelleark
Issue: "8166"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix retry not working with log-file-max-bytes
time: 2023-08-02T14:15:56.306027-07:00
custom:
Author: ChenyuLInx
Issue: "8297"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Detect changes to model access, version, or latest_version in state:modified
time: 2023-08-06T22:23:19.166334-04:00
custom:
Author: michelleark
Issue: "8189"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add connection status into list of statuses for dbt debug
time: 2023-08-10T18:48:59.221344+01:00
custom:
Author: aranke
Issue: "8350"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: fix fqn-selection for external versioned models
time: 2023-08-11T20:41:44.725144-04:00
custom:
Author: michelleark
Issue: "8374"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: 'Fix: DbtInternalError after model that previously ref''d external model is
deleted'
time: 2023-08-11T21:20:08.145554-04:00
custom:
Author: michelleark
Issue: "8375"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix using list command with path selector and project-dir
time: 2023-08-14T14:57:02.02816-04:00
custom:
Author: gshank
Issue: "8385"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Remedy performance regression by only writing run_results.json once.
time: 2023-08-15T10:44:44.836991-04:00
custom:
Author: peterallenwebb
Issue: "8360"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Use python version 3.10.7 in Docker image.
time: 2023-08-17T13:09:15.936349-05:00
custom:
Author: McKnight-42
Issue: "8444"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add support for swapping materialized views with tables/views and vice versa
time: 2023-08-17T18:57:39.01958-04:00
custom:
Author: mikealfare
Issue: "8449"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension`
time: 2023-08-18T09:53:48.154848-07:00
custom:
Author: QMalcolm
Issue: "8453"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Turn breaking changes to contracted models into warnings for unversioned models
time: 2023-08-18T10:38:02.251286-05:00
custom:
Author: emmyoop
Issue: 8384 8282

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: fix ambiguous reference error for tests and versions when model name is duplicated across
packages
time: 2023-08-24T16:10:24.437362-04:00
custom:
Author: michelleark
Issue: "8327 8493"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: 'Fix "Internal Error: Expected node <unique-id> not found in manifest" when
depends_on set on ModelNodeArgs'
time: 2023-08-28T12:58:58.061228-04:00
custom:
Author: michelleark
Issue: "8506"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Fix snapshot success message
time: 2023-08-30T15:08:03.429373-04:00
custom:
Author: gshank
Issue: "7583"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Switch from hologram to mashumaro jsonschema
time: 2023-07-18T14:54:28.41453-04:00
custom:
Author: gshank
Issue: "8426"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Bump manifest schema version to v11, freeze manifest v10
time: 2023-08-07T16:45:09.712744-04:00
custom:
Author: gshank
Issue: "8333"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: add tracking for plugin.get_nodes calls
time: 2023-08-09T09:48:34.819445-04:00
custom:
Author: michelleark
Issue: "8344"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: 'add internal flag: --no-partial-parse-file-diff to inform whether to compute
a file diff during partial parsing'
time: 2023-08-11T10:09:02.832241-04:00
custom:
Author: michelleark
Issue: "8363"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add return values to a number of functions for mypy
time: 2023-08-15T17:03:07.895252-04:00
custom:
Author: gshank
Issue: "8389"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix mypy warnings for ManifestLoader.load()
time: 2023-08-17T13:45:48.937252-04:00
custom:
Author: gshank
Issue: "8401"

View File

@@ -0,0 +1,7 @@
kind: Under the Hood
body: 'Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>,
relation agnostic in /macros/relations'
time: 2023-08-21T13:48:01.474731-04:00
custom:
Author: mikealfare
Issue: "8449"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Update typing to meet mypy standards
time: 2023-08-23T19:42:37.130694-04:00
custom:
Author: mikealfare
Issue: "8396"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Mypy errors - adapters/factory.py
time: 2023-08-30T14:02:31.519929-04:00
custom:
Author: gshank
Issue: "8387"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Add --no-inject-ephemeral-ctes flag for `compile` command, for usage by linting.
time: 2023-08-23T14:04:07.617476-04:00
custom:
Author: benmosher
Issue: "8480"

View File

@@ -0,0 +1,6 @@
kind: Features
body: Accept a `dbt-cloud` config in dbt_project.yml
time: 2023-08-30T21:28:28.976746-05:00
custom:
Author: emmyoop
Issue: "8438"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add explicit support for integers for the show command
time: 2023-08-03T09:35:02.163968-05:00
custom:
Author: dave-connors-3
Issue: "8153"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Update metric helper functions to work with new semantic layer metrics
time: 2023-09-06T12:02:12.156534-07:00
custom:
Author: QMalcolm
Issue: "8134"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: make version comparison insensitive to order
time: 2023-09-06T14:22:13.114549-04:00
custom:
Author: michelleark
Issue: "8571"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Parse the correct schema version from manifest
time: 2023-09-06T16:24:27.849069+01:00
custom:
Author: aranke
Issue: "8544"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add functional test for advanced ref override
time: 2023-09-06T11:02:49.253486-04:00
custom:
Author: michelleark
Issue: "8566"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Fix untyped functions in task/runnable.py (mypy warning)
time: 2023-09-06T13:24:35.448782-04:00
custom:
Author: michelleark
Issue: "8402"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add typing to __init__ in base.py
time: 2023-09-06T16:49:01.150713+01:00
custom:
Author: aranke
Issue: "8398"

View File

@@ -1,7 +1,7 @@
name: 🛠️ Implementation
description: This is an implementation ticket intended for use by the maintainers of dbt-core
title: "[<project>] <title>"
labels: ["user_docs"]
labels: ["user docs"]
body:
- type: markdown
attributes:
@@ -11,7 +11,7 @@ body:
label: Housekeeping
description: >
A couple friendly reminders:
1. Remove the `user_docs` label if the scope of this work does not require changes to https://docs.getdbt.com/docs: no end-user interface (e.g. yml spec, CLI, error messages, etc) or functional changes
1. Remove the `user docs` label if the scope of this work does not require changes to https://docs.getdbt.com/docs: no end-user interface (e.g. yml spec, CLI, error messages, etc) or functional changes
2. Link any blocking issues in the "Blocked on" field under the "Core devs & maintainers" project.
options:
- label: I am a maintainer of dbt-core
@@ -25,11 +25,29 @@ body:
required: true
- type: textarea
attributes:
label: Acceptance critera
label: Acceptance criteria
description: |
What is the definition of done for this ticket? Include any relevant edge cases and/or test cases
validations:
required: true
- type: textarea
attributes:
label: Impact to Other Teams
description: |
Will this change impact other teams? Include details of the kinds of changes required (new tests, code changes, related tickets) and _add the relevant `Impact:[team]` label_.
placeholder: |
Example: This change impacts `dbt-redshift` because the tests will need to be modified. The `Impact:[Adapter]` label has been added.
validations:
required: true
- type: textarea
attributes:
label: Will backports be required?
description: |
Will this change need to be backported to previous versions? Add details, possible blockers to backporting and _add the relevant backport labels `backport 1.x.latest`_
placeholder: |
Example: Backport to 1.6.latest, 1.5.latest and 1.4.latest. Since 1.4 isn't using click, the backport may be complicated. The `backport 1.6.latest`, `backport 1.5.latest` and `backport 1.4.latest` labels have been added.
validations:
required: true
- type: textarea
attributes:
label: Context

View File

@@ -2,10 +2,8 @@
# Checks that a file has been committed under the /.changes directory
# as a new CHANGELOG entry. Cannot check for a specific filename as
# it is dynamically generated by change type and timestamp.
# This workflow should not require any secrets since it runs for PRs
# from forked repos.
# By default, secrets are not passed to workflows running from
# a forked repo.
# This workflow runs on pull_request_target because it requires
# secrets to post comments.
# **why?**
# Ensure code change gets reflected in the CHANGELOG.
@@ -19,7 +17,7 @@
name: Check Changelog Entry
on:
pull_request:
pull_request_target:
types: [opened, reopened, labeled, unlabeled, synchronize]
workflow_dispatch:

36
.github/workflows/docs-issue.yml vendored Normal file
View File

@@ -0,0 +1,36 @@
# **what?**
# Open an issue in docs.getdbt.com when a PR is labeled `user docs`
# **why?**
# To reduce barriers for keeping docs up to date
# **when?**
# When a PR is labeled `user docs` and is merged. Runs on pull_request_target to run off the workflow already merged,
# not the workflow that existed on the PR branch. This allows old PRs to get comments.
name: Open issues in docs.getdbt.com repo when a PR is labeled
run-name: "Open an issue in docs.getdbt.com for PR #${{ github.event.pull_request.number }}"
on:
pull_request_target:
types: [labeled, closed]
defaults:
run:
shell: bash
permissions:
issues: write # opens new issues
pull-requests: write # comments on PRs
jobs:
open_issues:
if: contains( github.event.pull_request.labels.*.name, 'user docs') && github.event.pull_request.merged == true
uses: dbt-labs/actions/.github/workflows/open-issue-in-repo.yml@main
with:
issue_repository: "dbt-labs/docs.getdbt.com"
issue_title: "Docs Changes Needed from ${{ github.event.repository.name }} PR #${{ github.event.pull_request.number }}"
issue_body: "At a minimum, update body to include a link to the page on docs.getdbt.com requiring updates and what part(s) of the page you would like to see updated."
secrets: inherit

View File

@@ -36,7 +36,7 @@ defaults:
# top-level adjustments can be made here
env:
# number of parallel processes to spawn for python integration testing
PYTHON_INTEGRATION_TEST_WORKERS: ${{ vars.PYTHON_INTEGRATION_TEST_WORKERS }}
PYTHON_INTEGRATION_TEST_WORKERS: 5
jobs:
code-quality:
@@ -108,8 +108,9 @@ jobs:
- name: Upload Unit Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: unit
integration-metadata:
name: integration test metadata generation
@@ -221,17 +222,26 @@ jobs:
- name: Upload Integration Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v3
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: integration
integration-report:
name: integration test suite
if: ${{ always() }}
name: Integration Test Suite
runs-on: ubuntu-latest
needs: integration
steps:
- name: "[Notification] Integration test suite passes"
- name: "Integration Tests Failed"
if: ${{ contains(needs.integration.result, 'failure') || contains(needs.integration.result, 'cancelled') }}
# when this is true the next step won't execute
run: |
echo "::notice title="Integration test suite passes""
echo "::notice title='Integration test suite failed'"
exit 1
- name: "Integration Tests Passed"
run: |
echo "::notice title='Integration test suite passed'"
build:
name: build packages

View File

@@ -21,7 +21,7 @@ permissions: read-all
# top-level adjustments can be made here
env:
# number of parallel processes to spawn for python testing
PYTHON_INTEGRATION_TEST_WORKERS: ${{ vars.PYTHON_INTEGRATION_TEST_WORKERS }}
PYTHON_INTEGRATION_TEST_WORKERS: 5
jobs:
integration-metadata:

View File

@@ -37,7 +37,7 @@ repos:
alias: flake8-check
stages: [manual]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.4.0
rev: v1.4.1
hooks:
- id: mypy
# N.B.: Mypy is... a bit fragile.

View File

@@ -5,6 +5,108 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)
## dbt-core 1.7.0-b2 - September 01, 2023
### Features
- Add node attributes related to compilation to run_results.json ([#7519](https://github.com/dbt-labs/dbt-core/issues/7519))
- Support configuration of semantic models with the addition of enable/disable and group enablement. ([#7968](https://github.com/dbt-labs/dbt-core/issues/7968))
### Fixes
- Add support for swapping materialized views with tables/views and vice versa ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Turn breaking changes to contracted models into warnings for unversioned models ([#8384](https://github.com/dbt-labs/dbt-core/issues/8384), [#8282](https://github.com/dbt-labs/dbt-core/issues/8282))
- Ensure parsing does not break when `window_groupings` is not specified for `non_additive_dimension` ([#8453](https://github.com/dbt-labs/dbt-core/issues/8453))
- fix ambiguous reference error for tests and versions when model name is duplicated across packages ([#8327](https://github.com/dbt-labs/dbt-core/issues/8327), [#8493](https://github.com/dbt-labs/dbt-core/issues/8493))
- Fix "Internal Error: Expected node <unique-id> not found in manifest" when depends_on set on ModelNodeArgs ([#8506](https://github.com/dbt-labs/dbt-core/issues/8506))
- Fix snapshot success message ([#7583](https://github.com/dbt-labs/dbt-core/issues/7583))
### Docs
- Fix newline escapes and improve formatting in docker README ([dbt-docs/#8211](https://github.com/dbt-labs/dbt-docs/issues/8211))
### Under the Hood
- Switch from hologram to mashumaro jsonschema ([#8426](https://github.com/dbt-labs/dbt-core/issues/8426))
- Re-organize jinja macros: relation-specific in /macros/adapters/relations/<relation>, relation agnostic in /macros/relations ([#8449](https://github.com/dbt-labs/dbt-core/issues/8449))
- Update typing to meet mypy standards ([#8396](https://github.com/dbt-labs/dbt-core/issues/8396))
- Mypy errors - adapters/factory.py ([#8387](https://github.com/dbt-labs/dbt-core/issues/8387))
### Contributors
- [@jamezrin](https://github.com/jamezrin) ([#8211](https://github.com/dbt-labs/dbt-core/issues/8211))
## dbt-core 1.7.0-b1 - August 17, 2023
### Breaking Changes
- Removed the FirstRunResultError and AfterFirstRunResultError event types, using the existing RunResultError in their place. ([#7963](https://github.com/dbt-labs/dbt-core/issues/7963))
### Features
- Enable re-population of metadata vars post-environment change during programmatic invocation ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- Added support to configure a delimiter for a seed file, defaults to comma ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
- Allow specification of `create_metric: true` on measures ([#8125](https://github.com/dbt-labs/dbt-core/issues/8125))
### Fixes
- Copy dir during `dbt deps` if symlink fails ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- Fixed double-underline ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- Copy target_schema from config into snapshot node ([#6745](https://github.com/dbt-labs/dbt-core/issues/6745))
- Enable converting deprecation warnings to errors ([#8130](https://github.com/dbt-labs/dbt-core/issues/8130))
- Add status to Parse Inline Error ([#8173](https://github.com/dbt-labs/dbt-core/issues/8173))
- Ensure `warn_error_options` get serialized in `invocation_args_dict` ([#7694](https://github.com/dbt-labs/dbt-core/issues/7694))
- Stop detecting materialization macros based on macro name ([#6231](https://github.com/dbt-labs/dbt-core/issues/6231))
- Update `dbt deps` download retry logic to handle `EOFError` exceptions ([#6653](https://github.com/dbt-labs/dbt-core/issues/6653))
- Improve handling of CTE injection with ephemeral models ([#8213](https://github.com/dbt-labs/dbt-core/issues/8213))
- Fix unbound local variable error in `checked_agg_time_dimension_for_measure` ([#8230](https://github.com/dbt-labs/dbt-core/issues/8230))
- Ensure runtime errors are raised for graph runnable tasks (compile, show, run, etc) ([#8166](https://github.com/dbt-labs/dbt-core/issues/8166))
- Fix retry not working with log-file-max-bytes ([#8297](https://github.com/dbt-labs/dbt-core/issues/8297))
- Detect changes to model access, version, or latest_version in state:modified ([#8189](https://github.com/dbt-labs/dbt-core/issues/8189))
- Add connection status into list of statuses for dbt debug ([#8350](https://github.com/dbt-labs/dbt-core/issues/8350))
- fix fqn-selection for external versioned models ([#8374](https://github.com/dbt-labs/dbt-core/issues/8374))
- Fix: DbtInternalError after model that previously ref'd external model is deleted ([#8375](https://github.com/dbt-labs/dbt-core/issues/8375))
- Fix using list command with path selector and project-dir ([#8385](https://github.com/dbt-labs/dbt-core/issues/8385))
- Remedy performance regression by only writing run_results.json once. ([#8360](https://github.com/dbt-labs/dbt-core/issues/8360))
### Docs
- Corrected spelling of "Partiton" ([dbt-docs/#8100](https://github.com/dbt-labs/dbt-docs/issues/8100))
- Remove static SQL codeblock for metrics ([dbt-docs/#436](https://github.com/dbt-labs/dbt-docs/issues/436))
- fixed comment util.py ([dbt-docs/#None](https://github.com/dbt-labs/dbt-docs/issues/None))
- Display contract and column constraints on the model page ([dbt-docs/#433](https://github.com/dbt-labs/dbt-docs/issues/433))
- Display semantic model details in docs ([dbt-docs/#431](https://github.com/dbt-labs/dbt-docs/issues/431))
### Under the Hood
- Refactor flaky test pp_versioned_models ([#7781](https://github.com/dbt-labs/dbt-core/issues/7781))
- format exception from dbtPlugin.initialize ([#8152](https://github.com/dbt-labs/dbt-core/issues/8152))
- A way to control maxBytes for a single dbt.log file ([#8199](https://github.com/dbt-labs/dbt-core/issues/8199))
- Ref expressions with version can now be processed by the latest version of the high-performance dbt-extractor library. ([#7688](https://github.com/dbt-labs/dbt-core/issues/7688))
- Bump manifest schema version to v11, freeze manifest v10 ([#8333](https://github.com/dbt-labs/dbt-core/issues/8333))
- add tracking for plugin.get_nodes calls ([#8344](https://github.com/dbt-labs/dbt-core/issues/8344))
- add internal flag: --no-partial-parse-file-diff to inform whether to compute a file diff during partial parsing ([#8363](https://github.com/dbt-labs/dbt-core/issues/8363))
- Add return values to a number of functions for mypy ([#8389](https://github.com/dbt-labs/dbt-core/issues/8389))
- Fix mypy warnings for ManifestLoader.load() ([#8401](https://github.com/dbt-labs/dbt-core/issues/8401))
- Use python version 3.10.7 in Docker image. ([#8444](https://github.com/dbt-labs/dbt-core/issues/8444))
### Dependencies
- Bump mypy from 1.3.0 to 1.4.0 ([#7912](https://github.com/dbt-labs/dbt-core/pull/7912))
- Bump mypy from 1.4.0 to 1.4.1 ([#8219](https://github.com/dbt-labs/dbt-core/pull/8219))
- Update pin for click<9 ([#8232](https://github.com/dbt-labs/dbt-core/pull/8232))
- Add upper bound to sqlparse pin of <0.5 ([#8236](https://github.com/dbt-labs/dbt-core/pull/8236))
- Support dbt-semantic-interfaces 0.2.0 ([#8250](https://github.com/dbt-labs/dbt-core/pull/8250))
### Contributors
- [@anjutiwari](https://github.com/anjutiwari) ([#7428](https://github.com/dbt-labs/dbt-core/issues/7428), [#8223](https://github.com/dbt-labs/dbt-core/issues/8223))
- [@d-kaneshiro](https://github.com/d-kaneshiro) ([#None](https://github.com/dbt-labs/dbt-core/issues/None))
- [@gem7318](https://github.com/gem7318) ([#8010](https://github.com/dbt-labs/dbt-core/issues/8010))
- [@lllong33](https://github.com/lllong33) ([#5301](https://github.com/dbt-labs/dbt-core/issues/5301))
- [@marcodamore](https://github.com/marcodamore) ([#436](https://github.com/dbt-labs/dbt-core/issues/436))
- [@pgoslatara](https://github.com/pgoslatara) ([#8100](https://github.com/dbt-labs/dbt-core/issues/8100))
- [@ramonvermeulen](https://github.com/ramonvermeulen) ([#3990](https://github.com/dbt-labs/dbt-core/issues/3990))
## Previous Releases
For information on prior major and minor releases, see their changelogs:

View File

@@ -1,3 +1,4 @@
<!-- unneeded change -->
# Contributing to `dbt-core`
`dbt-core` is open source software. It is what it is today because community members have opened issues, provided feedback, and [contributed to the knowledge loop](https://www.getdbt.com/dbt-labs/values/). Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.

View File

@@ -0,0 +1,9 @@
ignore:
- ".github"
- ".changes"
coverage:
status:
project:
default:
target: auto
threshold: 0.01% # Reduce noise by ignoring rounding errors in coverage drops

View File

@@ -400,7 +400,7 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
@abc.abstractmethod
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False
self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
) -> Tuple[AdapterResponse, agate.Table]:
"""Execute the given SQL.
@@ -408,7 +408,28 @@ class BaseConnectionManager(metaclass=abc.ABCMeta):
:param bool auto_begin: If set, and dbt is not currently inside a
transaction, automatically begin one.
:param bool fetch: If set, fetch results.
:param int limit: If set, limits the result set
:return: A tuple of the query status and results (empty if fetch=False).
:rtype: Tuple[AdapterResponse, agate.Table]
"""
raise dbt.exceptions.NotImplementedError("`execute` is not implemented for this adapter!")
def add_select_query(self, sql: str) -> Tuple[Connection, Any]:
"""
This was added here because base.impl.BaseAdapter.get_column_schema_from_query expects it to be here.
That method wouldn't work unless the adapter used sql.impl.SQLAdapter, sql.connections.SQLConnectionManager
or defined this method on <Adapter>ConnectionManager before passing it in to <Adapter>Adapter.
See https://github.com/dbt-labs/dbt-core/issues/8396 for more information.
"""
raise dbt.exceptions.NotImplementedError(
"`add_select_query` is not implemented for this adapter!"
)
@classmethod
def data_type_code_to_name(cls, type_code: Union[int, str]) -> str:
"""Get the string representation of the data type from the type_code."""
# https://peps.python.org/pep-0249/#type-objects
raise dbt.exceptions.NotImplementedError(
"`data_type_code_to_name` is not implemented for this adapter!"
)

View File

@@ -43,7 +43,7 @@ from dbt.exceptions import (
UnexpectedNullError,
)
from dbt.adapters.protocol import AdapterConfig, ConnectionManagerProtocol
from dbt.adapters.protocol import AdapterConfig
from dbt.clients.agate_helper import empty_table, merge_tables, table_from_rows
from dbt.clients.jinja import MacroGenerator
from dbt.contracts.graph.manifest import Manifest, MacroManifest
@@ -60,7 +60,7 @@ from dbt.events.types import (
)
from dbt.utils import filter_null_values, executor, cast_to_str, AttrDict
from dbt.adapters.base.connections import Connection, AdapterResponse
from dbt.adapters.base.connections import Connection, AdapterResponse, BaseConnectionManager
from dbt.adapters.base.meta import AdapterMeta, available
from dbt.adapters.base.relation import (
ComponentName,
@@ -208,7 +208,7 @@ class BaseAdapter(metaclass=AdapterMeta):
Relation: Type[BaseRelation] = BaseRelation
Column: Type[BaseColumn] = BaseColumn
ConnectionManager: Type[ConnectionManagerProtocol]
ConnectionManager: Type[BaseConnectionManager]
# A set of clobber config fields accepted by this adapter
# for use in materializations
@@ -315,14 +315,21 @@ class BaseAdapter(metaclass=AdapterMeta):
@available.parse(lambda *a, **k: ("", empty_table()))
def get_partitions_metadata(self, table: str) -> Tuple[agate.Table]:
"""Obtain partitions metadata for a BigQuery partitioned table.
"""
TODO: Can we move this to dbt-bigquery?
Obtain partitions metadata for a BigQuery partitioned table.
:param str table_id: a partitioned table id, in standard SQL format.
:param str table: a partitioned table id, in standard SQL format.
:return: a partition metadata tuple, as described in
https://cloud.google.com/bigquery/docs/creating-partitioned-tables#getting_partition_metadata_using_meta_tables.
:rtype: agate.Table
"""
return self.connections.get_partitions_metadata(table=table)
if hasattr(self.connections, "get_partitions_metadata"):
return self.connections.get_partitions_metadata(table=table)
else:
raise NotImplementedError(
"`get_partitions_metadata` is not implemented for this adapter!"
)
###
# Methods that should never be overridden
@@ -453,9 +460,10 @@ class BaseAdapter(metaclass=AdapterMeta):
# it's possible that there were no relations in some schemas. We want
# to insert the schemas we query into the cache's `.schemas` attribute
# so we can check it later
cache_update: Set[Tuple[Optional[str], Optional[str]]] = set()
cache_update: Set[Tuple[Optional[str], str]] = set()
for relation in cache_schemas:
cache_update.add((relation.database, relation.schema))
if relation.schema:
cache_update.add((relation.database, relation.schema))
self.cache.update_schemas(cache_update)
def set_relations_cache(

View File

@@ -93,7 +93,7 @@ class AdapterMeta(abc.ABCMeta):
_available_: FrozenSet[str]
_parse_replacements_: Dict[str, Callable]
def __new__(mcls, name, bases, namespace, **kwargs):
def __new__(mcls, name, bases, namespace, **kwargs) -> "AdapterMeta":
# mypy does not like the `**kwargs`. But `ABCMeta` itself takes
# `**kwargs` in its argspec here (and passes them to `type.__new__`.
# I'm not sure there is any benefit to it after poking around a bit,

View File

@@ -25,9 +25,9 @@ class _QueryComment(local):
- a source_name indicating what set the current thread's query comment
"""
def __init__(self, initial):
def __init__(self, initial) -> None:
self.query_comment: Optional[str] = initial
self.append = False
self.append: bool = False
def add(self, sql: str) -> str:
if not self.query_comment:

View File

@@ -1,6 +1,6 @@
from collections.abc import Hashable
from dataclasses import dataclass, field
from typing import Optional, TypeVar, Any, Type, Dict, Iterator, Tuple, Set
from typing import Optional, TypeVar, Any, Type, Dict, Iterator, Tuple, Set, List
from dbt.contracts.graph.nodes import SourceDefinition, ManifestNode, ResultNode, ParsedNode
from dbt.contracts.relation import (
@@ -35,6 +35,10 @@ class BaseRelation(FakeAPIObject, Hashable):
include_policy: Policy = field(default_factory=lambda: Policy())
quote_policy: Policy = field(default_factory=lambda: Policy())
dbt_created: bool = False
# register relation types that can be renamed for the purpose of replacing relations using stages and backups
renameable_relations: List[str] = field(
default_factory=lambda: [RelationType.Table, RelationType.View]
)
def _is_exactish_match(self, field: ComponentName, value: str) -> bool:
if self.dbt_created and self.quote_policy.get_part(field) is False:
@@ -169,7 +173,6 @@ class BaseRelation(FakeAPIObject, Hashable):
return self.include(identifier=False).replace_path(identifier=None)
def _render_iterator(self) -> Iterator[Tuple[Optional[ComponentName], Optional[str]]]:
for key in ComponentName:
path_part: Optional[str] = None
if self.include_policy.get_part(key):
@@ -286,6 +289,10 @@ class BaseRelation(FakeAPIObject, Hashable):
)
return cls.from_dict(kwargs)
@property
def can_be_renamed(self):
return self.type in self.renameable_relations
def __repr__(self) -> str:
return "<{} {}>".format(self.__class__.__name__, self.render())

View File

@@ -19,7 +19,7 @@ Adapter = AdapterProtocol
class AdapterContainer:
def __init__(self):
def __init__(self) -> None:
self.lock = threading.Lock()
self.adapters: Dict[str, Adapter] = {}
self.plugins: Dict[str, AdapterPlugin] = {}

View File

@@ -1,6 +1,6 @@
import abc
import time
from typing import List, Optional, Tuple, Any, Iterable, Dict, Union
from typing import List, Optional, Tuple, Any, Iterable, Dict
import agate
@@ -131,14 +131,6 @@ class SQLConnectionManager(BaseConnectionManager):
return dbt.clients.agate_helper.table_from_data_flat(data, column_names)
@classmethod
def data_type_code_to_name(cls, type_code: Union[int, str]) -> str:
"""Get the string representation of the data type from the type_code."""
# https://peps.python.org/pep-0249/#type-objects
raise dbt.exceptions.NotImplementedError(
"`data_type_code_to_name` is not implemented for this adapter!"
)
def execute(
self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
) -> Tuple[AdapterResponse, agate.Table]:

View File

@@ -61,7 +61,6 @@ def args_to_context(args: List[str]) -> Context:
if len(args) == 1 and "," in args[0]:
args = args[0].split(",")
sub_command_name, sub_command, args = cli.resolve_command(cli_ctx, args)
# Handle source and docs group.
if isinstance(sub_command, Group):
sub_command_name, sub_command, args = sub_command.resolve_command(cli_ctx, args)
@@ -319,7 +318,6 @@ def command_params(command: CliCommand, args_dict: Dict[str, Any]) -> CommandPar
for k, v in args_dict.items():
k = k.lower()
# if a "which" value exists in the args dict, it should match the command provided
if k == WHICH_KEY:
if v != command.value:
@@ -344,7 +342,8 @@ def command_params(command: CliCommand, args_dict: Dict[str, Any]) -> CommandPar
if k == "macro" and command == CliCommand.RUN_OPERATION:
add_fn(v)
elif v in (None, False):
# None is a Singleton, False is a Flyweight, only one instance of each.
elif v is None or v is False:
add_fn(f"--no-{spinal_cased}")
elif v is True:
add_fn(f"--{spinal_cased}")

View File

@@ -141,6 +141,7 @@ class dbtRunner:
@p.macro_debugging
@p.partial_parse
@p.partial_parse_file_path
@p.partial_parse_file_diff
@p.populate_cache
@p.print
@p.printer_width
@@ -329,6 +330,7 @@ def docs_serve(ctx, **kwargs):
@p.state
@p.defer_state
@p.deprecated_state
@p.compile_inject_ephemeral_ctes
@p.target
@p.target_path
@p.threads

View File

@@ -40,6 +40,14 @@ compile_docs = click.option(
default=True,
)
compile_inject_ephemeral_ctes = click.option(
"--inject-ephemeral-ctes/--no-inject-ephemeral-ctes",
envvar=None,
help="Internal flag controlling injection of referenced ephemeral models' CTEs during `compile`.",
hidden=True,
default=True,
)
config_dir = click.option(
"--config-dir",
envvar=None,
@@ -257,6 +265,14 @@ partial_parse_file_path = click.option(
type=click.Path(exists=True, dir_okay=False, resolve_path=True),
)
partial_parse_file_diff = click.option(
"--partial-parse-file-diff/--no-partial-parse-file-diff",
envvar="DBT_PARTIAL_PARSE_FILE_DIFF",
help="Internal flag for whether to compute a file diff during partial parsing.",
hidden=True,
default=True,
)
populate_cache = click.option(
"--populate-cache/--no-populate-cache",
envvar="DBT_POPULATE_CACHE",
@@ -389,9 +405,9 @@ inline = click.option(
# Most CLI arguments should use the combined `select` option that aliases `--models` to `--select`.
# However, if you need to split out these separators (like `dbt ls`), use the `models` and `raw_select` options instead.
# See https://github.com/dbt-labs/dbt-core/pull/6774#issuecomment-1408476095 for more info.
models = click.option(*model_decls, **select_attrs)
raw_select = click.option(*select_decls, **select_attrs)
select = click.option(*select_decls, *model_decls, **select_attrs)
models = click.option(*model_decls, **select_attrs) # type: ignore[arg-type]
raw_select = click.option(*select_decls, **select_attrs) # type: ignore[arg-type]
select = click.option(*select_decls, *model_decls, **select_attrs) # type: ignore[arg-type]
selector = click.option(
"--selector",

View File

@@ -9,10 +9,23 @@ from typing import Iterable, List, Dict, Union, Optional, Any
from dbt.exceptions import DbtRuntimeError
BOM = BOM_UTF8.decode("utf-8") # '\ufeff'
class Integer(agate.data_types.DataType):
def cast(self, d):
# by default agate will cast none as a Number
# but we need to cast it as an Integer to preserve
# the type when merging and unioning tables
if type(d) == int or d is None:
return d
else:
raise agate.exceptions.CastError('Can not parse value "%s" as Integer.' % d)
def jsonify(self, d):
return d
class Number(agate.data_types.Number):
# undo the change in https://github.com/wireservice/agate/pull/733
# i.e. do not cast True and False to numeric 1 and 0
@@ -48,6 +61,7 @@ def build_type_tester(
) -> agate.TypeTester:
types = [
Integer(null_values=("null", "")),
Number(null_values=("null", "")),
agate.data_types.Date(null_values=("null", ""), date_format="%Y-%m-%d"),
agate.data_types.DateTime(null_values=("null", ""), datetime_format="%Y-%m-%d %H:%M:%S"),
@@ -135,12 +149,12 @@ def as_matrix(table):
return [r.values() for r in table.rows.values()]
def from_csv(abspath, text_columns):
def from_csv(abspath, text_columns, delimiter=","):
type_tester = build_type_tester(text_columns=text_columns)
with open(abspath, encoding="utf-8") as fp:
if fp.read(1) != BOM:
fp.seek(0)
return agate.Table.from_csv(fp, column_types=type_tester)
return agate.Table.from_csv(fp, column_types=type_tester, delimiter=delimiter)
class _NullMarker:
@@ -166,6 +180,13 @@ class ColumnTypeBuilder(Dict[str, NullableAgateType]):
elif isinstance(value, _NullMarker):
# use the existing value
return
# when one table column is Number while another is Integer, force the column to Number on merge
elif isinstance(value, Integer) and isinstance(existing_type, agate.data_types.Number):
# use the existing value
return
elif isinstance(existing_type, Integer) and isinstance(value, agate.data_types.Number):
# overwrite
super().__setitem__(key, value)
elif not isinstance(value, type(existing_type)):
# actual type mismatch!
raise DbtRuntimeError(
@@ -177,8 +198,9 @@ class ColumnTypeBuilder(Dict[str, NullableAgateType]):
result: Dict[str, agate.data_types.DataType] = {}
for key, value in self.items():
if isinstance(value, _NullMarker):
# this is what agate would do.
result[key] = agate.data_types.Number()
# agate would make it a Number but we'll make it Integer so that if this column
# gets merged with another Integer column, it won't get forced to a Number
result[key] = Integer()
else:
result[key] = value
return result

View File

@@ -191,7 +191,7 @@ NativeSandboxEnvironment.template_class = NativeSandboxTemplate # type: ignore
class TemplateCache:
def __init__(self):
def __init__(self) -> None:
self.file_cache: Dict[str, jinja2.Template] = {}
def get_node_template(self, node) -> jinja2.Template:

View File

@@ -320,6 +320,10 @@ class Compiler:
if model.compiled_code is None:
raise DbtRuntimeError("Cannot inject ctes into an uncompiled node", model)
# tech debt: safe flag/arg access (#6259)
if not getattr(self.config.args, "inject_ephemeral_ctes", True):
return (model, [])
# extra_ctes_injected flag says that we've already recursively injected the ctes
if model.extra_ctes_injected:
return (model, model.extra_ctes)

View File

@@ -426,8 +426,10 @@ class PartialProject(RenderComponents):
sources: Dict[str, Any]
tests: Dict[str, Any]
metrics: Dict[str, Any]
semantic_models: Dict[str, Any]
exposures: Dict[str, Any]
vars_value: VarProvider
dbt_cloud: Dict[str, Any]
dispatch = cfg.dispatch
models = cfg.models
@@ -436,6 +438,7 @@ class PartialProject(RenderComponents):
sources = cfg.sources
tests = cfg.tests
metrics = cfg.metrics
semantic_models = cfg.semantic_models
exposures = cfg.exposures
if cfg.vars is None:
vars_dict: Dict[str, Any] = {}
@@ -459,6 +462,8 @@ class PartialProject(RenderComponents):
manifest_selectors = SelectorDict.parse_from_selectors_list(
rendered.selectors_dict["selectors"]
)
dbt_cloud = cfg.dbt_cloud
project = Project(
project_name=name,
version=version,
@@ -492,12 +497,14 @@ class PartialProject(RenderComponents):
sources=sources,
tests=tests,
metrics=metrics,
semantic_models=semantic_models,
exposures=exposures,
vars=vars_value,
config_version=cfg.config_version,
unrendered=unrendered,
project_env_vars=project_env_vars,
restrict_access=cfg.restrict_access,
dbt_cloud=dbt_cloud,
)
# sanity check - this means an internal issue
project.validate()
@@ -598,6 +605,7 @@ class Project:
sources: Dict[str, Any]
tests: Dict[str, Any]
metrics: Dict[str, Any]
semantic_models: Dict[str, Any]
exposures: Dict[str, Any]
vars: VarProvider
dbt_version: List[VersionSpecifier]
@@ -609,6 +617,7 @@ class Project:
unrendered: RenderComponents
project_env_vars: Dict[str, Any]
restrict_access: bool
dbt_cloud: Dict[str, Any]
@property
def all_source_paths(self) -> List[str]:
@@ -673,11 +682,13 @@ class Project:
"sources": self.sources,
"tests": self.tests,
"metrics": self.metrics,
"semantic-models": self.semantic_models,
"exposures": self.exposures,
"vars": self.vars.to_dict(),
"require-dbt-version": [v.to_version_string() for v in self.dbt_version],
"config-version": self.config_version,
"restrict-access": self.restrict_access,
"dbt-cloud": self.dbt_cloud,
}
)
if self.query_comment:

View File

@@ -167,6 +167,7 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
sources=project.sources,
tests=project.tests,
metrics=project.metrics,
semantic_models=project.semantic_models,
exposures=project.exposures,
vars=project.vars,
config_version=project.config_version,
@@ -182,6 +183,7 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
args=args,
cli_vars=cli_vars,
dependencies=dependencies,
dbt_cloud=project.dbt_cloud,
)
# Called by 'load_projects' in this class
@@ -322,6 +324,7 @@ class RuntimeConfig(Project, Profile, AdapterRequiredConfig):
"sources": self._get_config_paths(self.sources),
"tests": self._get_config_paths(self.tests),
"metrics": self._get_config_paths(self.metrics),
"semantic_models": self._get_config_paths(self.semantic_models),
"exposures": self._get_config_paths(self.exposures),
}

View File

@@ -1,6 +1,8 @@
from __future__ import annotations
import json
import os
from typing import Any, Dict, NoReturn, Optional, Mapping, Iterable, Set, List
from typing import Any, Callable, Dict, NoReturn, Optional, Mapping, Iterable, Set, List
import threading
from dbt.flags import get_flags
@@ -86,33 +88,29 @@ def get_context_modules() -> Dict[str, Dict[str, Any]]:
class ContextMember:
def __init__(self, value, name=None):
def __init__(self, value: Any, name: Optional[str] = None) -> None:
self.name = name
self.inner = value
def key(self, default):
def key(self, default: str) -> str:
if self.name is None:
return default
return self.name
def contextmember(value):
if isinstance(value, str):
return lambda v: ContextMember(v, name=value)
return ContextMember(value)
def contextmember(value: Optional[str] = None) -> Callable:
return lambda v: ContextMember(v, name=value)
def contextproperty(value):
if isinstance(value, str):
return lambda v: ContextMember(property(v), name=value)
return ContextMember(property(value))
def contextproperty(value: Optional[str] = None) -> Callable:
return lambda v: ContextMember(property(v), name=value)
class ContextMeta(type):
def __new__(mcls, name, bases, dct):
context_members = {}
context_attrs = {}
new_dct = {}
def __new__(mcls, name, bases, dct: Dict[str, Any]) -> ContextMeta:
context_members: Dict[str, Any] = {}
context_attrs: Dict[str, Any] = {}
new_dct: Dict[str, Any] = {}
for base in bases:
context_members.update(getattr(base, "_context_members_", {}))
@@ -148,27 +146,28 @@ class Var:
return self._cli_vars
@property
def node_name(self):
def node_name(self) -> str:
if self._node is not None:
return self._node.name
else:
return "<Configuration>"
def get_missing_var(self, var_name):
raise RequiredVarNotFoundError(var_name, self._merged, self._node)
def get_missing_var(self, var_name: str) -> NoReturn:
# TODO function name implies a non exception resolution
raise RequiredVarNotFoundError(var_name, dict(self._merged), self._node)
def has_var(self, var_name: str):
def has_var(self, var_name: str) -> bool:
return var_name in self._merged
def get_rendered_var(self, var_name):
def get_rendered_var(self, var_name: str) -> Any:
raw = self._merged[var_name]
# if bool/int/float/etc are passed in, don't compile anything
if not isinstance(raw, str):
return raw
return get_rendered(raw, self._context)
return get_rendered(raw, dict(self._context))
def __call__(self, var_name, default=_VAR_NOTSET):
def __call__(self, var_name: str, default: Any = _VAR_NOTSET) -> Any:
if self.has_var(var_name):
return self.get_rendered_var(var_name)
elif default is not self._VAR_NOTSET:
@@ -178,13 +177,17 @@ class Var:
class BaseContext(metaclass=ContextMeta):
# subclass is TargetContext
def __init__(self, cli_vars):
self._ctx = {}
self.cli_vars = cli_vars
self.env_vars = {}
# Set by ContextMeta
_context_members_: Dict[str, Any]
_context_attrs_: Dict[str, Any]
def generate_builtins(self):
# subclass is TargetContext
def __init__(self, cli_vars: Dict[str, Any]) -> None:
self._ctx: Dict[str, Any] = {}
self.cli_vars: Dict[str, Any] = cli_vars
self.env_vars: Dict[str, Any] = {}
def generate_builtins(self) -> Dict[str, Any]:
builtins: Dict[str, Any] = {}
for key, value in self._context_members_.items():
if hasattr(value, "__get__"):
@@ -194,14 +197,14 @@ class BaseContext(metaclass=ContextMeta):
return builtins
# no dbtClassMixin so this is not an actual override
def to_dict(self):
def to_dict(self) -> Dict[str, Any]:
self._ctx["context"] = self._ctx
builtins = self.generate_builtins()
self._ctx["builtins"] = builtins
self._ctx.update(builtins)
return self._ctx
@contextproperty
@contextproperty()
def dbt_version(self) -> str:
"""The `dbt_version` variable returns the installed version of dbt that
is currently running. It can be used for debugging or auditing
@@ -221,7 +224,7 @@ class BaseContext(metaclass=ContextMeta):
"""
return dbt_version
@contextproperty
@contextproperty()
def var(self) -> Var:
"""Variables can be passed from your `dbt_project.yml` file into models
during compilation. These variables are useful for configuring packages
@@ -290,7 +293,7 @@ class BaseContext(metaclass=ContextMeta):
"""
return Var(self._ctx, self.cli_vars)
@contextmember
@contextmember()
def env_var(self, var: str, default: Optional[str] = None) -> str:
"""The env_var() function. Return the environment variable named 'var'.
If there is no such environment variable set, return the default.
@@ -318,7 +321,7 @@ class BaseContext(metaclass=ContextMeta):
if os.environ.get("DBT_MACRO_DEBUGGING"):
@contextmember
@contextmember()
@staticmethod
def debug():
"""Enter a debugger at this line in the compiled jinja code."""
@@ -357,7 +360,7 @@ class BaseContext(metaclass=ContextMeta):
"""
raise MacroReturn(data)
@contextmember
@contextmember()
@staticmethod
def fromjson(string: str, default: Any = None) -> Any:
"""The `fromjson` context method can be used to deserialize a json
@@ -378,7 +381,7 @@ class BaseContext(metaclass=ContextMeta):
except ValueError:
return default
@contextmember
@contextmember()
@staticmethod
def tojson(value: Any, default: Any = None, sort_keys: bool = False) -> Any:
"""The `tojson` context method can be used to serialize a Python
@@ -401,7 +404,7 @@ class BaseContext(metaclass=ContextMeta):
except ValueError:
return default
@contextmember
@contextmember()
@staticmethod
def fromyaml(value: str, default: Any = None) -> Any:
"""The fromyaml context method can be used to deserialize a yaml string
@@ -432,7 +435,7 @@ class BaseContext(metaclass=ContextMeta):
# safe_dump defaults to sort_keys=True, but we act like json.dumps (the
# opposite)
@contextmember
@contextmember()
@staticmethod
def toyaml(
value: Any, default: Optional[str] = None, sort_keys: bool = False
@@ -477,7 +480,7 @@ class BaseContext(metaclass=ContextMeta):
except TypeError:
return default
@contextmember
@contextmember()
@staticmethod
def set_strict(value: Iterable[Any]) -> Set[Any]:
"""The `set_strict` context method can be used to convert any iterable
@@ -519,7 +522,7 @@ class BaseContext(metaclass=ContextMeta):
except TypeError:
return default
@contextmember
@contextmember()
@staticmethod
def zip_strict(*args: Iterable[Any]) -> Iterable[Any]:
"""The `zip_strict` context method can be used to used to return
@@ -541,7 +544,7 @@ class BaseContext(metaclass=ContextMeta):
except TypeError as e:
raise ZipStrictWrongTypeError(e)
@contextmember
@contextmember()
@staticmethod
def log(msg: str, info: bool = False) -> str:
"""Logs a line to either the log file or stdout.
@@ -562,7 +565,7 @@ class BaseContext(metaclass=ContextMeta):
fire_event(JinjaLogDebug(msg=msg, node_info=get_node_info()))
return ""
@contextproperty
@contextproperty()
def run_started_at(self) -> Optional[datetime.datetime]:
"""`run_started_at` outputs the timestamp that this run started, e.g.
`2017-04-21 01:23:45.678`. The `run_started_at` variable is a Python
@@ -590,19 +593,19 @@ class BaseContext(metaclass=ContextMeta):
else:
return None
@contextproperty
@contextproperty()
def invocation_id(self) -> Optional[str]:
"""invocation_id outputs a UUID generated for this dbt run (useful for
auditing)
"""
return get_invocation_id()
@contextproperty
@contextproperty()
def thread_id(self) -> str:
"""thread_id outputs an ID for the current thread (useful for auditing)"""
return threading.current_thread().name
@contextproperty
@contextproperty()
def modules(self) -> Dict[str, Any]:
"""The `modules` variable in the Jinja context contains useful Python
modules for operating on data.
@@ -627,7 +630,7 @@ class BaseContext(metaclass=ContextMeta):
""" # noqa
return get_context_modules()
@contextproperty
@contextproperty()
def flags(self) -> Any:
"""The `flags` variable contains true/false values for flags provided
on the command line.
@@ -644,7 +647,7 @@ class BaseContext(metaclass=ContextMeta):
"""
return flags_module.get_flag_obj()
@contextmember
@contextmember()
@staticmethod
def print(msg: str) -> str:
"""Prints a line to stdout.
@@ -662,7 +665,7 @@ class BaseContext(metaclass=ContextMeta):
print(msg)
return ""
@contextmember
@contextmember()
@staticmethod
def diff_of_two_dicts(
dict_a: Dict[str, List[str]], dict_b: Dict[str, List[str]]
@@ -691,7 +694,7 @@ class BaseContext(metaclass=ContextMeta):
dict_diff.update({k: dict_a[k]})
return dict_diff
@contextmember
@contextmember()
@staticmethod
def local_md5(value: str) -> str:
"""Calculates an MD5 hash of the given string.

View File

@@ -19,7 +19,7 @@ class ConfiguredContext(TargetContext):
super().__init__(config.to_target_dict(), config.cli_vars)
self.config = config
@contextproperty
@contextproperty()
def project_name(self) -> str:
return self.config.project_name
@@ -80,11 +80,11 @@ class SchemaYamlContext(ConfiguredContext):
self._project_name = project_name
self.schema_yaml_vars = schema_yaml_vars
@contextproperty
@contextproperty()
def var(self) -> ConfiguredVar:
return ConfiguredVar(self._ctx, self.config, self._project_name)
@contextmember
@contextmember()
def env_var(self, var: str, default: Optional[str] = None) -> str:
return_value = None
if var.startswith(SECRET_ENV_PREFIX):
@@ -113,7 +113,7 @@ class MacroResolvingContext(ConfiguredContext):
def __init__(self, config):
super().__init__(config)
@contextproperty
@contextproperty()
def var(self) -> ConfiguredVar:
return ConfiguredVar(self._ctx, self.config, self.config.project_name)

View File

@@ -45,6 +45,8 @@ class UnrenderedConfig(ConfigSource):
model_configs = unrendered.get("tests")
elif resource_type == NodeType.Metric:
model_configs = unrendered.get("metrics")
elif resource_type == NodeType.SemanticModel:
model_configs = unrendered.get("semantic_models")
elif resource_type == NodeType.Exposure:
model_configs = unrendered.get("exposures")
else:
@@ -70,6 +72,8 @@ class RenderedConfig(ConfigSource):
model_configs = self.project.tests
elif resource_type == NodeType.Metric:
model_configs = self.project.metrics
elif resource_type == NodeType.SemanticModel:
model_configs = self.project.semantic_models
elif resource_type == NodeType.Exposure:
model_configs = self.project.exposures
else:
@@ -189,9 +193,21 @@ class ContextConfigGenerator(BaseContextConfigGenerator[C]):
def _update_from_config(self, result: C, partial: Dict[str, Any], validate: bool = False) -> C:
translated = self._active_project.credentials.translate_aliases(partial)
return result.update_from(
translated = self.translate_hook_names(translated)
updated = result.update_from(
translated, self._active_project.credentials.type, validate=validate
)
return updated
def translate_hook_names(self, project_dict):
# This is a kind of kludge because the fix for #6411 specifically allowed misspelling
# the hook field names in dbt_project.yml, which only ever worked because we didn't
# run validate on the dbt_project configs.
if "pre_hook" in project_dict:
project_dict["pre-hook"] = project_dict.pop("pre_hook")
if "post_hook" in project_dict:
project_dict["post-hook"] = project_dict.pop("post_hook")
return project_dict
def calculate_node_config_dict(
self,

View File

@@ -24,7 +24,7 @@ class DocsRuntimeContext(SchemaYamlContext):
self.node = node
self.manifest = manifest
@contextmember
@contextmember()
def doc(self, *args: str) -> str:
"""The `doc` function is used to reference docs blocks in schema.yml
files. It is analogous to the `ref` function. For more information,

View File

@@ -40,7 +40,7 @@ class MacroResolver:
self._build_internal_packages_namespace()
self._build_macros_by_name()
def _build_internal_packages_namespace(self):
def _build_internal_packages_namespace(self) -> None:
# Iterate in reverse-order and overwrite: the packages that are first
# in the list are the ones we want to "win".
self.internal_packages_namespace: MacroNamespace = {}
@@ -56,7 +56,7 @@ class MacroResolver:
# root package namespace
# non-internal packages (that aren't local or root)
# dbt internal packages
def _build_macros_by_name(self):
def _build_macros_by_name(self) -> None:
macros_by_name = {}
# all internal packages (already in the right order)
@@ -78,7 +78,7 @@ class MacroResolver:
self,
package_namespaces: Dict[str, MacroNamespace],
macro: Macro,
):
) -> None:
if macro.package_name in package_namespaces:
namespace = package_namespaces[macro.package_name]
else:
@@ -89,7 +89,7 @@ class MacroResolver:
raise DuplicateMacroNameError(macro, macro, macro.package_name)
package_namespaces[macro.package_name][macro.name] = macro
def add_macro(self, macro: Macro):
def add_macro(self, macro: Macro) -> None:
macro_name: str = macro.name
# internal macros (from plugins) will be processed separately from
@@ -103,11 +103,11 @@ class MacroResolver:
if macro.package_name == self.root_project_name:
self.root_package_macros[macro_name] = macro
def add_macros(self):
def add_macros(self) -> None:
for macro in self.macros.values():
self.add_macro(macro)
def get_macro(self, local_package, macro_name):
def get_macro(self, local_package, macro_name) -> Optional[Macro]:
local_package_macros = {}
# If the macro is explicitly prefixed with an internal namespace
# (e.g. 'dbt.some_macro'), look there first
@@ -125,7 +125,7 @@ class MacroResolver:
return self.macros_by_name[macro_name]
return None
def get_macro_id(self, local_package, macro_name):
def get_macro_id(self, local_package, macro_name) -> Optional[str]:
macro = self.get_macro(local_package, macro_name)
if macro is None:
return None

View File

@@ -67,7 +67,7 @@ class ManifestContext(ConfiguredContext):
dct.update(self.namespace)
return dct
@contextproperty
@contextproperty()
def context_macro_stack(self):
return self.macro_stack

View File

@@ -618,7 +618,7 @@ class RuntimeMetricResolver(BaseMetricResolver):
target_package=target_package,
)
return ResolvedMetricReference(target_metric, self.manifest, self.Relation)
return ResolvedMetricReference(target_metric, self.manifest)
# `var` implementations.
@@ -754,19 +754,19 @@ class ProviderContext(ManifestContext):
self.model,
)
@contextproperty
@contextproperty()
def dbt_metadata_envs(self) -> Dict[str, str]:
return get_metadata_vars()
@contextproperty
@contextproperty()
def invocation_args_dict(self):
return args_to_dict(self.config.args)
@contextproperty
@contextproperty()
def _sql_results(self) -> Dict[str, Optional[AttrDict]]:
return self.sql_results
@contextmember
@contextmember()
def load_result(self, name: str) -> Optional[AttrDict]:
if name in self.sql_results:
# handle the special case of "main" macro
@@ -787,7 +787,7 @@ class ProviderContext(ManifestContext):
# Handle trying to load a result that was never stored
return None
@contextmember
@contextmember()
def store_result(
self, name: str, response: Any, agate_table: Optional[agate.Table] = None
) -> str:
@@ -803,7 +803,7 @@ class ProviderContext(ManifestContext):
)
return ""
@contextmember
@contextmember()
def store_raw_result(
self,
name: str,
@@ -815,7 +815,7 @@ class ProviderContext(ManifestContext):
response = AdapterResponse(_message=message, code=code, rows_affected=rows_affected)
return self.store_result(name, response, agate_table)
@contextproperty
@contextproperty()
def validation(self):
def validate_any(*args) -> Callable[[T], None]:
def inner(value: T) -> None:
@@ -836,7 +836,7 @@ class ProviderContext(ManifestContext):
}
)
@contextmember
@contextmember()
def write(self, payload: str) -> str:
# macros/source defs aren't 'writeable'.
if isinstance(self.model, (Macro, SourceDefinition)):
@@ -845,11 +845,11 @@ class ProviderContext(ManifestContext):
self.model.write_node(self.config.project_root, self.model.build_path, payload)
return ""
@contextmember
@contextmember()
def render(self, string: str) -> str:
return get_rendered(string, self._ctx, self.model)
@contextmember
@contextmember()
def try_or_compiler_error(
self, message_if_exception: str, func: Callable, *args, **kwargs
) -> Any:
@@ -858,21 +858,22 @@ class ProviderContext(ManifestContext):
except Exception:
raise CompilationError(message_if_exception, self.model)
@contextmember
@contextmember()
def load_agate_table(self) -> agate.Table:
if not isinstance(self.model, SeedNode):
raise LoadAgateTableNotSeedError(self.model.resource_type, node=self.model)
assert self.model.root_path
path = os.path.join(self.model.root_path, self.model.original_file_path)
column_types = self.model.config.column_types
delimiter = self.model.config.delimiter
try:
table = agate_helper.from_csv(path, text_columns=column_types)
table = agate_helper.from_csv(path, text_columns=column_types, delimiter=delimiter)
except ValueError as e:
raise LoadAgateTableValueError(e, node=self.model)
table.original_abspath = os.path.abspath(path)
return table
@contextproperty
@contextproperty()
def ref(self) -> Callable:
"""The most important function in dbt is `ref()`; it's impossible to
build even moderately complex models without it. `ref()` is how you
@@ -913,11 +914,11 @@ class ProviderContext(ManifestContext):
"""
return self.provider.ref(self.db_wrapper, self.model, self.config, self.manifest)
@contextproperty
@contextproperty()
def source(self) -> Callable:
return self.provider.source(self.db_wrapper, self.model, self.config, self.manifest)
@contextproperty
@contextproperty()
def metric(self) -> Callable:
return self.provider.metric(self.db_wrapper, self.model, self.config, self.manifest)
@@ -978,7 +979,7 @@ class ProviderContext(ManifestContext):
""" # noqa
return self.provider.Config(self.model, self.context_config)
@contextproperty
@contextproperty()
def execute(self) -> bool:
"""`execute` is a Jinja variable that returns True when dbt is in
"execute" mode.
@@ -1039,7 +1040,7 @@ class ProviderContext(ManifestContext):
""" # noqa
return self.provider.execute
@contextproperty
@contextproperty()
def exceptions(self) -> Dict[str, Any]:
"""The exceptions namespace can be used to raise warnings and errors in
dbt userspace.
@@ -1077,15 +1078,15 @@ class ProviderContext(ManifestContext):
""" # noqa
return wrapped_exports(self.model)
@contextproperty
@contextproperty()
def database(self) -> str:
return self.config.credentials.database
@contextproperty
@contextproperty()
def schema(self) -> str:
return self.config.credentials.schema
@contextproperty
@contextproperty()
def var(self) -> ModelConfiguredVar:
return self.provider.Var(
context=self._ctx,
@@ -1102,22 +1103,22 @@ class ProviderContext(ManifestContext):
"""
return self.db_wrapper
@contextproperty
@contextproperty()
def api(self) -> Dict[str, Any]:
return {
"Relation": self.db_wrapper.Relation,
"Column": self.adapter.Column,
}
@contextproperty
@contextproperty()
def column(self) -> Type[Column]:
return self.adapter.Column
@contextproperty
@contextproperty()
def env(self) -> Dict[str, Any]:
return self.target
@contextproperty
@contextproperty()
def graph(self) -> Dict[str, Any]:
"""The `graph` context variable contains information about the nodes in
your dbt project. Models, sources, tests, and snapshots are all
@@ -1233,23 +1234,23 @@ class ProviderContext(ManifestContext):
ret["compiled_sql"] = ret["compiled_code"]
return ret
@contextproperty
@contextproperty()
def pre_hooks(self) -> Optional[List[Dict[str, Any]]]:
return None
@contextproperty
@contextproperty()
def post_hooks(self) -> Optional[List[Dict[str, Any]]]:
return None
@contextproperty
@contextproperty()
def sql(self) -> Optional[str]:
return None
@contextproperty
@contextproperty()
def sql_now(self) -> str:
return self.adapter.date_function()
@contextmember
@contextmember()
def adapter_macro(self, name: str, *args, **kwargs):
"""This was deprecated in v0.18 in favor of adapter.dispatch"""
msg = (
@@ -1261,7 +1262,7 @@ class ProviderContext(ManifestContext):
)
raise CompilationError(msg)
@contextmember
@contextmember()
def env_var(self, var: str, default: Optional[str] = None) -> str:
"""The env_var() function. Return the environment variable named 'var'.
If there is no such environment variable set, return the default.
@@ -1305,7 +1306,7 @@ class ProviderContext(ManifestContext):
else:
raise EnvVarMissingError(var)
@contextproperty
@contextproperty()
def selected_resources(self) -> List[str]:
"""The `selected_resources` variable contains a list of the resources
selected based on the parameters provided to the dbt command.
@@ -1314,7 +1315,7 @@ class ProviderContext(ManifestContext):
"""
return selected_resources.SELECTED_RESOURCES
@contextmember
@contextmember()
def submit_python_job(self, parsed_model: Dict, compiled_code: str) -> AdapterResponse:
# Check macro_stack and that the unique id is for a materialization macro
if not (
@@ -1357,7 +1358,7 @@ class MacroContext(ProviderContext):
class ModelContext(ProviderContext):
model: ManifestNode
@contextproperty
@contextproperty()
def pre_hooks(self) -> List[Dict[str, Any]]:
if self.model.resource_type in [NodeType.Source, NodeType.Test]:
return []
@@ -1366,7 +1367,7 @@ class ModelContext(ProviderContext):
h.to_dict(omit_none=True) for h in self.model.config.pre_hook # type: ignore[union-attr] # noqa
]
@contextproperty
@contextproperty()
def post_hooks(self) -> List[Dict[str, Any]]:
if self.model.resource_type in [NodeType.Source, NodeType.Test]:
return []
@@ -1375,7 +1376,7 @@ class ModelContext(ProviderContext):
h.to_dict(omit_none=True) for h in self.model.config.post_hook # type: ignore[union-attr] # noqa
]
@contextproperty
@contextproperty()
def sql(self) -> Optional[str]:
# only doing this in sql model for backward compatible
if self.model.language == ModelLanguage.sql: # type: ignore[union-attr]
@@ -1392,7 +1393,7 @@ class ModelContext(ProviderContext):
else:
return None
@contextproperty
@contextproperty()
def compiled_code(self) -> Optional[str]:
if getattr(self.model, "defer_relation", None):
# TODO https://github.com/dbt-labs/dbt-core/issues/7976
@@ -1403,15 +1404,15 @@ class ModelContext(ProviderContext):
else:
return None
@contextproperty
@contextproperty()
def database(self) -> str:
return getattr(self.model, "database", self.config.credentials.database)
@contextproperty
@contextproperty()
def schema(self) -> str:
return getattr(self.model, "schema", self.config.credentials.schema)
@contextproperty
@contextproperty()
def this(self) -> Optional[RelationProxy]:
"""`this` makes available schema information about the currently
executing model. It's is useful in any context in which you need to
@@ -1446,7 +1447,7 @@ class ModelContext(ProviderContext):
return None
return self.db_wrapper.Relation.create_from(self.config, self.model)
@contextproperty
@contextproperty()
def defer_relation(self) -> Optional[RelationProxy]:
"""
For commands which add information about this node's corresponding
@@ -1660,7 +1661,7 @@ class TestContext(ProviderContext):
)
self.namespace = macro_namespace
@contextmember
@contextmember()
def env_var(self, var: str, default: Optional[str] = None) -> str:
return_value = None
if var.startswith(SECRET_ENV_PREFIX):

View File

@@ -14,7 +14,7 @@ class SecretContext(BaseContext):
"""This context is used in profiles.yml + packages.yml. It can render secret
env vars that aren't usable elsewhere"""
@contextmember
@contextmember()
def env_var(self, var: str, default: Optional[str] = None) -> str:
"""The env_var() function. Return the environment variable named 'var'.
If there is no such environment variable set, return the default.

View File

@@ -9,7 +9,7 @@ class TargetContext(BaseContext):
super().__init__(cli_vars=cli_vars)
self.target_dict = target_dict
@contextproperty
@contextproperty()
def target(self) -> Dict[str, Any]:
"""`target` contains information about your connection to the warehouse
(specified in profiles.yml). Some configs are shared between all

Some files were not shown because too many files have changed in this diff Show More