* Include null checks in utils test base
* Add tests for the schema test
* Add tests for this macro
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Honor `--skip-profile-setup` parameter when inside an existing project
* Use project name as the profile name
* Use separate file connections for reading and writing
* Raise a custom exception when no adapters are installed
* Test skipping interactive profile setup when inside a dbt project
* Replace `assert_not_called()` since it does not work
* Verbose CLI argument for skipping profile setup
* Use separate file connections for reading and writing
* Check empty list in a Pythonic manner
* CT-2461: Work toward model deprecation
* CT-2461: Remove unneeded conversions
* CT-2461: Fix up unit tests for new fields, correct a couple oversights
* CT-2461: Remaining implementation and tests for model/ref deprecation warnings
* CT-2461: Changelog entry for deprecation warnings
* CT-2461: Refine datetime handling and tests
* CT-2461: Fix up unit test data
* CT-2461: Fix some more unit test data.
* CT-2461: Fix merge issues
* CT-2461: Code review items.
* CT-2461: Improve version -> str conversion
* Allow missing `profiles.yml` for `dbt deps` and `dbt init`
* Some commands allow the `--profiles-dir` to not exist
* Remove fix to verify that CI tests work
* Allow missing `profiles.yml` for `dbt deps` and `dbt init`
* CI is not finding any installed adapters
* Remove functional test for `dbt init`
* Adding perf testing GHA
* Fixing tigger syntax
* Fixing PR creation issue
* Updating testing var
* Remove unneeded branch names
* Fixing branch naming convention
* Standardizing branch name to var
* Consolidating PR jobs
* Updating inputs and making more readable
* Splitting steps up
* Making some updates here to simplify and update
* Remove tab
* Cleaned up testing TODOs before committing
* Fixing spacing
* Fixing spacing issue
* Create publication.py, various Publication classes, Dependency class
* Load dependencies.yml and the corresponding publication file
* Add "public_nodes" and populate ref_lookup
* resolve_ref working
* Add public nodes to parent and child maps
* Bump manifest version and fix tests, use ModelDependsOn
* Split out PublicationArtifact and PublicationConfig, store public_models
separately
* Store dependencies in publication artifact
* change detection of PublicModel for >= python3.10
* Handle removing references for re-processing if publication has changed
* Handle only changed publication artifacts
* Add some logging events
* Remove duplicate nodes from manifest
* refactor relation_from_relation_name
* Remove duplicate writing of manifest.json
* Add public_nodes to flat_graph
* Move some file name constants to core/dbt/constants.py
* Remove "environment" from ProjectDependency. Add
database/schema/identifier to PublicModel. Update TargetNotFound
exception.
* Include external publication dependencies in publication artifact dependencies
* Remove create_from_relation_name, call create_from_node instead
* Change PublicationArtifactChanged message to debug level
* Make write_publication_artifact a function in parser/manifest.py
* Create fixture to create minimal alternate project (just models)
* develop multi project test case
* Latest version should use un-suffixed alias
* Latest version can be in un-suffixed file
* FYI when unpinned ref to model with prerelease version
* [WIP] Nicer error if versioned ref to unversioned model
* Revert "Latest version should use un-suffixed alias"
This reverts commit 3616c52c1eed7588b9e210e1c957dfda598be550.
* Revert "[WIP] Nicer error if versioned ref to unversioned model"
This reverts commit c9ae4af1cfbd6b7bfc5dcbb445556233eb4bd2c0.
* Define real event for UnpinnedRefNewVersionAvailable
* Update pp test for implicit unsuffixed defined_in
* Add changelog entry
* Fix unit test
* marky feedback
* Add test case for UnpinnedRefNewVersionAvailable event
* Adding a new column is not a breaking contract change
* Add changelog entry
* More structured exception
* same_contract: False if non-breaking changes
* PR feedback: rm build_contract_checksum, more comments
* CT-2317: Reset invocation id in preflight for each dbt command.
* CT-2317: Add unit test for invocation_id behavior.
* CT-2317: Add changelog entry.
* CT-2317: Modify freshness test to ignore invocation_id
* CT-2317: Assign invocation_id before tracking initialization.
* CT-2317: Fix unit test failures and a bunch of other stuff
* CT-2317: Remove checks which make outdated assumptions about invocation_id being stable between runs
* CT-2317: Review tweak, more unit test fixes.
* Removed options for `dbt parse`
* Fix misspellings
* Capitalize JSON when appropriate
* Update help text for --write-json/--no-write-json
* Update help text for --config-dir
* Update help text for --resource-types
* Removed decorators for removed dbt parse options
* Remove `--write-manifest` flag from `parse`
* Remove `--parse-only` flag from `compile`
* Update help text for `dbt list --output`
* Standardize on one line per argument
* Factor 3 from 12 Factor CLI Apps
* Update help text for `dbt --version`
* Standardize capitalization of resource types for `dbt build`
* `debug --config-dir` is a boolean flag
* Update help text for `--version-check`
* Specify `-q` as a conventional alias for `--quiet`
* Update help text for `debug --config-dir`
* Update help text for `debug`
* Treat more dense text blobs as binary for `git grep`
* Update help text for `--version-check`
* Update help text for `--defer`
* Update help text for `--indirect-selection`
* Co-locate log colorization with other log settings
* Update help text for `--log-format*`, `--log-level*`, and `--use-colors*`
* Temporarily re-add option for CI tests
* Remove `--parse-only` flag from `show`
* Remove `--write-manifest` flag from `parse` (again)
* Snapshot strategies: newline for subquery
* add changie output
* add test for snapshot ending in comment
* remove import to be flake8 compliant
* add seed import
* add newlines for flake8 compliance
* typo fix
* Fixing up a test, adding a comment or two
* removed un-needed test fixtures
* removed even more un-needed fixtures, collapsed test to single class
* removed errant breakpoint()
* Fix a little typo
---------
Co-authored-by: Ian Knox <ian.knox@dbtlabs.com>
Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com>
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: (Almost) complete support for model level constraints
* CT-1922: Fix typo affecting correct model constraint parsing.
* CT-1922: Rework base class for model tests for greater simplicity
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: Revise unit tests for new model-level constraints property
* CT-1922: (Almost) complete support for model level constraints
* first pass
* implement in core
* add proto
* WIP
* resolve errors in columns_spec_ddl
* changelog
* update comment
* move logic over to python
* rename and use enum
* update default constraint_support dict
* generate new proto definition after conflicts
* reorganize code and break warnings into each constraint
* fix postgres constraint support
* remove breakpoint
* convert constraint support to constant
* update postgres
* add to export
* add to export
* regen proto types file
* standardize names
* put back mypy error
* more naming + add back comma
* add constraint support to model level constraints
* update event message and method signature
* rename method
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: Revise unit tests for new model-level constraints property
* CT-1922: (Almost) complete support for model level constraints
* CT-1922: Fix typo affecting correct model constraint parsing.
* CT-1922: Improve whitespace handling
* CT-1922: Render raw constraints to constraint list directly
* make method return consistent
* regenerate proto defn
* update evvent test
* add some code cleanup
---------
Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
* CT-1922: Rough in functionality for parsing model level constraints
* CT-1922: Revise unit tests for new model-level constraints property
* CT-1922: (Almost) complete support for model level constraints
* CT-1922: Fix typo affecting correct model constraint parsing.
* CT-1922: Minor code review refinements
* CT-1922: Improve whitespace handling
* CT-1922: Render raw constraints to constraint list directly
* CT-1922: Rework base class for model tests for greater simplicity
* CT-1922: Remove debugging properties. Oops.
* CT-1922: Fix type annotation
* improved first line of error
* added basic printing of yaml and sql cols as columns
* added changie log
* used listed dictionary as input to match columns
* swapped order of col headers for printing
* used listed dictionary as input to match columns
* removed merge conflict text from file
* Touch-ups
* Update log introspection in functional tests
* Update format_column macro. Case insensitive test
* PR feedback: just data_type, not formatted
---------
Co-authored-by: Kyle Kent <kyle.kent321@gmail.com>
* remove trial nodes before building subdag
* add changie
* Update graph.py
remove comment
* further optimize by sorting node search by degree
* change degree to product of in and out degree
* Add tests for logging jinja2.Undefined objects
[CT-2259](https://github.com/dbt-labs/dbt-core/issues/7108) identifies
an issue wherein dbt-core 1.0-1.3 raise errors if a jinja2.Undefined
object is attempted to be logged. This generally happened in the form
of `{{ log(undefined_variable, info=True) }}`. This commit adding this
test exists for two reasons
1. Ensure we don't have a regression in this going forward
2. Exist as a commit to be used for backport fixes for dbt-core 1.0-1.3
* Add tests for checking `DBT_ENV_SECRET_`s don't break logging
[CT-1783](https://github.com/dbt-labs/dbt-core/issues/6568) describes
a bug in dbt-core 1.0-1.3 wherein when a `DBT_ENV_SECRET_` all
`{{ log("logging stuff", info=True) }}` invocations break. This commit
adds a test for this for two reasons:
1. Ensure we don't regress to this behavior going forward
2. Act as a base commit for making the backport fixes to dbt-core 1.0-1.3
* Add tests ensuring failed event serialization is handled correctly
[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113) states
that failed serialization should result in an exception handling path
which will fire another event instead of raising an exception. This is
hard to test perfectly because the exception handling path for
serialization depending on whether pytest is present. If pytest isn't
present, a new event documentation the failed serialization is fired.
If pytest is present, the failed serialization gets raised as an exception.
Thus this added test ensures that the expected exception is raised and
assumes that the correct event will be fired normally.
* Log warning when event serialization fails in `msg_to_dict`
This commit updates the `msg_to_dict` exception handling path to
fire a warning level event instead of raising an exception.
Truthfully, we're not sure if this exception handling path is even
possible to hit. That's because we recently switched from betterproto
to google's protobuf. However, exception path is the subject of
[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113). Though we
don't think it's actually possible to hit it anymore, we still want
to handle the case if it is.
* Update serialization failure note to be a warn level event in `BaseEvent`
[CT-2264](https://github.com/dbt-labs/dbt-core/issues/7113) wants
logging messages about event serialization failure to be `WARNING`
level events. This does that.
* Add changie info for changes
* Add test to check exception handling of `msg_to_dict`
* One argument per line
* Tests for multiple `--select` or `--exclude`
* Allow `--select` and `--exclude` multiple times
* Changelog entry
* MultiOption options must be specified with type=tuple or type=ChoiceTuple
* Testing for `--output-keys` and `--resource-type`
* Validate that any new param with `MultiOption` should also have `type=tuple` (or `ChoiceTuple`) and `multiple=True`
* first pass
* adding tests
* changelog
* split up tests due to order importance
* update test
* add back comment
* rename base test classes
* move sql
* fix test name
* move sql
* test changes to match main
* organize and cleanup fixtures
* more cleanup of tests
* add utility function to EventManager for explicitly adding callbacks
Technically these aren't necessary in their current state. We could instead
have people do `<InstantiatedEventManager>.callbacks.extend(...)` directly.
However, it's not hard to imagine a world wherein extra things need to take
place when a callback is added. Thus abstracting to a utility method
now means that as the implementation of how callbacks are actually added
changes, the invocation to do so can stay the same.
* update `setup_event_logger` to optionally take in callbacks add them to the EventManager
* update preflight decorator to check for and pass along callbacks for event logger setup
* Add `callbacks` to `dbtRunner`
On instantiation of `dbtRunner` one can now provide `callbacks`. These
callbacks are for the `EventLogger`. When `invoke` is called on a `dbtRunner`,
the `callbacks` are added to the cli context object. In the preflight
decorator these callbacks are extracted from the cli context and then
passed to the `setup_event_logger`, finally `setup_event_logger` ensures
the callbacks are added to the global `EVENT_MANAGER`.
* add test to check dbtRunner callbacks get properly set
I believe technically this tests qualifies as more of an integration
test, but no other tests like it currently exist (that I could find
via a cursory search). The `tests/unit/test_dbt_runner.py` seemed like
the most intuitive spot. However, if somewhere else makes sense, I'd be
happy to move it.
* add changie documentation for CT-1928
* Convert simple copy.
* Adjust class names for import.
* adjust test namespacing
* Resolve test error.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* ct-2198: clean up some type names and uses
* CT-2198: Unify constraints and constraints_check properties on columns
* Make mypy version consistently 0.981 (#7134)
* CT 1808 diff based partial parsing (#6873)
* model contracts on models materialized as views (#7120)
* first pass
* rename tests
* fix failing test
* changelog
* fix functional test
* Update core/dbt/parser/base.py
* Update core/dbt/parser/schemas.py
* Create method for env var deprecation (#7086)
* update to allow adapters to change model name resolution in py models (#7115)
* update to allow adapters to change model name resolution in py models
* add changie
* fix newline adds
* move quoting into macro
* use single quotes
* add env DBT_PROJECT_DIR support #6078 (#6659)
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
* Add new index.html and changelog yaml files from dbt-docs (#7141)
* Make version configs optional (#7060)
* [CT-1584] New top level commands: interactive compile (#7008)
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* CT-2198: Add changelog entry
* CT-2198: Fix tests which broke after merge
* CT-2198: Add explicit validation of constraint types w/ unit test
* CT-2198: Move access property, per code review
* CT-2198: Remove a redundant macro
* CT-1298: Rework constraints to be adapter-generated in Python code
* CT-2198: Clarify function name per review
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Stu Kilgore <stu.kilgore@dbtlabs.com>
Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
Co-authored-by: Leo Schick <67712864+leo-schick@users.noreply.github.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: FishtownBuildBot <77737458+FishtownBuildBot@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* add protobuf message/class for new CommandCompleted event
For [CT-2049](https://github.com/dbt-labs/dbt-core/issues/6878) we
concluded that we wanted a new event type, [CommandCompleted](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1419718606)
with [four (4) values](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1426118283):
which command was run, whether the command succeeded, the timestamp
that the command finished, and how long the command took. This commit
adds the new event proto defition, the auto generated proto_types, and
the instantiatable even type.
* begin emitting CommandCompleted event in the preflight decorator
The [preflight decorator](4186f99b74/core/dbt/cli/requires.py (L19))
runs at the start of every CLI invocation. Thus is a perfect candidate
for emitting the CommandCompleted event. This is noted in the [dicussion
on CT-2049](https://github.com/dbt-labs/dbt-core/issues/6878#issuecomment-1428643539).
* add CommandCompleted event to event unit tests
* Add: changelog entry
* fire CommandCompleted event reguardless of upstream exceptions
Previously, if `--fail-fast` was specified and an issue was run into
or an unhandled issue became an exception, the CommandCompleted event
would not get fired because at this point in the stack we'd be in
exception thrown handling mode. If an exception does reach this point,
we want to still fire the event and also continue to propogate the
exception. Hence the bare `raise` exists to reraise the caught exception
* Update CommandCompleted event to be a `Debug` level event
We don't actually "always" need this event to be logged. Thus we've
updated it to `Debug` level. [Discussion Context](https://github.com/dbt-labs/dbt-core/pull/7180#discussion_r1139281963)
* Init roadmap
* Rework the top paragraph
* Clean-up the whole thing
* Typos and stuff
* Add a missing word
* Fix typo
* Update "when" note
* Next draft
* Propose rename
* Resolve TODOs, still needs a reread
* Being cute
* Another read through
* Fix sentence fragment
---------
Co-authored-by: Florian Eiden <florian.eiden@dbtlabs.com>
* first pass
* WIP
* add notes/stubs on more pieces
* more work
* more cleanup
* cleanup
* add more cleanup and generalization
* update to use reusable workflow
* add TODO
* Add back initialization events
* Fix log_cache_events. Default stdout logger knows less than it used to
* Add back exception handling events
* Revert "Add back exception handling events"
This reverts commit 26f22d91b660f51f0df6a59f9e5cae16b0ee6fe5.
* Add changelog entry
* Fix test by stringifying dict values
* Add generated CLI API docs
---------
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* part 1 of env var for core team
* add logic to use env vars to generate changelog
* modify version bump to add members via env var
* pull in main and tweak
* add token
* changes for testing
* split step
* remove leading slash
* add version check
* more debugging
* try curl
* try more things
* try more things
* chnage auth
* put back token
* update permissions
* add back fishtown pat
* use new pat
* fix typo
* swap token
* comment out list teams
* change url
* debug path
* add continue
* change core case
* more tweaks
* send output to file
* add file view
* make array
* tweak
* remove []
* add quotes
* add tojson
* add quotes to set
* tweak
* fix id
* tweaks
* more
* more
* remove new lines
* more tweaks
* update to generate changelog
* remove debugging bits
* use central version-bump
* use correct author list
* testing with changelog team automation
* add new token to input
* move secret
* remove testing aspects from workflow
* clean up team logic
* explicitly send secret
* move bumpversion comment
* move comments
* point workflow back tp main
* point to branch for testing
* point back to main
* inherit secrets
* first pass at automating latest branches
* checkout repo first
* fetch all history
* reorg
* debugging
* update test id
* swap lines
* incorporate new branch aciton
* tweak vars
* Formatting
* Changelog entry
* Rename to BaseSimpleSeedColumnOverride
* Better error handling
* Update test to include the BOM test
* Cleanup and formating
* Unused import remove
* nit line
* Pr comments
* update regex to match all iterations
* convert to num to match all adapters
* add comments, remove extra .
* clarify with more comments
* Update .bumpversion.cfg
Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
---------
Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
* Add clearer directions for custom test suite vars in Makefile.
* Fix up PR for review
* Fix erroneous whitespace.
* Fix a spelling error.
* Add documentation to discourage makefile edits but provide override tooling.
* Fix quotation marks. Very strange behavior
* Compact code and verify quotations happy inside bash and python.
* Fold comments into Makefile.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Convert test and make it a bit more pytest-onic
* Ax old integration test.
* Run black on test conversion
* I didn't like how pytest was running the fixture so wrapped it into a closure.
* Merge converted test into persist docs.
* Move persist docs tests to the adapter zone. Prep for adapter tests.
* Fix up test names
* Fix name to be less confusing.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Test converted and reformatted for pytest.
* Ax old versions of 052 test
* Nix the 'os' import and black format
* Change names of models to be more PEP like
* cleanup code
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Convert test and make it a bit more pytest-onic
* Ax old integration test.
* Run black on test conversion
* I didn't like how pytest was running the fixture so wrapped it into a closure.
* Merge converted test into persist docs.
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* init commit for column_types test conversion
* init start of test_column_types.py
* pass tes macros into both tests
* remove alt tests, remove old tests, push up working conversion
* rename base class, move to adapter zone so adapters can use
* typo fix
* Code cleanup and adding stderr to capture dbt
* Debug with --log-format json now prints structured logs.
* Add changelog.
* Move logs into miscellaneous and add values to test.
* nix whitespace and fix log levels
* List will now do structured logging when log format set to json.
* Add a quick None check.
* Add a get guard to class check.
* Better null checking
* The boolean doesn't reflect the original logic but a try-catch does.
* Address some code review comments and get us working again.
* Simplify logic now that we have a namespace object for self.config.args.
* Simplify logic for json log format checking.
* Simplify code for allowing our GraphTest cases to pass while also hiding compile stats from dbt ls/list .
* Simplify structured logging types.
* Fix up boolean logic and simplify via De'Morgan.
* Nix unneeded fixture.
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* CT-1786: Port docs tets to pytest
* Add generated CLI API docs
* CT-1786: Comply with the new style requirements
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* add defer_to_manifest in before_run to fix faulty deferred docs generate
* add a changelog
* add declaration of defer_to_manifest to FreshnessTask and GraphRunnableTask
* fix: add defer_to_manifest method to ListTask
* Re-factor list of YAML keys for hooks to late-render
* Add `pre_` and `post_hook` to list of late-rendered hooks
* Check for non-empty set intersection
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
* Test functional synonymy of `*_hook` with `*-hook`
Test that `pre_hook`/`post_hook` are functionally synonymous with `pre-hook`/`post-hook` for model project config
* Undo bugfix to validate the new test fails
* Revert "Undo bugfix to validate the new test fails"
This reverts commit e83a2be2eb.
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
* add meta attribute to nodeinfo for events
* also add meta to dataclass
* add to unit test to ensure meta is added
* adding functional test to check that meta is passed to nodeinfo during logging
* changelog
* remove used imported
* add tests with non-string keys
* renaming test dict keys
* add non-string value
* resolve failing test
* test additional non-string values
* fix flake8
* Stringify meta dict in node_info
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
* convert the test and fix an error due to a dead code seed
* Get rid of old test
* Remove unfortunately added files. Don't use that *
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Update types.proto
* pre-commit passes
* Cleanup tests and tweak EventLevels
* Put node_info back on SQLCommit. Add "level" to fire_event function.
* use event.message() in warn_or_error
* Fix logging test
* Changie
* Fix a couple of unit tests
* import Protocol from typing_extensions for 3.7
* ✨ adding pre-commit install to make dev
* 🎨 updating format of Makefile and CONTRIBUTING.md
* 📝 adding changelog via changie new
* ✨ adding dev_req to Makefile + docs
* 🎨 remove dev_req from docs, dry makefile
* Align names of `.PHONY` targets with their associated rules
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* starting to move jinja exceptions
* convert some exceptions
* add back old functions for backward compatibility
* organize
* more conversions
* more conversions
* add changelog
* split out CacheInconsistency
* more conversions
* convert even more
* convert parsingexceptions
* fix tests
* more conversions
* more conversions
* finish converting exception functions
* convert more tests
* standardize to msg
* remove some TODOs
* fix test param and check the rest
* add comment, move exceptions
* add types
* fix type errors
* fix type for adapter_response
* remove 0.13 version from message
* pass predicated to merge strategy
* postgres delete and insert
* merge with predicates
* update to use arbitrary list of predicates, not dictionaries, merge and delete
* changie
* add functional test to adapter zone
* comma in test config
* add test for incremental predicates delete and insert postgres
* update test structure for inheritance
* handle predicates config for backwards compatibility
* test for predicates keyword
* Add generated CLI API docs
Co-authored-by: Colin <colin.rogers@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* Remove unneeded SQL compilation attributes from SeedNode
* Fix various places that referenced removed attributes
* Cleanup a few Unions
* More formatting in nodes.py
* Mypy passing. Untested.
* Unit tests working
* use "doc" in documentation unique_ids
* update some doc_ids
* Fix some artifact tests. Still need previous version.
* Update manifest/v8.json
* Move relation_names to parsing
* Fix a couple of tests
* Update some artifacts. snapshot_seed has wrong schema.
* Changie
* Tweak NodeType.Documentation
* Put store_failures property in the right place
* Fix setting relation_name
* update changie to require issue or pr, and allow multiple
* remove extraneous data from changelog files.
* allow for multiple PR/issues to be entered
* update contributing guide
* remove issue number from bot changelogs
* update format of PR
* fix dependency changelogs
* remove extra line
* remove extra lines, tweak contributor wording
* Update CONTRIBUTING.md
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Get running with Python 3.11
* More tests passing, mypy still unhappy
* Upgrade to 3.11, and bump mashumaro
* patch importlib.import_module last
* lambda: Policy() default_factory on include and quote policy
* Add changelog entry
* Put a lambda on it
* Fix text formatting for log file
* Handle variant type return from e.log_level()
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Josh Taylor <joshuataylorx@gmail.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
* feat: add a list of default values to the ctx manager
* tests: dbt.get.config default values
* feat: validate the num of args in config.get
* feat: jinja template for dbt.config.get default values
* docs: changie yaml
* fix:typo on error message
Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
* v0 - new dbt deps type: tarball url
in support of
https://github.com/dbt-labs/dbt-core/issues/4205
* flake8 fixes
* adding max size tarball condition
* clean up imports
* typing
* adding sha1 and subdirectory options; improve logging feedback
sha1: allow user to specify sha1 in packages.yaml, will only install if package matches
subdirectory: allow user to specify subdirectory of package in tarfile, if the package is a non standard structure (like with git subdirectory option)
* simple tests added
* flake fixes
* changes to support tests; adding exceptions; fire_event logging
* new logging events
* tarball exceptions added
* build out tests
* removing in memory tarball test
* update type codes to M - Misc
* adding new events to test_events
* fix spacing for flake
* add retry download code - as used in registry calls
* clean
* remove saving tar in memory inside tarfile object
will hit url multiple times instead
* remove duplicative code after refactor
* black updates
* black formatting
* black formatting
* refactor - no more in-memory tarfile - all as file operations now
- remove tarfile passing, always use tempfile instead
- reorganize system.* functions, removing duplicative code
- more notes on current flow and structure - esp need for pattern of 1) unpack 2) scan for package dir 3) copy to destination.
- cleaning
* cleaning and sync to new tarball code
* cleaning and sync to new tarball code
* requested changes from PR
https://github.com/dbt-labs/dbt-core/pull/4689#discussion_r812970847
* reversions from revision 2
removing sha1 check to simplify/mirror hub install pattern
* simplify/mirror hub install pattern
to simplify/mirror hub install pattern
- removing sha1 check
- supply name/version to act as our 'metadata' source
* simplify/mirror hub install pattern
simplify with goal of mirroring hub install pattern
- supporting subfolders like git packages, and sha1 checks are removed
- existing code from RegistryPinnedPackage (install() and download_and_untar()) performs the operations
- RegistryPinnedPackage install() and download_and_untar() are not currently set up as functions that can be used across classes - this should be moved to dbt.deps.base, or to a dbt.deps.common file - need dbt labs feedback on how to proceed (or leave as is)
* remove revisions, no longer doing package check
* slim down to basic tests
more complex features have been removed (sha1, subfolder) so testing is much simpler!
* fix naming to match hubs behavior
remove version from package folder name
* refactor install and download to upstream PinnedPackage class
i'm on the fence if this is right approach, but seems like most sensible after some thought
* Create Features-20221107-105018.yaml
* fix flake, black, mypy errors
* additional flake/black fixes
* Update .changes/unreleased/Features-20221107-105018.yaml
fix username on changelog
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* change to fstring
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* cleaning - remove comment
* remove comment/question for dbt team
* in support of issuecomment 1334055944
https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334055944
* in support of issuecomment 1334118433
https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334118433
* black fixes; remove debug bits
* remove `.format` & add 'tarball' as version
'tarball' as version so that the temp files format nicely:
[tempfile_location]/dbt_utils_2..tar.gz # old
vs
[tempfile_location]/dbt_utils_1.tarball.tar.gz # current
* port os.path refs in `PinnedPackage._install` to pathlib
* lowercase as per PR feedback
* update tests after removing version arg
goes along with 8787ba41af
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* removed Compiled versions of nodes
* Remove compiled fields from dictionary if not compiled
* check compiled is False instead of attribute existence in env_var
processing
* Update artifacts test (CompiledSnapshotNode did not have SnapshotConfig)
* Changie
* more complicated 'compiling' check in env_var
* Update test_exit_codes.py
* CT-1405: Refactor event logging code
* CT-1405: Add changelog entry
* CT-1405: Add code to protect against using closed streams from past tests.
* CT-1405: Restore unit test which was only failing locally
* CT-1405: Document a hack with issue # to resolve it in the future
* CT-1405: Make black happy
* CT-1405: Get rid of confusing factory function and duplicated function
* CT-1405: Remove unused event from types.proto and auto-gen'd file
* Fix the partial parse path
Partial parse should use project root or it does not resolve to correct path.
Eg. `target-path: ../some/dir/target`, if not ran from root, creates an erroneous folder.
* Run pre-commit
* Changie
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
- Add --target-path to dbt snapshot command. ([#7418](https://github.com/dbt-labs/dbt-core/issues/7418))
- dbt build selection of tests' descendants ([#7289](https://github.com/dbt-labs/dbt-core/issues/7289))
- fix groupable node partial parsing, raise DbtReferenceError at runtime for safety ([#7437](https://github.com/dbt-labs/dbt-core/issues/7437))
- Fix partial parsing of latest_version changes for downstream references ([#7369](https://github.com/dbt-labs/dbt-core/issues/7369))
- Use "add_node" to update depends_on.nodes ([#7453](https://github.com/dbt-labs/dbt-core/issues/7453))
- Fix var precedence in configs: root vars override package vars ([#6705](https://github.com/dbt-labs/dbt-core/issues/6705))
- Fix inverted `--print/--no-print` flag ([#7517](https://github.com/dbt-labs/dbt-core/issues/7517))
- Back-compat for previous return type of 'collect_freshness' macro ([#7489](https://github.com/dbt-labs/dbt-core/issues/7489))
- print model version in dbt show if specified ([#7407](https://github.com/dbt-labs/dbt-core/issues/7407))
- enable dbt show for seeds ([#7273](https://github.com/dbt-labs/dbt-core/issues/7273))
- push down limit filtering to adapter ([#7390](https://github.com/dbt-labs/dbt-core/issues/7390))
- Allow missing `profiles.yml` for `dbt deps` and `dbt init` ([#7511](https://github.com/dbt-labs/dbt-core/issues/7511))
-`run_results.json` is now written after every node completes. ([#7302](https://github.com/dbt-labs/dbt-core/issues/7302))
- Do not rewrite manifest.json during 'docs serve' command ([#7553](https://github.com/dbt-labs/dbt-core/issues/7553))
- Pin protobuf to greater than 4.0.0 ([#7565](https://github.com/dbt-labs/dbt-core/issues/7565))
- inject sql header in query for show ([#7413](https://github.com/dbt-labs/dbt-core/issues/7413))
- Pin urllib3 to ~=1.0 ([#7573](https://github.com/dbt-labs/dbt-core/issues/7573))
- Throw error for duplicated versioned and unversioned models ([#7487](https://github.com/dbt-labs/dbt-core/issues/7487))
### Under the Hood
- Update docs link in ContractBreakingChangeError message ([#7366](https://github.com/dbt-labs/dbt-core/issues/7366))
- Reduce memory footprint of cached statement results. ([#7281](https://github.com/dbt-labs/dbt-core/issues/7281))
- Remove noisy parsing events: GenericTestFileParse, MacroFileParse, Note events for static model parsing ([#6671](https://github.com/dbt-labs/dbt-core/issues/6671))
- Update --help text for cache-related parameters ([#7381](https://github.com/dbt-labs/dbt-core/issues/7381))
- Small UX improvements to model versions: Support defining latest_version in unsuffixed file by default. Notify on unpinned ref when a prerelease version is available. ([#7443](https://github.com/dbt-labs/dbt-core/issues/7443))
- Added warnings for model and ref deprecations ([#7433](https://github.com/dbt-labs/dbt-core/issues/7433))
- Update drop_relation macro to allow for configuration of drop statement separately from object name ([#7625](https://github.com/dbt-labs/dbt-core/issues/7625))
- accept publications in dbt.invoke ([#7372](https://github.com/dbt-labs/dbt-core/issues/7372))
### Fixes
- Honor `--skip-profile-setup` parameter when inside an existing project ([#7594](https://github.com/dbt-labs/dbt-core/issues/7594))
- Fix: Relative project paths weren't working with deps ([#7491](https://github.com/dbt-labs/dbt-core/issues/7491))
- Exclude password fields from Jinja rendering. ([#7629](https://github.com/dbt-labs/dbt-core/issues/7629))
- Add --target-path to more CLI subcommands ([#7646](https://github.com/dbt-labs/dbt-core/issues/7646))
- Stringify flag paths for Jinja context ([#7495](https://github.com/dbt-labs/dbt-core/issues/7495))
- write run_results.json for run operation ([#7502](https://github.com/dbt-labs/dbt-core/issues/7502))
### Under the Hood
- Add ability to instantiate Flags class from dict ([#7607](https://github.com/dbt-labs/dbt-core/issues/7607))
- Add other relation to reffable nodes ([#7550](https://github.com/dbt-labs/dbt-core/issues/7550))
- Move node patch method to schema parser patch_node_properties and refactor schema parsing ([#7430](https://github.com/dbt-labs/dbt-core/issues/7430))
- When to use a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) vs the [GITHUB_TOKEN](https://docs.github.com/en/actions/security-guides/automatic-token-authentication) generated for the action?
The `GITHUB_TOKEN` is used by default. In most cases it is sufficient for what you need.
If you expect the workflow to result in a commit to that should retrigger workflows, you will need to use a Personal Access Token for the bot to commit the file. When using the GITHUB_TOKEN, the resulting commit will not trigger another GitHub Actions Workflow run. This is due to limitations set by GitHub. See [the docs](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#using-the-github_token-in-a-workflow) for a more detailed explanation.
For example, we must use a PAT in our workflow to commit a new changelog yaml file for bot PRs. Once the file has been committed to the branch, it should retrigger the check to validate that a changelog exists on the PR. Otherwise, it would stay in a failed state since the check would never retrigger.
@@ -105,7 +105,7 @@ Some triggers of note that we use:
```
# **what?**
# Describe what the action does.
# Describe what the action does.
# **why?**
# Why does this action exist?
@@ -138,7 +138,7 @@ Some triggers of note that we use:
- Print out all variables you will reference as the first step of a job. This allows for easier debugging. The first job should log all inputs. Subsequent jobs should reference outputs of other jobs, if present.
@@ -158,14 +158,14 @@ Some triggers of note that we use:
echo "The build_script_path: ${{ inputs.build_script_path }}"
echo "The s3_bucket_name: ${{ inputs.s3_bucket_name }}"
echo "The package_test_command: ${{ inputs.package_test_command }}"
# collect all the variables that need to be used in subsequent jobs
- Don’t use external actions for things that can easily be accomplished manually.
- Always read through what an external action does before using it! Often an action in the GitHub Actions Marketplace can be replaced with a few lines in bash. This is much more maintainable (and won’t change under us) and clear as to what’s actually happening. It also prevents any
- Pin actions _we don't control_ to tags.
- Pin actions _we don't control_ to tags.
### Connecting to AWS
- Authenticate with the aws managed workflow
```yaml
- name: Configure AWS credentials from Test account
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
