* first pass at automating latest branches
* checkout repo first
* fetch all history
* reorg
* debugging
* update test id
* swap lines
* incorporate new branch aciton
* tweak vars
* Formatting
* Changelog entry
* Rename to BaseSimpleSeedColumnOverride
* Better error handling
* Update test to include the BOM test
* Cleanup and formating
* Unused import remove
* nit line
* Pr comments
* update regex to match all iterations
* convert to num to match all adapters
* add comments, remove extra .
* clarify with more comments
* Update .bumpversion.cfg
Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
---------
Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
* Add clearer directions for custom test suite vars in Makefile.
* Fix up PR for review
* Fix erroneous whitespace.
* Fix a spelling error.
* Add documentation to discourage makefile edits but provide override tooling.
* Fix quotation marks. Very strange behavior
* Compact code and verify quotations happy inside bash and python.
* Fold comments into Makefile.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Convert test and make it a bit more pytest-onic
* Ax old integration test.
* Run black on test conversion
* I didn't like how pytest was running the fixture so wrapped it into a closure.
* Merge converted test into persist docs.
* Move persist docs tests to the adapter zone. Prep for adapter tests.
* Fix up test names
* Fix name to be less confusing.
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Test converted and reformatted for pytest.
* Ax old versions of 052 test
* Nix the 'os' import and black format
* Change names of models to be more PEP like
* cleanup code
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Convert test and make it a bit more pytest-onic
* Ax old integration test.
* Run black on test conversion
* I didn't like how pytest was running the fixture so wrapped it into a closure.
* Merge converted test into persist docs.
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* init commit for column_types test conversion
* init start of test_column_types.py
* pass tes macros into both tests
* remove alt tests, remove old tests, push up working conversion
* rename base class, move to adapter zone so adapters can use
* typo fix
* Code cleanup and adding stderr to capture dbt
* Debug with --log-format json now prints structured logs.
* Add changelog.
* Move logs into miscellaneous and add values to test.
* nix whitespace and fix log levels
* List will now do structured logging when log format set to json.
* Add a quick None check.
* Add a get guard to class check.
* Better null checking
* The boolean doesn't reflect the original logic but a try-catch does.
* Address some code review comments and get us working again.
* Simplify logic now that we have a namespace object for self.config.args.
* Simplify logic for json log format checking.
* Simplify code for allowing our GraphTest cases to pass while also hiding compile stats from dbt ls/list .
* Simplify structured logging types.
* Fix up boolean logic and simplify via De'Morgan.
* Nix unneeded fixture.
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* CT-1786: Port docs tets to pytest
* Add generated CLI API docs
* CT-1786: Comply with the new style requirements
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* add defer_to_manifest in before_run to fix faulty deferred docs generate
* add a changelog
* add declaration of defer_to_manifest to FreshnessTask and GraphRunnableTask
* fix: add defer_to_manifest method to ListTask
* Re-factor list of YAML keys for hooks to late-render
* Add `pre_` and `post_hook` to list of late-rendered hooks
* Check for non-empty set intersection
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
* Test functional synonymy of `*_hook` with `*-hook`
Test that `pre_hook`/`post_hook` are functionally synonymous with `pre-hook`/`post-hook` for model project config
* Undo bugfix to validate the new test fails
* Revert "Undo bugfix to validate the new test fails"
This reverts commit e83a2be2eb6f0ef4d95ffcfaf77ea9287eb0fbca.
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
* add meta attribute to nodeinfo for events
* also add meta to dataclass
* add to unit test to ensure meta is added
* adding functional test to check that meta is passed to nodeinfo during logging
* changelog
* remove used imported
* add tests with non-string keys
* renaming test dict keys
* add non-string value
* resolve failing test
* test additional non-string values
* fix flake8
* Stringify meta dict in node_info
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
* convert the test and fix an error due to a dead code seed
* Get rid of old test
* Remove unfortunately added files. Don't use that *
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Update types.proto
* pre-commit passes
* Cleanup tests and tweak EventLevels
* Put node_info back on SQLCommit. Add "level" to fire_event function.
* use event.message() in warn_or_error
* Fix logging test
* Changie
* Fix a couple of unit tests
* import Protocol from typing_extensions for 3.7
* ✨ adding pre-commit install to make dev
* 🎨 updating format of Makefile and CONTRIBUTING.md
* 📝 adding changelog via changie new
* ✨ adding dev_req to Makefile + docs
* 🎨 remove dev_req from docs, dry makefile
* Align names of `.PHONY` targets with their associated rules
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* starting to move jinja exceptions
* convert some exceptions
* add back old functions for backward compatibility
* organize
* more conversions
* more conversions
* add changelog
* split out CacheInconsistency
* more conversions
* convert even more
* convert parsingexceptions
* fix tests
* more conversions
* more conversions
* finish converting exception functions
* convert more tests
* standardize to msg
* remove some TODOs
* fix test param and check the rest
* add comment, move exceptions
* add types
* fix type errors
* fix type for adapter_response
* remove 0.13 version from message
* pass predicated to merge strategy
* postgres delete and insert
* merge with predicates
* update to use arbitrary list of predicates, not dictionaries, merge and delete
* changie
* add functional test to adapter zone
* comma in test config
* add test for incremental predicates delete and insert postgres
* update test structure for inheritance
* handle predicates config for backwards compatibility
* test for predicates keyword
* Add generated CLI API docs
Co-authored-by: Colin <colin.rogers@dbtlabs.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* Remove unneeded SQL compilation attributes from SeedNode
* Fix various places that referenced removed attributes
* Cleanup a few Unions
* More formatting in nodes.py
* Mypy passing. Untested.
* Unit tests working
* use "doc" in documentation unique_ids
* update some doc_ids
* Fix some artifact tests. Still need previous version.
* Update manifest/v8.json
* Move relation_names to parsing
* Fix a couple of tests
* Update some artifacts. snapshot_seed has wrong schema.
* Changie
* Tweak NodeType.Documentation
* Put store_failures property in the right place
* Fix setting relation_name
* update changie to require issue or pr, and allow multiple
* remove extraneous data from changelog files.
* allow for multiple PR/issues to be entered
* update contributing guide
* remove issue number from bot changelogs
* update format of PR
* fix dependency changelogs
* remove extra line
* remove extra lines, tweak contributor wording
* Update CONTRIBUTING.md
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Get running with Python 3.11
* More tests passing, mypy still unhappy
* Upgrade to 3.11, and bump mashumaro
* patch importlib.import_module last
* lambda: Policy() default_factory on include and quote policy
* Add changelog entry
* Put a lambda on it
* Fix text formatting for log file
* Handle variant type return from e.log_level()
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Josh Taylor <joshuataylorx@gmail.com>
Co-authored-by: Michelle Ark <michelle.ark@dbtlabs.com>
* feat: add a list of default values to the ctx manager
* tests: dbt.get.config default values
* feat: validate the num of args in config.get
* feat: jinja template for dbt.config.get default values
* docs: changie yaml
* fix:typo on error message
Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
Co-authored-by: Chenyu Li <chenyulee777@gmail.com>
* v0 - new dbt deps type: tarball url
in support of
https://github.com/dbt-labs/dbt-core/issues/4205
* flake8 fixes
* adding max size tarball condition
* clean up imports
* typing
* adding sha1 and subdirectory options; improve logging feedback
sha1: allow user to specify sha1 in packages.yaml, will only install if package matches
subdirectory: allow user to specify subdirectory of package in tarfile, if the package is a non standard structure (like with git subdirectory option)
* simple tests added
* flake fixes
* changes to support tests; adding exceptions; fire_event logging
* new logging events
* tarball exceptions added
* build out tests
* removing in memory tarball test
* update type codes to M - Misc
* adding new events to test_events
* fix spacing for flake
* add retry download code - as used in registry calls
* clean
* remove saving tar in memory inside tarfile object
will hit url multiple times instead
* remove duplicative code after refactor
* black updates
* black formatting
* black formatting
* refactor - no more in-memory tarfile - all as file operations now
- remove tarfile passing, always use tempfile instead
- reorganize system.* functions, removing duplicative code
- more notes on current flow and structure - esp need for pattern of 1) unpack 2) scan for package dir 3) copy to destination.
- cleaning
* cleaning and sync to new tarball code
* cleaning and sync to new tarball code
* requested changes from PR
https://github.com/dbt-labs/dbt-core/pull/4689#discussion_r812970847
* reversions from revision 2
removing sha1 check to simplify/mirror hub install pattern
* simplify/mirror hub install pattern
to simplify/mirror hub install pattern
- removing sha1 check
- supply name/version to act as our 'metadata' source
* simplify/mirror hub install pattern
simplify with goal of mirroring hub install pattern
- supporting subfolders like git packages, and sha1 checks are removed
- existing code from RegistryPinnedPackage (install() and download_and_untar()) performs the operations
- RegistryPinnedPackage install() and download_and_untar() are not currently set up as functions that can be used across classes - this should be moved to dbt.deps.base, or to a dbt.deps.common file - need dbt labs feedback on how to proceed (or leave as is)
* remove revisions, no longer doing package check
* slim down to basic tests
more complex features have been removed (sha1, subfolder) so testing is much simpler!
* fix naming to match hubs behavior
remove version from package folder name
* refactor install and download to upstream PinnedPackage class
i'm on the fence if this is right approach, but seems like most sensible after some thought
* Create Features-20221107-105018.yaml
* fix flake, black, mypy errors
* additional flake/black fixes
* Update .changes/unreleased/Features-20221107-105018.yaml
fix username on changelog
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* change to fstring
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* cleaning - remove comment
* remove comment/question for dbt team
* in support of issuecomment 1334055944
https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334055944
* in support of issuecomment 1334118433
https://github.com/dbt-labs/dbt-core/pull/4689#issuecomment-1334118433
* black fixes; remove debug bits
* remove `.format` & add 'tarball' as version
'tarball' as version so that the temp files format nicely:
[tempfile_location]/dbt_utils_2..tar.gz # old
vs
[tempfile_location]/dbt_utils_1.tarball.tar.gz # current
* port os.path refs in `PinnedPackage._install` to pathlib
* lowercase as per PR feedback
* update tests after removing version arg
goes along with 8787ba41af585b954d18c24c2dccbf147035cd9f
Co-authored-by: Emily Rockman <ebuschang@gmail.com>
* removed Compiled versions of nodes
* Remove compiled fields from dictionary if not compiled
* check compiled is False instead of attribute existence in env_var
processing
* Update artifacts test (CompiledSnapshotNode did not have SnapshotConfig)
* Changie
* more complicated 'compiling' check in env_var
* Update test_exit_codes.py
* CT-1405: Refactor event logging code
* CT-1405: Add changelog entry
* CT-1405: Add code to protect against using closed streams from past tests.
* CT-1405: Restore unit test which was only failing locally
* CT-1405: Document a hack with issue # to resolve it in the future
* CT-1405: Make black happy
* CT-1405: Get rid of confusing factory function and duplicated function
* CT-1405: Remove unused event from types.proto and auto-gen'd file
* Fix the partial parse path
Partial parse should use project root or it does not resolve to correct path.
Eg. `target-path: ../some/dir/target`, if not ran from root, creates an erroneous folder.
* Run pre-commit
* Changie
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
- Added favor-state flag to optionally favor state nodes even if unselected node exists ([#2968](https://github.com/dbt-labs/dbt-core/issues/2968))
- Update structured logging. Convert to using protobuf messages. Ensure events are enriched with node_info. ([#5610](https://github.com/dbt-labs/dbt-core/issues/5610))
- Friendlier error messages when packages.yml is malformed ([#5486](https://github.com/dbt-labs/dbt-core/issues/5486))
- Migrate dbt-utils current_timestamp macros into core + adapters ([#5521](https://github.com/dbt-labs/dbt-core/issues/5521))
- Allow partitions in external tables to be supplied as a list ([#5929](https://github.com/dbt-labs/dbt-core/issues/5929))
- extend -f flag shorthand for seed command ([#5990](https://github.com/dbt-labs/dbt-core/issues/5990))
- This pulls the profile name from args when constructing a RuntimeConfig in lib.py, enabling the dbt-server to override the value that's in the dbt_project.yml ([#6201](https://github.com/dbt-labs/dbt-core/issues/6201))
- Adding tarball install method for packages. Allowing package tarball to be specified via url in the packages.yaml. ([#4205](https://github.com/dbt-labs/dbt-core/issues/4205))
- Added an md5 function to the base context ([#6246](https://github.com/dbt-labs/dbt-core/issues/6246))
- Exposures support metrics in lineage ([#6057](https://github.com/dbt-labs/dbt-core/issues/6057))
- Add support for Python 3.11 ([#6147](https://github.com/dbt-labs/dbt-core/issues/6147))
- Account for disabled flags on models in schema files more completely ([#3992](https://github.com/dbt-labs/dbt-core/issues/3992))
- Add validation of enabled config for metrics, exposures and sources ([#6030](https://github.com/dbt-labs/dbt-core/issues/6030))
- check length of args of python model function before accessing it ([#6041](https://github.com/dbt-labs/dbt-core/issues/6041))
- Add functors to ensure event types with str-type attributes are initialized to spec, even when provided non-str type params. ([#5436](https://github.com/dbt-labs/dbt-core/issues/5436))
- Allow hooks to fail without halting execution flow ([#5625](https://github.com/dbt-labs/dbt-core/issues/5625))
- Clarify Error Message for how many models are allowed in a Python file ([#6245](https://github.com/dbt-labs/dbt-core/issues/6245))
- After this, will be possible to use default values for dbt.config.get ([#6309](https://github.com/dbt-labs/dbt-core/issues/6309))
- Use full path for writing manifest ([#6055](https://github.com/dbt-labs/dbt-core/issues/6055))
- [CT-1284] Change Python model default materialization to table ([#6345](https://github.com/dbt-labs/dbt-core/issues/6345))
- Repair a regression which prevented basic logging before the logging subsystem is completely configured. ([#6434](https://github.com/dbt-labs/dbt-core/issues/6434))
### Docs
- minor doc correction ([dbt-docs/#5791](https://github.com/dbt-labs/dbt-docs/issues/5791))
- Generate API docs for new CLI interface ([dbt-docs/#5528](https://github.com/dbt-labs/dbt-docs/issues/5528))
- Add dmypy cache to gitignore ([#6028](https://github.com/dbt-labs/dbt-core/issues/6028))
- Provide useful errors when the value of 'materialized' is invalid ([#5229](https://github.com/dbt-labs/dbt-core/issues/5229))
- Clean up string formatting ([#6068](https://github.com/dbt-labs/dbt-core/issues/6068))
- Fixed extra whitespace in strings introduced by black. ([#1350](https://github.com/dbt-labs/dbt-core/issues/1350))
- Remove the 'root_path' field from most nodes ([#6171](https://github.com/dbt-labs/dbt-core/issues/6171))
- Combine certain logging events with different levels ([#6173](https://github.com/dbt-labs/dbt-core/issues/6173))
- Convert threading tests to pytest ([#5942](https://github.com/dbt-labs/dbt-core/issues/5942))
- Convert postgres index tests to pytest ([#5770](https://github.com/dbt-labs/dbt-core/issues/5770))
- Convert use color tests to pytest ([#5771](https://github.com/dbt-labs/dbt-core/issues/5771))
- Add github actions workflow to generate high level CLI API docs ([#5942](https://github.com/dbt-labs/dbt-core/issues/5942))
- Functionality-neutral refactor of event logging system to improve encapsulation and modularity. ([#6139](https://github.com/dbt-labs/dbt-core/issues/6139))
- Consolidate ParsedNode and CompiledNode classes ([#6383](https://github.com/dbt-labs/dbt-core/issues/6383))
- Prevent doc gen workflow from running on forks ([#6386](https://github.com/dbt-labs/dbt-core/issues/6386))
- Fix intermittent database connection failure in Windows CI test ([#6394](https://github.com/dbt-labs/dbt-core/issues/6394))
- Refactor and clean up manifest nodes ([#6426](https://github.com/dbt-labs/dbt-core/issues/6426))
- Restore important legacy logging behaviors, following refactor which removed them ([#6437](https://github.com/dbt-labs/dbt-core/issues/6437))
### Dependencies
- Update pathspec requirement from ~=0.9.0 to >=0.9,<0.11in/core([#5917](https://github.com/dbt-labs/dbt-core/pull/5917))
body:Replaced the EmptyLine event with a more general Formatting event, and added
a Note event.
time:2023-01-20T17:22:54.45828-05:00
custom:
Author:peterallenwebb
Issue:"6481"
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
