Compare commits

..

1 Commits

Author SHA1 Message Date
Michelle Ark
1552eccb05 update index.html for dbt docs 2024-05-22 17:47:27 -04:00
513 changed files with 7060 additions and 53350 deletions

View File

@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.11.0a1
current_version = 1.9.0a1
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number

View File

@@ -3,9 +3,6 @@
For information on prior major and minor releases, see their changelogs:
* [1.10](https://github.com/dbt-labs/dbt-core/blob/1.10.latest/CHANGELOG.md)
* [1.9](https://github.com/dbt-labs/dbt-core/blob/1.9.latest/CHANGELOG.md)
* [1.8](https://github.com/dbt-labs/dbt-core/blob/1.8.latest/CHANGELOG.md)
* [1.7](https://github.com/dbt-labs/dbt-core/blob/1.7.latest/CHANGELOG.md)
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
* [1.5](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)

View File

@@ -0,0 +1,6 @@
kind: Dependencies
body: Remove logbook dependency
time: 2024-05-09T09:37:17.745129-05:00
custom:
Author: emmyoop
Issue: "8027"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Bump minimum jsonschema version to `4.19.1`
time: 2025-06-16T14:44:08.512306-05:00
custom:
Author: QMalcolm
Issue: "11740"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Allow for either pydantic v1 and v2
time: 2025-06-20T12:36:00.196384-05:00
custom:
Author: QMalcolm
Issue: "11634"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Bump dbt-common minimum to 1.25.1
time: 2025-07-02T14:47:20.772002-05:00
custom:
Author: QMalcolm
Issue: "11789"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Upgrade to dbt-semantic-interfaces==0.9.0 for more robust saved query support.
time: 2025-07-09T13:22:13.688162-07:00
custom:
Author: courtneyholcomb
Issue: "11809"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: upgrade protobuf to 6.0
time: 2025-08-13T14:48:32.380929-07:00
custom:
Author: colin-rogers-dbt
Issue: "11916"

View File

@@ -1,6 +0,0 @@
kind: Dependencies
body: Bump dbt-adapters minimum to 1.16.5
time: 2025-08-19T08:57:14.396843-05:00
custom:
Author: QMalcolm
Issue: "11932"

View File

@@ -0,0 +1,6 @@
kind: Docs
body: Fix rendering docs with saved queries
time: 2024-05-22T17:47:13.414938-04:00
custom:
Author: ChenyuLInx michelleark
Issue: "10168"

View File

@@ -0,0 +1,6 @@
kind: Features
body: serialize inferred primary key
time: 2024-05-06T17:56:42.757673-05:00
custom:
Author: dave-connors-3
Issue: "9824"

View File

@@ -0,0 +1,6 @@
kind: Features
body: 'Add unit_test: selection method'
time: 2024-05-07T16:27:17.047585-04:00
custom:
Author: michelleark
Issue: "10053"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Add file_format to catalog integration config
time: 2025-05-29T08:53:11.64904-07:00
custom:
Author: colin-rogers-dbt
Issue: "11695"

View File

@@ -1,6 +0,0 @@
kind: Features
body: "11561"
time: 2025-06-11T16:02:17.334525-04:00
custom:
Author: michelleark
Issue: deprecate --models,--model, and -m flags

View File

@@ -1,7 +0,0 @@
kind: Features
body: Update jsonschemas with builtin data test properties and exposure configs in
dbt_project.yml for more accurate deprecations
time: 2025-06-17T14:25:16.976867-04:00
custom:
Author: michelleark
Issue: "11335"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support loaded_at_query and loaded_at_field on source and table configs
time: 2025-06-23T11:31:30.897805-04:00
custom:
Author: michelleark
Issue: "11659"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Begin validating configs from model sql files
time: 2025-06-25T15:18:18.164-05:00
custom:
Author: QMalcolm
Issue: "11727"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Deprecate `overrides` property for sources
time: 2025-07-01T16:49:57.979871-05:00
custom:
Author: peterallenwebb
Issue: "11566"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Create constrained namespace for dbt engine env vars
time: 2025-07-03T17:53:41.406701-05:00
custom:
Author: QMalcolm
Issue: "11340"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Gate jsonschema validations by adapter
time: 2025-07-14T23:25:24.475471-05:00
custom:
Author: QMalcolm
Issue: "11680"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Deprecate top-level argument properties in generic tests
time: 2025-07-21T17:31:00.960402-04:00
custom:
Author: michelleark
Issue: "11847"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Deprecate {{ modules.itertools }} usage
time: 2025-07-28T11:54:43.28275-04:00
custom:
Author: michelleark
Issue: "11725"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Default require_generic_test_arguments_property flag to True - The 'arguments' property will be parsed as keyword arguments to data tests, if provided
time: 2025-08-11T15:03:47.369377+02:00
custom:
Author: michelleark
Issue: "11911"

View File

@@ -1,6 +0,0 @@
kind: Features
body: Support Nested Key Traversal in dbt ls json output
time: 2025-08-15T13:14:58.162067-05:00
custom:
Author: trouze
Issue: "11919"

View File

@@ -1,6 +0,0 @@
kind: Features
body: No-op when project-level `quoting.snowflake_ignore_case` is set.
time: 2025-08-21T09:25:32.049328307+02:00
custom:
Author: aksestok
Issue: "11882"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Remove unused check_new method
time: 2023-06-01T20:41:57.556342+02:00
custom:
Author: kevinneville
Issue: "7586"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: 'Restore previous behavior for --favor-state: only favor defer_relation if not
selected in current command"'
time: 2024-05-08T15:11:27.510912+02:00
custom:
Author: jtcohen6
Issue: "10107"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Unit test fixture (csv) returns null for empty value
time: 2024-05-09T09:14:11.772709-04:00
custom:
Author: michelleark
Issue: "9881"

View File

@@ -0,0 +1,7 @@
kind: Fixes
body: Fix json format log and --quiet for ls and jinja print by converting print call
to fire events
time: 2024-05-16T15:39:13.896723-07:00
custom:
Author: ChenyuLInx
Issue: "8756"

View File

@@ -0,0 +1,6 @@
kind: Fixes
body: Add resource type to saved_query
time: 2024-05-16T22:35:10.287514-07:00
custom:
Author: ChenyuLInx
Issue: "10168"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Don't warn for metricflow_time_spine with non-day grain
time: 2025-05-28T09:20:55.866514-07:00
custom:
Author: courtneyholcomb
Issue: "11690"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix source freshness set via config to handle explicit nulls
time: 2025-05-30T00:58:04.94133-05:00
custom:
Author: QMalcolm
Issue: "11685"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Ensure build_after is present in model freshness in parsing, otherwise skip
freshness definition
time: 2025-06-05T11:06:45.329942-07:00
custom:
Author: QMalcolm
Issue: "11709"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Ensure source node `.freshness` is equal to node's `.config.freshness`
time: 2025-06-09T17:52:39.978403-05:00
custom:
Author: QMalcolm
Issue: "11717"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ignore invalid model freshness configs in inline model configs
time: 2025-06-10T21:12:41.972614-07:00
custom:
Author: colin-rogers-dbt
Issue: "11728"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Fix store_failures hierarachical config parsing
time: 2025-06-12T14:51:59.358498-04:00
custom:
Author: michelleark
Issue: "10165"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: "Remove model freshness property support in favor of config level support"
time: 2025-06-16T08:56:00.641553-05:00
custom:
Author: QMalcolm
Issue: "11713"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Bump dbt-common to 1.25.0 to access WarnErrorOptionsV2
time: 2025-06-24T09:12:58.904713-04:00
custom:
Author: michelleark
Issue: "11755"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: ensure consistent casing in column names while processing user unit tests
time: 2025-06-24T15:20:47.589212-07:00
custom:
Author: colin-rogers-dbt
Issue: "11770"

View File

@@ -1,7 +0,0 @@
kind: Fixes
body: Update jsonschema definitions with nested config defs, cloud info, and dropping
source overrides
time: 2025-07-07T10:34:18.982725-05:00
custom:
Author: QMalcolm
Issue: N/A

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Make `GenericJSONSchemaValidationDeprecation` a "preview" deprecation
time: 2025-07-10T17:01:48.903582-05:00
custom:
Author: QMalcolm
Issue: "11814"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Correct JSONSchema Semantic Layer node issues
time: 2025-07-11T09:54:39.86192-05:00
custom:
Author: MichelleArk
Issue: "11818"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Improve SL JSONSchema definitions
time: 2025-07-14T12:50:00.3541-05:00
custom:
Author: QMalcolm
Issue: N/A

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: raise MissingPlusPrefixDeprecation instead of GenericJSONSchemaValidationDeprecation when config missing plus prefix in dbt_project.yml
time: 2025-07-14T18:41:31.322137-04:00
custom:
Author: michelleark
Issue: "11826"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Propagate config.meta and config.tags to top-level on source nodes
time: 2025-07-16T16:45:35.683199-04:00
custom:
Author: michelleark
Issue: "11839"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Safe handling of malformed config.tags on sources/tables
time: 2025-07-24T10:44:10.828775-04:00
custom:
Author: michelleark
Issue: "11855"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Quoting the event_time field when the configuration says so
time: 2025-07-25T11:29:03.342884+02:00
custom:
Author: pablomc87
Issue: "11858"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Raise PropertyMovedToConfigDeprecation instead of CustomTopLevelKeyDeprecation when additional attribute is a valid node config
time: 2025-07-31T16:21:42.938703-04:00
custom:
Author: michelleark
Issue: "11879"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Avoid redundant node patch removal during partial parsing
time: 2025-08-04T07:40:06.993913-06:00
custom:
Author: wircho
Issue: "11886"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Comply with strict `str` type when `block.contents` is `None`
time: 2025-08-04T07:42:54.612616-06:00
custom:
Author: wircho
Issue: "11492"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Remove duplicative PropertyMovedToConfigDeprecation for source freshness
time: 2025-08-05T10:43:09.502585-04:00
custom:
Author: michelleark
Issue: "11880"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Add path to MissingArgumentsPropertyInGenericTestDeprecation message
time: 2025-08-20T19:53:39.282829+02:00
custom:
Author: michelleark
Issue: "11940"

View File

@@ -1,6 +0,0 @@
kind: Fixes
body: Unhide sample mode CLI flag
time: 2025-08-26T14:48:59.685423-05:00
custom:
Author: QMalcolm
Issue: "11959"

View File

@@ -0,0 +1,6 @@
kind: Security
body: Explicitly bind to localhost in docs serve
time: 2024-05-22T09:45:40.748185-04:00
custom:
Author: ChenyuLInx michelleark
Issue: "10209"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Clear error message for Private package in dbt-core
time: 2024-05-02T15:44:30.713097-07:00
custom:
Author: ChenyuLInx
Issue: "10083"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Enable use of context in serialization
time: 2024-05-06T14:55:11.1812-04:00
custom:
Author: gshank
Issue: "10093"

View File

@@ -0,0 +1,6 @@
kind: Under the Hood
body: Make RSS high water mark measurement more accurate on Linux
time: 2024-05-19T15:59:46.700842315-04:00
custom:
Author: peterallenwebb
Issue: "10177"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Prevent overcounting PropertyMovedToConfigDeprecation for source freshness
time: 2025-05-27T16:21:36.551426+01:00
custom:
Author: aranke
Issue: "11660"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: call adapter.add_catalog_integration during parse_manifest
time: 2025-08-04T21:13:46.077488-07:00
custom:
Author: colin-rogers-dbt
Issue: "11889"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Fix docker os dependency install issue
time: 2025-08-19T12:34:48.402158-05:00
custom:
Author: QMalcolm
Issue: "11934"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Ensure dbt-core modules aren't importing versioned artifact resources directly
time: 2025-08-22T16:27:52.403177-05:00
custom:
Author: QMalcolm
Issue: "11951"

View File

@@ -1,6 +0,0 @@
kind: Under the Hood
body: Update jsonschemas used for schema-based deprecations
time: 2025-09-03T16:18:33.931722-04:00
custom:
Author: michellark
Issue: "11987"

View File

@@ -7,7 +7,6 @@ ignore =
W503 # makes Flake8 work like black
W504
E203 # makes Flake8 work like black
E704 # makes Flake8 work like black
E741
E501 # long line checking is done in black
exclude = test/

View File

@@ -61,7 +61,7 @@ body:
label: Environment
description: |
examples:
- **OS**: Ubuntu 24.04
- **OS**: Ubuntu 20.04
- **Python**: 3.9.12 (`python3 --version`)
- **dbt-core**: 1.1.1 (`dbt --version`)
value: |

View File

@@ -1,18 +0,0 @@
name: 📄 Code docs
description: Report an issue for markdown files within this repo, such as README, ARCHITECTURE, etc.
title: "[Code docs] <title>"
labels: ["triage"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this code docs issue!
- type: textarea
attributes:
label: Please describe the issue and your proposals.
description: |
Links? References? Anything that will give us more context about the issue you are encountering!
Tip: You can attach images by clicking this area to highlight it and then dragging files in.
validations:
required: false

View File

@@ -1,8 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Documentation
url: https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose
about: Problems and issues with dbt product documentation hosted on docs.getdbt.com. Issues for markdown files within this repo, such as README, should be opened using the "Code docs" template.
- name: Ask the community for help
url: https://github.com/dbt-labs/docs.getdbt.com/discussions
about: Need help troubleshooting? Check out our guide on how to ask
@@ -12,6 +9,15 @@ contact_links:
- name: Participate in Discussions
url: https://github.com/dbt-labs/dbt-core/discussions
about: Do you have a Big Idea for dbt? Read open discussions, or start a new one
- name: Create an issue for adapters
url: https://github.com/dbt-labs/dbt-adapters/issues/new/choose
about: Report a bug or request a feature for an adapter
- name: Create an issue for dbt-redshift
url: https://github.com/dbt-labs/dbt-redshift/issues/new/choose
about: Report a bug or request a feature for dbt-redshift
- name: Create an issue for dbt-bigquery
url: https://github.com/dbt-labs/dbt-bigquery/issues/new/choose
about: Report a bug or request a feature for dbt-bigquery
- name: Create an issue for dbt-snowflake
url: https://github.com/dbt-labs/dbt-snowflake/issues/new/choose
about: Report a bug or request a feature for dbt-snowflake
- name: Create an issue for dbt-spark
url: https://github.com/dbt-labs/dbt-spark/issues/new/choose
about: Report a bug or request a feature for dbt-spark

View File

@@ -55,7 +55,7 @@ body:
label: Environment
description: |
examples:
- **OS**: Ubuntu 24.04
- **OS**: Ubuntu 20.04
- **Python**: 3.9.12 (`python3 --version`)
- **dbt-core (working version)**: 1.1.1 (`dbt --version`)
- **dbt-core (regression version)**: 1.2.0 (`dbt --version`)

8
.github/_README.md vendored
View File

@@ -120,7 +120,7 @@ Some triggers of note that we use:
```yaml
jobs:
dependency_changelog:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- name: Get File Name Timestamp
@@ -188,12 +188,6 @@ ___
- The [GitHub CLI](https://cli.github.com/) is available in the default runners
- Actions run in your context. ie, using an action from the marketplace that uses the GITHUB_TOKEN uses the GITHUB_TOKEN generated by your workflow run.
### Runners
- We dynamically set runners based on repository vars. Admins can view repository vars and reset them. Current values are the following but are subject to change:
- `vars.UBUNTU_LATEST` -> `ubuntu-latest`
- `vars.WINDOWS_LATEST` -> `windows-latest`
- `vars.MACOS_LATEST` -> `macos-14`
### Actions from the Marketplace
- Dont use external actions for things that can easily be accomplished manually.
- Always read through what an external action does before using it! Often an action in the GitHub Actions Marketplace can be replaced with a few lines in bash. This is much more maintainable (and wont change under us) and clear as to whats actually happening. It also prevents any

View File

@@ -33,7 +33,7 @@ on:
jobs:
build:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Wrangle latest tag

View File

@@ -3,24 +3,24 @@ on:
workflow_dispatch:
inputs:
package:
description: The package to publish
required: true
description: The package to publish
required: true
version_number:
description: The version number
required: true
description: The version number
required: true
jobs:
build:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler
with:
package: ${{ github.event.inputs.package }}
new_version: ${{ github.event.inputs.new_version }}
gh_token: ${{ secrets.GITHUB_TOKEN }}
- name: Print the results
run: |
echo "Is it latest? Survey says: ${{ steps.is_latest.outputs.latest }} !"
- uses: actions/checkout@v3
- name: Wrangle latest tag
id: is_latest
uses: ./.github/actions/latest-wrangler
with:
package: ${{ github.event.inputs.package }}
new_version: ${{ github.event.inputs.new_version }}
gh_token: ${{ secrets.GITHUB_TOKEN }}
- name: Print the results
run: |
echo "Is it latest? Survey says: ${{ steps.is_latest.outputs.latest }} !"

View File

@@ -0,0 +1,10 @@
name: "Set up postgres (linux)"
description: "Set up postgres service on linux vm for dbt integration tests"
runs:
using: "composite"
steps:
- shell: bash
run: |
sudo systemctl start postgresql.service
pg_isready
sudo -u postgres bash ${{ github.action_path }}/setup_db.sh

View File

@@ -0,0 +1 @@
../../../test/setup_db.sh

View File

@@ -0,0 +1,24 @@
name: "Set up postgres (macos)"
description: "Set up postgres service on macos vm for dbt integration tests"
runs:
using: "composite"
steps:
- shell: bash
run: |
brew services start postgresql
echo "Check PostgreSQL service is running"
i=10
COMMAND='pg_isready'
while [ $i -gt -1 ]; do
if [ $i == 0 ]; then
echo "PostgreSQL service not ready, all attempts exhausted"
exit 1
fi
echo "Check PostgreSQL service status"
eval $COMMAND && break
echo "PostgreSQL service not ready, wait 10 more sec, attempts left: $i"
sleep 10
((i--))
done
createuser -s postgres
bash ${{ github.action_path }}/setup_db.sh

View File

@@ -0,0 +1 @@
../../../test/setup_db.sh

View File

@@ -5,22 +5,8 @@ runs:
steps:
- shell: pwsh
run: |
Write-Host -Object "Installing PostgreSQL 16 as windows service..."
$installerArgs = @("--install_runtimes 0", "--superpassword root", "--enable_acledit 1", "--unattendedmodeui none", "--mode unattended")
$filePath = Invoke-DownloadWithRetry -Url "https://get.enterprisedb.com/postgresql/postgresql-16.1-1-windows-x64.exe" -Path "$env:PGROOT/postgresql-16.1-1-windows-x64.exe"
Start-Process -FilePath $filePath -ArgumentList $installerArgs -Wait -PassThru
Write-Host -Object "Validating PostgreSQL 16 Install..."
Get-Service -Name postgresql*
$pgReady = Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
$exitCode = $pgReady.ExitCode
if ($exitCode -ne 0) {
Write-Host -Object "PostgreSQL is not ready. Exitcode: $exitCode"
exit $exitCode
}
Write-Host -Object "Starting PostgreSQL 16 Service..."
$pgService = Get-Service -Name postgresql-x64-16
$pgService = Get-Service -Name postgresql*
Set-Service -InputObject $pgService -Status running -StartupType automatic
Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
$env:Path += ";$env:PGBIN"
bash ${{ github.action_path }}/setup_db.sh

View File

@@ -1,7 +1,7 @@
Resolves #
resolves #
<!---
Include the number of the issue addressed by this PR above, if applicable.
Include the number of the issue addressed by this PR above if applicable.
PRs for code changes without an associated issue *will not be merged*.
See CONTRIBUTING.md for more information.
@@ -26,8 +26,8 @@ Resolves #
### Checklist
- [ ] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me.
- [ ] I have run this code in development, and it appears to resolve the stated issue.
- [ ] This PR includes tests, or tests are not required or relevant for this PR.
- [ ] This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
- [ ] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions.
- [ ] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [ ] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions

View File

@@ -1,186 +0,0 @@
# **what?**
# Enforces 2 reviews when artifact or validation files are modified.
# **why?**
# Ensure artifact changes receive proper review from designated team members. GitHub doesn't support
# multiple reviews on a single PR based on files changed, so we need to enforce this manually.
# **when?**
# This will run when reviews are submitted and dismissed.
name: "Enforce Additional Reviews on Artifact and Validations Changes"
permissions:
checks: write
pull-requests: write
contents: read
on:
# trigger check on review events. use pull_request_target for forks.
pull_request_target:
types: [opened, reopened, ready_for_review, synchronize, review_requested]
pull_request_review:
types: [submitted, edited, dismissed]
# only run this once per PR at a time
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
env:
required_approvals: 2
team: "core-group"
jobs:
check-reviews:
name: "Validate Additional Reviews"
runs-on: ubuntu-latest
steps:
- name: "Get list of changed files"
id: changed_files
run: |
# Fetch files as JSON and process with jq to sanitize output
gh api repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files \
| jq -r '.[].filename' \
| while IFS= read -r file; do
# Sanitize the filename by removing any special characters and command injection attempts
clean_file=$(echo "$file" | sed 's/[^a-zA-Z0-9\.\/\-_]//g')
echo "$clean_file"
done > changed_files.txt
echo "CHANGED_FILES<<EOF" >> $GITHUB_OUTPUT
cat changed_files.txt >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Check if any artifact files changed"
id: artifact_files_changed
run: |
artifact_changes=false
while IFS= read -r file; do
# Only process if file path looks legitimate
if [[ "$file" =~ ^[a-zA-Z0-9\.\/\-_]+$ ]]; then
if [[ "$file" == "core/dbt/artifacts/"* ]] ; then
artifact_changes=true
break
fi
fi
done < changed_files.txt
echo "artifact_changes=$artifact_changes" >> $GITHUB_OUTPUT
- name: "Get Core Team Members"
if: steps.artifact_files_changed.outputs.artifact_changes == 'true'
id: core_members
run: |
gh api -H "Accept: application/vnd.github+json" \
/orgs/dbt-labs/teams/${{ env.team }}/members > core_members.json
# Extract usernames and set as multiline output
echo "membership<<EOF" >> $GITHUB_OUTPUT
jq -r '.[].login' core_members.json >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
env:
GH_TOKEN: ${{ secrets.IT_TEAM_MEMBERSHIP }}
- name: "Verify ${{ env.required_approvals }} core team approvals"
if: steps.artifact_files_changed.outputs.artifact_changes == 'true'
id: check_approvals
run: |
# Get all reviews
REVIEWS=$(gh api repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/reviews)
echo "All reviews:"
echo "$REVIEWS"
# Count approved reviews from core team members (only most recent review per user)
CORE_APPROVALS=0
while IFS= read -r member; do
echo "Checking member: $member"
APPROVED=$(echo "$REVIEWS" | jq --arg user "$member" '
group_by(.user.login) |
map(select(.[0].user.login == $user) |
sort_by(.submitted_at) |
last) |
map(select(.state == "APPROVED" and (.state != "DISMISSED"))) |
length')
echo "Latest review state for $member: $APPROVED"
CORE_APPROVALS=$((CORE_APPROVALS + APPROVED))
echo "Running total: $CORE_APPROVALS"
done <<< "${{ steps.core_members.outputs.membership }}"
echo "CORE_APPROVALS=$CORE_APPROVALS" >> $GITHUB_OUTPUT
echo "CORE_APPROVALS=$CORE_APPROVALS"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Find Comment"
if: steps.artifact_files_changed.outputs.artifact_changes == 'true' && steps.check_approvals.outputs.CORE_APPROVALS < env.required_approvals
uses: peter-evans/find-comment@v2
id: find-comment
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: 'github-actions[bot]'
body-includes: "### Additional Artifact Review Required"
- name: "Create Comment"
if: steps.artifact_files_changed.outputs.artifact_changes == 'true' && steps.find-comment.outputs.comment-id == '' && steps.check_approvals.outputs.CORE_APPROVALS < env.required_approvals
uses: peter-evans/create-or-update-comment@v3
with:
issue-number: ${{ github.event.pull_request.number }}
body: |
### Additional Artifact Review Required
Changes to artifact directory files requires at least ${{ env.required_approvals }} approvals from core team members.
- name: "Notify if not enough approvals"
if: steps.artifact_files_changed.outputs.artifact_changes == 'true'
run: |
if [[ "${{ steps.check_approvals.outputs.CORE_APPROVALS }}" -ge "${{ env.required_approvals }}" ]]; then
title="Extra requirements met"
message="Changes to artifact directory files requires at least ${{ env.required_approvals }} approvals from core team members. Current number of core team approvals: ${{ steps.check_approvals.outputs.CORE_APPROVALS }} "
echo "::notice title=$title::$message"
echo "REVIEW_STATUS=success" >> $GITHUB_OUTPUT
else
title="PR Approval Requirements Not Met"
message="Changes to artifact directory files requires at least ${{ env.required_approvals }} approvals from core team members. Current number of core team approvals: ${{ steps.check_approvals.outputs.CORE_APPROVALS }} "
echo "::notice title=$title::$message"
echo "REVIEW_STATUS=neutral" >> $GITHUB_OUTPUT
fi
id: review_check
- name: "Set check status"
id: status_check
run: |
if [[ "${{ steps.artifact_files_changed.outputs.artifact_changes }}" == 'false' ]]; then
# no extra review required
echo "current_status=success" >> $GITHUB_OUTPUT
elif [[ "${{ steps.review_check.outputs.REVIEW_STATUS }}" == "success" ]]; then
# we have all the required reviews
echo "current_status=success" >> $GITHUB_OUTPUT
else
# neutral exit - neither success nor failure
# we can't fail here because we use multiple triggers for this workflow and they won't reset the check
# workaround is to use a neutral exit to skip the check run until it's actually successful
echo "current_status=neutral" >> $GITHUB_OUTPUT
fi
- name: "Post Event"
# This step posts the status of the check because the workflow is triggered by multiple events
# and we need to ensure the check is always updated. Otherwise we would end up with duplicate
# checks in the GitHub UI.
run: |
if [[ "${{ steps.status_check.outputs.current_status }}" == "success" ]]; then
state="success"
else
state="failure"
fi
gh api \
--method POST \
-H "Accept: application/vnd.github+json" \
/repos/${{ github.repository }}/statuses/${{ github.event.pull_request.base.sha }} \
-f state="$state" \
-f description="Artifact Review Check" \
-f context="Artifact Review Check" \
-f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
env:
GH_TOKEN: ${{ secrets.FISHTOWN_BOT_PAT }}

View File

@@ -1,50 +0,0 @@
# **what?**
# Check if the an issue is opened near or during an extended holiday period.
# If so, post an automatically-generated comment about the holiday for bug reports.
# Also provide specific information to customers of dbt Cloud.
# **why?**
# Explain why responses will be delayed during our holiday period.
# **when?**
# This will run when new issues are opened.
name: Auto-Respond to Bug Reports During Holiday Period
on:
issues:
types:
- opened
permissions:
contents: read
issues: write
jobs:
auto-response:
runs-on: ${{ vars.UBUNTU_LATEST }}
steps:
- name: Check if current date is within holiday period
id: date-check
run: |
current_date=$(date -u +"%Y-%m-%d")
start_date="2024-12-23"
end_date="2025-01-05"
if [[ "$current_date" < "$start_date" || "$current_date" > "$end_date" ]]; then
echo "outside_holiday=true" >> $GITHUB_ENV
else
echo "outside_holiday=false" >> $GITHUB_ENV
fi
- name: Post comment
if: ${{ env.outside_holiday == 'false' && contains(github.event.issue.labels.*.name, 'bug') }}
run: |
gh issue comment ${{ github.event.issue.number }} --repo ${{ github.repository }} --body "Thank you for your bug report! Our team is will be out of the office for [Christmas and our Global Week of Rest](https://handbook.getdbt.com/docs/time_off#2024-us-holidays), from December 25, 2024, through January 3, 2025.
We will review your issue as soon as possible after returning.
Thank you for your understanding, and happy holidays! 🎄🎉
If you are a customer of dbt Cloud, please contact our Customer Support team via the dbt Cloud web interface or email **support@dbtlabs.com**."
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -28,7 +28,7 @@ permissions:
jobs:
backport:
name: Backport
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
# Only react to merged PRs for security reasons.
# See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target.
if: >

View File

@@ -41,7 +41,7 @@ jobs:
include:
- label: "dependencies"
changie_kind: "Dependencies"
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:

View File

@@ -4,16 +4,12 @@ on:
pull_request:
types: [ opened, reopened, labeled, unlabeled, synchronize ]
paths-ignore: [ '.changes/**', '.github/**', 'tests/**', '**.md', '**.yml' ]
merge_group:
types: [checks_requested]
workflow_dispatch:
permissions:
contents: read
workflow_dispatch:
jobs:
check-artifact-changes:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.labels.*.name, 'artifact_minor_upgrade') }}
steps:
- name: Checkout code
@@ -36,7 +32,7 @@ jobs:
run: |
echo "CI failure: Artifact changes checked in core/dbt/artifacts directory."
echo "Files changed: ${{ steps.check_artifact_changes.outputs.artifacts_changed_files }}"
echo "To bypass this check, confirm that the change is not breaking (https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/artifacts/README.md#breaking-changes) and add the 'artifact_minor_upgrade' label to the PR. Modifications and additions to all fields require updates to https://github.com/dbt-labs/dbt-jsonschema."
echo "To bypass this check, confirm that the change is not breaking (https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/artifacts/README.md#breaking-changes) and add the 'artifact_minor_upgrade' label to the PR."
exit 1
- name: CI check passed

View File

@@ -36,6 +36,6 @@ jobs:
uses: dbt-labs/actions/.github/workflows/open-issue-in-repo.yml@main
with:
issue_repository: "dbt-labs/docs.getdbt.com"
issue_title: "[Core] Docs Changes Needed from ${{ github.event.repository.name }} Issue #${{ github.event.issue.number }}"
issue_body: "At a minimum, update body to include a link to the page on docs.getdbt.com requiring updates and what part(s) of the page you would like to see updated.\n Originating from this issue: https://github.com/dbt-labs/dbt-core/issues/${{ github.event.issue.number }}"
issue_title: "Docs Changes Needed from ${{ github.event.repository.name }} Issue #${{ github.event.issue.number }}"
issue_body: "At a minimum, update body to include a link to the page on docs.getdbt.com requiring updates and what part(s) of the page you would like to see updated."
secrets: inherit

View File

@@ -20,8 +20,6 @@ on:
- "*.latest"
- "releases/*"
pull_request:
merge_group:
types: [checks_requested]
workflow_dispatch:
permissions: read-all
@@ -54,14 +52,13 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.9'
python-version: '3.8'
- name: Install python dependencies
run: |
python -m pip install --user --upgrade pip
python -m pip --version
make dev
make dev_req
mypy --version
dbt --version
@@ -77,7 +74,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13" ]
python-version: [ "3.8", "3.9", "3.10", "3.11", "3.12" ]
env:
TOXENV: "unit"
@@ -114,7 +111,7 @@ jobs:
- name: Upload Unit Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: unit
@@ -142,7 +139,7 @@ jobs:
- name: generate include
id: generate-include
run: |
INCLUDE=('"python-version":"3.9","os":"windows-latest"' '"python-version":"3.9","os":"macos-14"' )
INCLUDE=('"python-version":"3.8","os":"windows-latest"' '"python-version":"3.8","os":"macos-12"' )
INCLUDE_GROUPS="["
for include in ${INCLUDE[@]}; do
for group in $(seq 1 ${{ env.PYTHON_INTEGRATION_TEST_WORKERS }}); do
@@ -154,7 +151,7 @@ jobs:
echo "include=${INCLUDE_GROUPS}"
echo "include=${INCLUDE_GROUPS}" >> $GITHUB_OUTPUT
integration-postgres:
integration:
name: (${{ matrix.split-group }}) integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }}
@@ -164,104 +161,9 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13" ]
os: ["ubuntu-latest"]
python-version: [ "3.8", "3.9", "3.10", "3.11", "3.12" ]
os: [ubuntu-20.04]
split-group: ${{ fromJson(needs.integration-metadata.outputs.split-groups) }}
env:
TOXENV: integration
DBT_INVOCATION_ENV: github-actions
DBT_TEST_USER_1: dbt_test_user_1
DBT_TEST_USER_2: dbt_test_user_2
DBT_TEST_USER_3: dbt_test_user_3
DD_CIVISIBILITY_AGENTLESS_ENABLED: true
DD_API_KEY: ${{ secrets.DATADOG_API_KEY }}
DD_SITE: datadoghq.com
DD_ENV: ci
DD_SERVICE: ${{ github.event.repository.name }}
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: password
POSTGRES_USER: postgres
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- name: Check out the repository
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Run postgres setup script
run: |
./test/setup_db.sh
env:
PGHOST: localhost
PGPORT: 5432
PGPASSWORD: password
- name: Install python tools
run: |
python -m pip install --user --upgrade pip
python -m pip --version
python -m pip install tox
tox --version
- name: Run integration tests
uses: nick-fields/retry@v3
with:
timeout_minutes: 30
max_attempts: 3
command: tox -- --ddtrace
env:
PYTEST_ADDOPTS: ${{ format('--splits {0} --group {1}', env.PYTHON_INTEGRATION_TEST_WORKERS, matrix.split-group) }}
- name: Get current date
if: always()
id: date
run: |
CURRENT_DATE=$(date +'%Y-%m-%dT%H_%M_%S') # no colons allowed for artifacts
echo "date=$CURRENT_DATE" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v4
if: always()
with:
name: logs_${{ matrix.python-version }}_${{ matrix.os }}_${{ matrix.split-group }}_${{ steps.date.outputs.date }}
path: ./logs
- name: Upload Integration Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: integration
integration-mac-windows:
name: (${{ matrix.split-group }}) integration test / python ${{ matrix.python-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 30
needs:
- integration-metadata
strategy:
fail-fast: false
matrix:
split-group: ${{ fromJson(needs.integration-metadata.outputs.split-groups) }}
# this include is where we add the mac and windows os
include: ${{ fromJson(needs.integration-metadata.outputs.include) }}
env:
TOXENV: integration
@@ -284,14 +186,13 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
- name: Set up postgres (linux)
if: runner.os == 'Linux'
uses: ./.github/actions/setup-postgres-linux
- name: Set up postgres (macos)
if: runner.os == 'macOS'
uses: nick-fields/retry@v3
with:
timeout_minutes: 10
max_attempts: 3
command: ./test/setup_db.sh
uses: ./.github/actions/setup-postgres-macos
- name: Set up postgres (windows)
if: runner.os == 'Windows'
@@ -328,7 +229,7 @@ jobs:
- name: Upload Integration Test Coverage to Codecov
if: ${{ matrix.python-version == '3.11' }}
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: integration
@@ -337,10 +238,10 @@ jobs:
if: ${{ always() }}
name: Integration Test Suite
runs-on: ubuntu-latest
needs: [integration-mac-windows, integration-postgres]
needs: integration
steps:
- name: "Integration Tests Failed"
if: ${{ contains(needs.integration-mac-windows.result, 'failure') || contains(needs.integration-mac-windows.result, 'cancelled') || contains(needs.integration-postgres.result, 'failure') || contains(needs.integration-postgres.result, 'cancelled') }}
if: ${{ contains(needs.integration.result, 'failure') || contains(needs.integration.result, 'cancelled') }}
# when this is true the next step won't execute
run: |
echo "::notice title='Integration test suite failed'"
@@ -362,7 +263,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.9'
python-version: '3.8'
- name: Install python dependencies
run: |

View File

@@ -38,7 +38,7 @@ permissions:
jobs:
set-variables:
name: Setting Variables
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
outputs:
cache_key: ${{ steps.variables.outputs.cache_key }}
release_id: ${{ steps.semver.outputs.base-version }}
@@ -72,7 +72,7 @@ jobs:
latest-runner:
name: Build or Fetch Runner
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
needs: [set-variables]
env:
RUSTFLAGS: "-D warnings"
@@ -137,7 +137,7 @@ jobs:
# a successfully tested and built binary from being cached.
needs: [set-variables, latest-runner]
name: Model a release
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- name: '[DEBUG] print variables'
@@ -150,7 +150,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.9"
python-version: "3.8"
- name: Install dbt
run: pip install dbt-postgres==${{ needs.set-variables.outputs.release_id }}
@@ -206,7 +206,7 @@ jobs:
# depends on `model` as a separate job so that the baseline can be committed to more than one branch
# i.e. release branch and main
needs: [set-variables, latest-runner, model]
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
strategy:
matrix:
@@ -253,7 +253,7 @@ jobs:
push: 'origin origin/${{ matrix.target-branch }}'
- name: Create Pull Request
uses: peter-evans/create-pull-request@v7
uses: peter-evans/create-pull-request@v6
with:
author: 'Github Build Bot <buildbot@fishtownanalytics.com>'
base: ${{ matrix.base-branch }}

View File

@@ -31,7 +31,7 @@ env:
jobs:
aggregate-release-data:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
outputs:
version_number: ${{ steps.nightly-release-version.outputs.number }}
@@ -76,7 +76,7 @@ jobs:
echo "name=${{ env.RELEASE_BRANCH }}" >> $GITHUB_OUTPUT
log-outputs-aggregate-release-data:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
needs: [aggregate-release-data]
steps:

View File

@@ -75,7 +75,7 @@ defaults:
jobs:
job-setup:
name: Log Inputs
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
outputs:
starting_sha: ${{ steps.set_sha.outputs.starting_sha }}
steps:
@@ -125,7 +125,7 @@ jobs:
needs: [bump-version-generate-changelog]
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- name: Print variables
@@ -188,7 +188,7 @@ jobs:
# determine if we need to release dbt-core or both dbt-core and dbt-postgres
name: Determine Docker Package
if: ${{ !failure() && !cancelled() }}
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
needs: [pypi-release]
outputs:
matrix: ${{ steps.determine-docker-package.outputs.matrix }}
@@ -247,24 +247,3 @@ jobs:
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}
testing-slack-notification:
# sends notifications to #slackbot-test
name: Testing - Slack Notification
if: ${{ failure() && inputs.test_run && !inputs.nightly_release }}
needs:
[
bump-version-generate-changelog,
build-test-package,
github-release,
pypi-release,
docker-release,
]
uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_TESTING_WEBHOOK_URL }}

View File

@@ -9,21 +9,15 @@
# occur so we want to proactively alert to it.
#
# **when?**
# Only can be run manually
# On pushes to `develop` and release branches. Manual runs are also enabled.
name: Artifact Schema Check
on:
# pull_request:
# types: [ opened, reopened, labeled, unlabeled, synchronize ]
# paths-ignore: [ '.changes/**', '.github/**', 'tests/**', '**.md', '**.yml' ]
pull_request:
types: [ opened, reopened, labeled, unlabeled, synchronize ]
paths-ignore: [ '.changes/**', '.github/**', 'tests/**', '**.md', '**.yml' ]
workflow_dispatch:
inputs:
target_branch:
description: "The branch to check against"
type: string
default: 'main'
required: true
# no special access is needed
permissions: read-all
@@ -36,20 +30,19 @@ env:
jobs:
checking-schemas:
name: "Post-merge schema changes required"
runs-on: ${{ vars.UBUNTU_LATEST }}
name: "Checking schemas"
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.8
- name: Checkout dbt repo
uses: actions/checkout@v4
with:
path: ${{ env.DBT_REPO_DIRECTORY }}
ref: ${{ inputs.target_branch }}
- name: Check for changes in core/dbt/artifacts
# https://github.com/marketplace/actions/paths-changes-filter

View File

@@ -14,8 +14,6 @@ on:
- "*.latest"
- "releases/*"
pull_request:
merge_group:
types: [checks_requested]
workflow_dispatch:
permissions: read-all
@@ -47,7 +45,7 @@ jobs:
# run the performance measurements on the current or default branch
test-schema:
name: Test Log Schema
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
timeout-minutes: 30
needs:
- integration-metadata
@@ -69,24 +67,6 @@ jobs:
DBT_TEST_USER_2: dbt_test_user_2
DBT_TEST_USER_3: dbt_test_user_3
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: password
POSTGRES_USER: postgres
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- name: checkout dev
uses: actions/checkout@v4
@@ -96,7 +76,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.9"
python-version: "3.8"
- name: Install python dependencies
run: |
@@ -105,13 +85,8 @@ jobs:
pip install tox
tox --version
- name: Run postgres setup script
run: |
./test/setup_db.sh
env:
PGHOST: localhost
PGPORT: 5432
PGPASSWORD: password
- name: Set up postgres
uses: ./.github/actions/setup-postgres-linux
- name: ls
run: ls

View File

@@ -27,6 +27,7 @@ on:
description: 'Version of Python to Test Against'
type: choice
options:
- '3.8'
- '3.9'
- '3.10'
- '3.11'
@@ -35,7 +36,7 @@ on:
type: choice
options:
- 'ubuntu-latest'
- 'macos-14'
- 'macos-12'
- 'windows-latest'
num_runs_per_batch:
description: 'Max number of times to run the test per batch. We always run 10 batches.'
@@ -51,7 +52,7 @@ defaults:
jobs:
debug:
runs-on: ${{ vars.UBUNTU_LATEST }}
runs-on: ubuntu-latest
steps:
- name: "[DEBUG] Output Inputs"
run: |
@@ -95,17 +96,13 @@ jobs:
run: make dev
- name: "Set up postgres (linux)"
if: inputs.os == '${{ vars.UBUNTU_LATEST }}'
if: inputs.os == 'ubuntu-latest'
run: make setup-db
# mac and windows don't use make due to limitations with docker with those runners in GitHub
- name: Set up postgres (macos)
if: runner.os == 'macOS'
uses: nick-fields/retry@v3
with:
timeout_minutes: 10
max_attempts: 3
command: ./test/setup_db.sh
- name: "Set up postgres (macos)"
if: inputs.os == 'macos-12'
uses: ./.github/actions/setup-postgres-macos
- name: "Set up postgres (windows)"
if: inputs.os == 'windows-latest'

6
.gitignore vendored
View File

@@ -57,9 +57,6 @@ test.env
makefile.test.env
*.pytest_cache/
# Unit test artifacts
index.html
# Translations
*.mo
@@ -108,6 +105,3 @@ venv/
# poetry
poetry.lock
# asdf
.tool-versions

View File

@@ -1,4 +1,4 @@
[settings]
profile=black
extend_skip_glob=.github/*,third-party-stubs/*,scripts/*
known_first_party=dbt,dbt_adapters,dbt_common,dbt_extractor,dbt_semantic_interfaces
known_first_party=dbt,dbt_adapters,dbt_common,dbt_extractor,dbt_semantic_interface

View File

@@ -1,9 +1,9 @@
# Configuration for pre-commit hooks (see https://pre-commit.com/).
# Eventually the hooks described here will be run as tests before merging each PR.
exclude: ^(core/dbt/docs/build/|core/dbt/common/events/types_pb2.py|core/dbt/adapters/events/adapter_types_pb2.py)
exclude: ^(core/dbt/docs/build/|core/dbt/common/events/types_pb2.py|core/dbt/events/core_types_pb2.py|core/dbt/adapters/events/adapter_types_pb2.py)
# Force all unspecified python hooks to run python 3.9
# Force all unspecified python hooks to run python 3.8
default_language_version:
python: python3
@@ -15,19 +15,16 @@ repos:
args: [--unsafe]
- id: check-json
- id: end-of-file-fixer
exclude: schemas/dbt/manifest/
- id: trailing-whitespace
exclude_types:
- "markdown"
- id: check-case-conflict
- repo: https://github.com/pycqa/isort
# rev must match what's in dev-requirements.txt
rev: 5.13.2
rev: 5.12.0
hooks:
- id: isort
- repo: https://github.com/psf/black
# rev must match what's in dev-requirements.txt
rev: 24.3.0
rev: 22.3.0
hooks:
- id: black
- id: black
@@ -37,7 +34,6 @@ repos:
- "--check"
- "--diff"
- repo: https://github.com/pycqa/flake8
# rev must match what's in dev-requirements.txt
rev: 4.0.1
hooks:
- id: flake8
@@ -45,7 +41,6 @@ repos:
alias: flake8-check
stages: [manual]
- repo: https://github.com/pre-commit/mirrors-mypy
# rev must match what's in dev-requirements.txt
rev: v1.4.1
hooks:
- id: mypy
@@ -69,12 +64,3 @@ repos:
args: [--show-error-codes, --pretty]
files: ^core/dbt/
language: system
- repo: local
hooks:
- id: no_versioned_artifact_resource_imports
name: no_versioned_artifact_resource_imports
entry: python custom-hooks/no_versioned_artifact_resource_imports.py
language: system
files: ^core/dbt/
types: [python]
pass_filenames: true

View File

@@ -6,6 +6,7 @@ Most of the python code in the repository is within the `core/dbt` directory.
- [`single python files`](core/dbt/README.md): A number of individual files, such as 'compilation.py' and 'exceptions.py'
The main subdirectories of core/dbt:
- [`adapters`](core/dbt/adapters/README.md): Define base classes for behavior that is likely to differ across databases
- [`clients`](core/dbt/clients/README.md): Interface with dependencies (agate, jinja) or across operating systems
- [`config`](core/dbt/config/README.md): Reconcile user-supplied configuration from connection profiles, project files, and Jinja macros
- [`context`](core/dbt/context/README.md): Build and expose dbt-specific Jinja functionality
@@ -13,7 +14,7 @@ The main subdirectories of core/dbt:
- [`deps`](core/dbt/deps/README.md): Package installation and dependency resolution
- [`events`](core/dbt/events/README.md): Logging events
- [`graph`](core/dbt/graph/README.md): Produce a `networkx` DAG of project resources, and selecting those resources given user-supplied criteria
- [`include`](core/dbt/include/README.md): Set up the starter project scaffold.
- [`include`](core/dbt/include/README.md): The dbt "global project," which defines default implementations of Jinja2 macros
- [`parser`](core/dbt/parser/README.md): Read project files, validate, construct python objects
- [`task`](core/dbt/task/README.md): Set forth the actions that dbt can perform when invoked
@@ -31,7 +32,7 @@ This is the docs website code. It comes from the dbt-docs repository, and is gen
## Adapters
dbt uses an adapter-plugin pattern to extend support to different databases, warehouses, query engines, etc.
Note: dbt-postgres used to exist in dbt-core but is now in [the dbt-adapters repo](https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-postgres)
Note: dbt-postgres used to exist in dbt-core but is now in [its own repo](https://github.com/dbt-labs/dbt-postgres)
Each adapter is a mix of python, Jinja2, and SQL. The adapter code also makes heavy use of Jinja2 to wrap modular chunks of SQL functionality, define default implementations, and allow plugins to override it.

View File

@@ -10,8 +10,6 @@
For information on prior major and minor releases, see their changelogs:
* [1.10](https://github.com/dbt-labs/dbt-core/blob/1.10.latest/CHANGELOG.md)
* [1.9](https://github.com/dbt-labs/dbt-core/blob/1.9.latest/CHANGELOG.md)
* [1.8](https://github.com/dbt-labs/dbt-core/blob/1.8.latest/CHANGELOG.md)
* [1.7](https://github.com/dbt-labs/dbt-core/blob/1.7.latest/CHANGELOG.md)
* [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)

View File

@@ -170,9 +170,9 @@ Finally, you can also run a specific test or group of tests using [`pytest`](htt
```sh
# run all unit tests in a file
python3 -m pytest tests/unit/test_invocation_id.py
python3 -m pytest tests/unit/test_base_column.py
# run a specific unit test
python3 -m pytest tests/unit/test_invocation_id.py::TestInvocationId::test_invocation_id
python3 -m pytest tests/unit/test_base_column.py::TestNumericType::test__numeric_type
# run specific Postgres functional tests
python3 -m pytest tests/functional/sources
```
@@ -224,9 +224,7 @@ Code can be merged into the current development branch `main` by opening a pull
Automated tests run via GitHub Actions. If you're a first-time contributor, all tests (including code checks and unit tests) will require a maintainer to approve. Changes in the `dbt-core` repository trigger integration tests against Postgres. dbt Labs also provides CI environments in which to test changes to other adapters, triggered by PRs in those adapters' repositories, as well as periodic maintenance checks of each adapter in concert with the latest `dbt-core` code changes.
We require signed git commits. See docs [here](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) for setting up code signing.
Once all tests are passing, all comments are resolved, and your PR has been approved, a `dbt-core` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada:
Once all tests are passing and your PR has been approved, a `dbt-core` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada:
## Troubleshooting Tips

View File

@@ -33,6 +33,9 @@ RUN apt-get update \
python-is-python3 \
python-dev-is-python3 \
python3-pip \
python3.8 \
python3.8-dev \
python3.8-venv \
python3.9 \
python3.9-dev \
python3.9-venv \
@@ -50,7 +53,7 @@ RUN curl -LO https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_V
&& tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
&& rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
RUN pip3 install -U tox wheel six setuptools pre-commit
RUN pip3 install -U tox wheel six setuptools
# These args are passed in via docker-compose, which reads then from the .env file.
# On Linux, run `make .env` to create the .env file for the current user.

View File

@@ -35,7 +35,7 @@ dev_req: ## Installs dbt-* packages in develop mode along with only development
.PHONY: dev
dev: dev_req ## Installs dbt-* packages in develop mode along with development dependencies and pre-commit.
@\
$(DOCKER_CMD) pre-commit install
pre-commit install
.PHONY: dev-uninstall
dev-uninstall: ## Uninstall all packages in venv except for build tools
@@ -43,6 +43,10 @@ dev-uninstall: ## Uninstall all packages in venv except for build tools
pip freeze | grep -v "^-e" | cut -d "@" -f1 | xargs pip uninstall -y; \
pip uninstall -y dbt-core
.PHONY: core_proto_types
core_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/core_types.proto
.PHONY: mypy
mypy: .env ## Runs mypy against staged changes for static type checking.
@\
@@ -140,7 +144,3 @@ help: ## Show this help message.
@echo
@echo 'options:'
@echo 'use USE_DOCKER=true to run target in a docker container'
.PHONY: json_schema
json_schema: ## Update generated JSON schema using code changes.
scripts/collect-artifact-schema.py --path schemas

View File

@@ -1,41 +1,13 @@
ignore:
- ".github"
- ".changes"
coverage:
status:
project:
default:
target: auto
threshold: 0.1% # Reduce noise by ignoring rounding errors in coverage drops
informational: true
patch:
default:
target: auto
threshold: 80%
informational: true
comment:
layout: "header, diff, flags, components" # show component info in the PR comment
component_management:
default_rules: # default rules that will be inherited by all components
statuses:
- type: project # in this case every component that doens't have a status defined will have a project type one
target: auto
threshold: 0.1%
- type: patch
target: 80%
individual_components:
- component_id: unittests
name: "Unit Tests"
flag_regexes:
- "unit"
statuses:
- type: patch
target: 80%
threshold: 5%
- component_id: integrationtests
name: "Integration Tests"
flag_regexes:
- "integration"

View File

@@ -1,3 +1,3 @@
recursive-include dbt/include *.py *.sql *.yml *.html *.md .gitkeep .gitignore *.json
recursive-include dbt/include *.py *.sql *.yml *.html *.md .gitkeep .gitignore
include dbt/py.typed
recursive-include dbt/task/docs *.html

View File

@@ -1,26 +0,0 @@
# type: ignore
"""Shim to allow support for both Pydantic 1 and Pydantic 2.
dbt-core must support both major versions of Pydantic because dbt-core users might be using an environment with
either version, and we can't restrict them to one or the other. Here, we essentially import all Pydantic objects
from version 1 that we use. Throughout the repo, we import these objects from this file instead of from Pydantic
directly, meaning that we essentially only use Pydantic 1 in dbt-core currently, but without forcing that restriction
on dbt users. The development environment for this repo should be pinned to Pydantic 1 to ensure devs get appropriate
type hints.
"""
from importlib.metadata import version
pydantic_version = version("pydantic")
# Pydantic uses semantic versioning, i.e. <major>.<minor>.<patch>, and we need to know the major
pydantic_major = pydantic_version.split(".")[0]
if pydantic_major == "1":
from pydantic import BaseSettings # noqa: F401
elif pydantic_major == "2":
from pydantic.v1 import BaseSettings # noqa: F401
else:
raise RuntimeError(
f"Currently only pydantic 1 and 2 are supported, found pydantic {pydantic_version}"
)

Some files were not shown because too many files have changed in this diff Show More