Merge remote-tracking branch 'origin/releases/0.21.1' into 0.21.latest

* Add get_where_subquery to test namespace (#4197)

* Add get_where_subquery to test namespace

* Add integration test

* Fix test, add comment, smarter approach

* Fix unit tests

* Add changelog entry

* Fixup changelog

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.21.0b2
+current_version = 0.21.1
 parse = (?P<major>\d+)
 	\.(?P<minor>\d+)
 	\.(?P<patch>\d+)

.github/workflows/integration.yml

@@ -182,7 +182,7 @@ jobs:
 
       - name: Install python dependencies
         run: |
-          pip install --upgrade pip
+          pip install --user --upgrade pip
           pip install tox
           pip --version
           tox --version

.github/workflows/main.yml

@@ -61,7 +61,7 @@ jobs:
 
       - name: Install python dependencies
         run: |
-          pip install --upgrade pip
+          pip install --user --upgrade pip
           pip install tox
           pip --version
           tox --version
@@ -96,7 +96,7 @@ jobs:
 
       - name: Install python dependencies
         run: |
-          pip install --upgrade pip
+          pip install --user --upgrade pip
           pip install tox
           pip --version
           tox --version
@@ -133,7 +133,7 @@ jobs:
 
       - name: Install python dependencies
         run: |
-          pip install --upgrade pip
+          pip install --user --upgrade pip
           pip install --upgrade setuptools wheel twine check-wheel-contents
           pip --version
 
@@ -177,7 +177,7 @@ jobs:
 
       - name: Install python dependencies
         run: |
-          pip install --upgrade pip
+          pip install --user --upgrade pip
           pip install --upgrade wheel
           pip --version
 

CHANGELOG.md

@@ -1,10 +1,52 @@
-## dbt 0.21.0 (Release TBD)
+## dbt 0.21.1 (November 29, 2021)
 
-## dbt 0.21.0b2 (August 19, 2021)
+## dbt 0.21.1rc2 (November 15, 2021)
 
 
+### Fixes
+- Add `get_where_subquery` to test macro namespace, fixing custom generic tests that rely on introspecting the `model` arg at parse time ([#4195](https://github.com/dbt-labs/dbt/issues/4195), [#4197](https://github.com/dbt-labs/dbt/pull/4197))
+
+## dbt 0.21.1rc1 (November 03, 2021)
+
+
+### Fixes
+- Performance: Use child_map to find tests for nodes in resolve_graph ([#4012](https://github.com/dbt-labs/dbt/issues/4012), [#4057](https://github.com/dbt-labs/dbt/pull/4057))
+- Switch `unique_field` from abstractproperty to optional property. Add docstring ([#4025](https://github.com/dbt-labs/dbt-core/issues/4025), [#4028](https://github.com/dbt-labs/dbt-core/pull/4028))
+- Fix multiple partial parsing errors ([#3996](https://github.com/dbt-labs/dbt/issues/3006), [#4020](https://github.com/dbt-labs/dbt/pull/4018))
+- Include only relational nodes in `database_schema_set` ([#4063](https://github.com/dbt-labs/dbt-core/issues/4063), [#4077](https://github.com/dbt-labs/dbt-core/pull/4077))
+- Fixed bug with `error_if` test option ([#4070](https://github.com/dbt-labs/dbt-core/pull/4070))
+- Added support for tests on databases that lack real boolean types. ([#4084](https://github.com/dbt-labs/dbt-core/issues/4084))
+- Prefer macros defined in the project over the ones in a package by default ([#4106](https://github.com/dbt-labs/dbt-core/issues/4106), [#4114](https://github.com/dbt-labs/dbt-core/pull/4114))
+- Scrub secrets coming from `CommandError`s so they don't get exposed in logs. ([#4138](https://github.com/dbt-labs/dbt-core/pull/4138))
+- Syntax fix in `alter_relation_add_remove_columns` if only removing columns in `on_schema_change: sync_all_columns` ([#4147](https://github.com/dbt-labs/dbt-core/issues/4147))
+- Increase performance of graph subset selection ([#4135](https://github.com/dbt-labs/dbt-core/issues/4135),[#4155](https://github.com/dbt-labs/dbt-core/pull/4155))
+- Add downstream test edges for `build` task _only_. Restore previous graph construction, compilation performance, and node selection behavior (`test+`) for all other tasks ([#4135](https://github.com/dbt-labs/dbt-core/issues/4135), [#4143](https://github.com/dbt-labs/dbt-core/pull/4143))
+- Don't require a strict/proper subset when adding testing edges to specialized graph for `build` ([#4158](https://github.com/dbt-labs/dbt-core/issues/4135), [#4158](https://github.com/dbt-labs/dbt-core/pull/4160))
+- Capping `google-api-core` to version `1.31.3` due to `protobuf` dependency conflict ([#4192](https://github.com/dbt-labs/dbt-core/pull/4192))
+
+Contributors:
+- [@ljhopkins2](https://github.com/ljhopkins2) ([#4077](https://github.com/dbt-labs/dbt-core/pull/4077))
+- [@JCZuurmond](https://github.com/jczuurmond) ([#4114](https://github.com/dbt-labs/dbt-core/pull/4114))
+
+## dbt 0.21.0 (October 04, 2021)
+
+
+## dbt 0.21.0rc2 (September 27, 2021)
+
+
+### Fixes
+- Fix batching for large seeds on Snowflake ([#3941](https://github.com/dbt-labs/dbt/issues/3941), [#3942](https://github.com/dbt-labs/dbt/pull/3942))
+- Avoid infinite recursion in `state:modified.macros` check ([#3904](https://github.com/dbt-labs/dbt/issues/3904), [#3957](https://github.com/dbt-labs/dbt/pull/3957))
+- Cast log messages to strings before scrubbing of prefixed env vars ([#3971](https://github.com/dbt-labs/dbt/issues/3971), [#3972](https://github.com/dbt-labs/dbt/pull/3972))
+
+### Under the hood
+- Bump artifact schema versions for 0.21.0 ([#3945](https://github.com/dbt-labs/dbt/pull/3945))
+
+## dbt 0.21.0rc1 (September 20, 2021)
+
 ### Features
 
+- Make `--models` and `--select` synonyms, except for `ls` (to preserve existing behavior) ([#3210](https://github.com/dbt-labs/dbt/pull/3210), [#3791](https://github.com/dbt-labs/dbt/pull/3791))
 - Experimental parser now detects macro overrides of ref, source, and config builtins. ([#3581](https://github.com/dbt-labs/dbt/issues/3866), [#3582](https://github.com/dbt-labs/dbt/pull/3877))
 - Add connect_timeout profile configuration for Postgres and Redshift adapters. ([#3581](https://github.com/dbt-labs/dbt/issues/3581), [#3582](https://github.com/dbt-labs/dbt/pull/3582))
 - Enhance BigQuery copy materialization ([#3570](https://github.com/dbt-labs/dbt/issues/3570), [#3606](https://github.com/dbt-labs/dbt/pull/3606)):
@@ -16,6 +58,7 @@
 - Added default field in the `selectors.yml` to allow user to define default selector ([#3448](https://github.com/dbt-labs/dbt/issues/3448), [#3875](https://github.com/dbt-labs/dbt/issues/3875), [#3892](https://github.com/dbt-labs/dbt/issues/3892))
 - Added timing and thread information to sources.json artifact ([#3804](https://github.com/dbt-labs/dbt/issues/3804), [#3894](https://github.com/dbt-labs/dbt/pull/3894))
 - Update cli and rpc flags for the `build` task to align with other commands (`--resource-type`, `--store-failures`) ([#3596](https://github.com/dbt-labs/dbt/issues/3596), [#3884](https://github.com/dbt-labs/dbt/pull/3884))
+- Log tests that are not indirectly selected. Add `--greedy` flag to `test`, `list`, `build` and `greedy` property in yaml selectors ([#3723](https://github.com/dbt-labs/dbt/pull/3723), [#3833](https://github.com/dbt-labs/dbt/pull/3833))
 
 ### Fixes
 
@@ -29,13 +72,13 @@
 
 ### Under the hood
 
-- Use GitHub Actions for CI ([#3688](https://github.com/dbt-labs/dbt/issues/3688), [#3669](https://github.com/dbt-labs/dbt/pull/3669))
 - Better dbt hub registry packages version logging that prompts the user for upgrades to relevant packages ([#3560](https://github.com/dbt-labs/dbt/issues/3560), [#3763](https://github.com/dbt-labs/dbt/issues/3763), [#3759](https://github.com/dbt-labs/dbt/pull/3759))
 - Allow the default seed macro's SQL parameter, `%s`, to be replaced by dispatching a new macro, `get_binding_char()`. This enables adapters with parameter marker characters such as `?` to not have to override `basic_load_csv_rows`. ([#3622](https://github.com/dbt-labs/dbt/issues/3622), [#3623](https://github.com/dbt-labs/dbt/pull/3623))
 - Alert users on package rename ([hub.getdbt.com#180](https://github.com/dbt-labs/hub.getdbt.com/issues/810), [#3825](https://github.com/dbt-labs/dbt/pull/3825))
 - Add `adapter_unique_id` to invocation context in anonymous usage tracking, to better understand dbt adoption ([#3713](https://github.com/dbt-labs/dbt/issues/3713), [#3796](https://github.com/dbt-labs/dbt/issues/3796))
 - Specify `macro_namespace = 'dbt'` for all dispatched macros in the global project, making it possible to dispatch to macro implementations defined in packages. Dispatch `generate_schema_name` and `generate_alias_name` ([#3456](https://github.com/dbt-labs/dbt/issues/3456), [#3851](https://github.com/dbt-labs/dbt/issues/3851))
-- Retry transient GitHub failures during download ([#3729](https://github.com/dbt-labs/dbt/pull/3729))
+- Retry transient GitHub failures during download ([#3546](https://github.com/dbt-labs/dbt/pull/3546), [#3729](https://github.com/dbt-labs/dbt/pull/3729))
+- Don't reload and validate schema files if they haven't changed ([#3563](https://github.com/dbt-labs/dbt/issues/3563), [#3888](https://github.com/dbt-labs/dbt/issues/3888))
 
 Contributors:
 
@@ -44,7 +87,7 @@ Contributors:
 - [@dbrtly](https://github.com/dbrtly) ([#3834](https://github.com/dbt-labs/dbt/pull/3834))
 - [@swanderz](https://github.com/swanderz) [#3623](https://github.com/dbt-labs/dbt/pull/3623)
 - [@JasonGluck](https://github.com/JasonGluck) ([#3582](https://github.com/dbt-labs/dbt/pull/3582))
-- [@joellabes](https://github.com/joellabes) ([#3669](https://github.com/dbt-labs/dbt/pull/3669))
+- [@joellabes](https://github.com/joellabes) ([#3669](https://github.com/dbt-labs/dbt/pull/3669), [#3833](https://github.com/dbt-labs/dbt/pull/3833))
 - [@juma-adoreme](https://github.com/juma-adoreme) ([#3838](https://github.com/dbt-labs/dbt/pull/3838))
 - [@annafil](https://github.com/annafil) ([#3825](https://github.com/dbt-labs/dbt/pull/3825))
 - [@AndreasTA-AW](https://github.com/AndreasTA-AW) ([#3691](https://github.com/dbt-labs/dbt/pull/3691))
@@ -52,6 +95,7 @@ Contributors:
 - [@TeddyCr](https://github.com/TeddyCr) ([#3448](https://github.com/dbt-labs/dbt/pull/3865))
 - [@sdebruyn](https://github.com/sdebruyn) ([#3906](https://github.com/dbt-labs/dbt/pull/3906))
 
+
 ## dbt 0.21.0b2 (August 19, 2021)
 
 ### Features
@@ -67,7 +111,6 @@ Contributors:
 ### Under the hood
 
 - Add `build` RPC method, and a subset of flags for `build` task ([#3595](https://github.com/dbt-labs/dbt/issues/3595), [#3674](https://github.com/dbt-labs/dbt/pull/3674))
-- Get more information on partial parsing version mismatches ([#3757](https://github.com/dbt-labs/dbt/issues/3757), [#3758](https://github.com/dbt-labs/dbt/pull/3758))
 
 ## dbt 0.21.0b1 (August 03, 2021)
 
@@ -118,6 +161,7 @@ Contributors:
 - Better error handling for BigQuery job labels that are too long. ([#3612](https://github.com/dbt-labs/dbt/pull/3612), [#3703](https://github.com/dbt-labs/dbt/pull/3703))
 - Get more information on partial parsing version mismatches ([#3757](https://github.com/dbt-labs/dbt/issues/3757), [#3758](https://github.com/dbt-labs/dbt/pull/3758))
 - Switch to full reparse on partial parsing exceptions. Log and report exception information. ([#3725](https://github.com/dbt-labs/dbt/issues/3725), [#3733](https://github.com/dbt-labs/dbt/pull/3733))
+- Use GitHub Actions for CI ([#3688](https://github.com/dbt-labs/dbt/issues/3688), [#3669](https://github.com/dbt-labs/dbt/pull/3669))
 
 ### Fixes
 

core/dbt/compilation.py

@@ -111,12 +111,13 @@ def _get_tests_for_node(manifest: Manifest, unique_id: UniqueID) -> List[UniqueI
     """ Get a list of tests that depend on the node with the
     provided unique id """
 
-    return [
-        node.unique_id
-        for _, node in manifest.nodes.items()
-        if node.resource_type == NodeType.Test and
-        unique_id in node.depends_on_nodes
-    ]
+    tests = []
+    if unique_id in manifest.child_map:
+        for child_unique_id in manifest.child_map[unique_id]:
+            if child_unique_id.startswith('test.'):
+                tests.append(child_unique_id)
+
+    return tests
 
 
 class Linker:
@@ -417,7 +418,7 @@ class Compiler:
             else:
                 dependency_not_found(node, dependency)
 
-    def link_graph(self, linker: Linker, manifest: Manifest):
+    def link_graph(self, linker: Linker, manifest: Manifest, add_test_edges: bool = False):
         for source in manifest.sources.values():
             linker.add_node(source.unique_id)
         for node in manifest.nodes.values():
@@ -430,27 +431,29 @@ class Compiler:
         if cycle:
             raise RuntimeError("Found a cycle: {}".format(cycle))
 
-        self.resolve_graph(linker, manifest)
+        if add_test_edges:
+            manifest.build_parent_and_child_maps()
+            self.add_test_edges(linker, manifest)
 
-    def resolve_graph(self, linker: Linker, manifest: Manifest) -> None:
+    def add_test_edges(self, linker: Linker, manifest: Manifest) -> None:
         """ This method adds additional edges to the DAG. For a given non-test
         executable node, add an edge from an upstream test to the given node if
-        the set of nodes the test depends on is a proper/strict subset of the
-        upstream nodes for the given node. """
+        the set of nodes the test depends on is a subset of the upstream nodes
+        for the given node. """
 
         # Given a graph:
         # model1 --> model2 --> model3
-        #   |         |
-        #   |        \/
-        #  \/      test 2
+        #   |             |
+        #   |            \/
+        #  \/          test 2
         # test1
         #
         # Produce the following graph:
         # model1 --> model2 --> model3
-        #   |         |         /\ /\
-        #   |        \/         |  |
-        #  \/      test2 -------   |
-        # test1 -------------------
+        #   |       /\    |      /\ /\
+        #   |       |    \/      |  |
+        #  \/       |  test2 ----|  |
+        # test1 ----|---------------|
 
         for node_id in linker.graph:
             # If node is executable (in manifest.nodes) and does _not_
@@ -488,21 +491,19 @@ class Compiler:
                     )
 
                     # If the set of nodes that an upstream test depends on
-                    # is a proper (or strict) subset of all upstream nodes of
-                    # the current node, add an edge from the upstream test
-                    # to the current node. Must be a proper/strict subset to
-                    # avoid adding a circular dependency to the graph.
-                    if (test_depends_on < upstream_nodes):
+                    # is a subset of all upstream nodes of the current node,
+                    # add an edge from the upstream test to the current node.
+                    if (test_depends_on.issubset(upstream_nodes)):
                         linker.graph.add_edge(
                             upstream_test,
                             node_id
                         )
 
-    def compile(self, manifest: Manifest, write=True) -> Graph:
+    def compile(self, manifest: Manifest, write=True, add_test_edges=False) -> Graph:
         self.initialize()
         linker = Linker()
 
-        self.link_graph(linker, manifest)
+        self.link_graph(linker, manifest, add_test_edges)
 
         stats = _generate_stats(manifest)
 

core/dbt/context/providers.py

@@ -144,7 +144,7 @@ class BaseDatabaseWrapper:
         elif isinstance(namespace, str):
             search_packages = self._adapter.config.get_macro_search_order(namespace)
             if not search_packages and namespace in self._adapter.config.dependencies:
-                search_packages = [namespace]
+                search_packages = [self.config.project_name, namespace]
             if not search_packages:
                 raise CompilationException(
                     f'In adapter.dispatch, got a string packages argument '
@@ -164,10 +164,10 @@ class BaseDatabaseWrapper:
                     macro = self._namespace.get_from_package(
                         package_name, search_name
                     )
-                except CompilationException as exc:
-                    raise CompilationException(
-                        f'In dispatch: {exc.msg}',
-                    ) from exc
+                except CompilationException:
+                    # Only raise CompilationException if macro is not found in
+                    # any package
+                    macro = None
 
                 if package_name is None:
                     attempts.append(search_name)
@@ -1439,8 +1439,13 @@ class TestContext(ProviderContext):
     # 'depends_on.macros' by using the TestMacroNamespace
     def _build_test_namespace(self):
         depends_on_macros = []
+        # all generic tests use a macro named 'get_where_subquery' to wrap 'model' arg
+        # see generic_test_builders.build_model_str
+        get_where_subquery = self.macro_resolver.macros_by_name.get('get_where_subquery')
+        if get_where_subquery:
+            depends_on_macros.append(get_where_subquery.unique_id)
         if self.model.depends_on and self.model.depends_on.macros:
-            depends_on_macros = self.model.depends_on.macros
+            depends_on_macros.extend(self.model.depends_on.macros)
         lookup_macros = depends_on_macros.copy()
         for macro_unique_id in lookup_macros:
             lookup_macro = self.macro_resolver.macros.get(macro_unique_id)

core/dbt/contracts/connection.py

@@ -128,10 +128,14 @@ class Credentials(
             'type not implemented for base credentials class'
         )
 
-    @abc.abstractproperty
+    @property
     def unique_field(self) -> str:
+        """Hashed and included in anonymous telemetry to track adapter adoption.
+        Return the field from Credentials that can uniquely identify
+        one team/organization using this adapter
+        """
         raise NotImplementedError(
-            'type not implemented for base credentials class'
+            'unique_field not implemented for base credentials class'
         )
 
     def hashed_unique_field(self) -> str:

core/dbt/contracts/graph/manifest.py

@@ -169,6 +169,43 @@ class RefableLookup(dbtClassMixin):
         return manifest.nodes[unique_id]
 
 
+class DisabledLookup(dbtClassMixin):
+    # model, seed, snapshot
+    _lookup_types: ClassVar[set] = set(NodeType.refable())
+
+    def __init__(self, manifest: 'Manifest'):
+        self.storage: Dict[str, Dict[PackageName, List[Any]]] = {}
+        self.populate(manifest)
+
+    def populate(self, manifest):
+        for node in manifest.disabled:
+            self.add_node(node)
+        for node in list(chain.from_iterable(manifest._disabled.values())):
+            self.add_node(node)
+
+    def add_node(self, node: ManifestNode):
+        if node.resource_type in self._lookup_types:
+            if node.name not in self.storage:
+                self.storage[node.name] = {}
+            if node.package_name not in self.storage[node.name]:
+                self.storage[node.name][node.package_name] = []
+            self.storage[node.name][node.package_name].append(node)
+
+    # This should return a list of disabled nodes
+    def find(self, key, package: PackageName):
+        if key not in self.storage:
+            return None
+
+        pkg_dct: Mapping[PackageName, List[ManifestNode]] = self.storage[key]
+
+        if not pkg_dct:
+            return None
+        elif package in pkg_dct:
+            return pkg_dct[package]
+        else:
+            return None
+
+
 class AnalysisLookup(RefableLookup):
     _lookup_types: ClassVar[set] = set([NodeType.Analysis])
 
@@ -568,7 +605,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
     state_check: ManifestStateCheck = field(default_factory=ManifestStateCheck)
     # Moved from the ParseResult object
     source_patches: MutableMapping[SourceKey, SourcePatch] = field(default_factory=dict)
-    # following is from ParseResult
+    # following contains new disabled nodes until parsing is finished. This changes in 1.0.0
     _disabled: MutableMapping[str, List[CompileResultNode]] = field(default_factory=dict)
     _doc_lookup: Optional[DocLookup] = field(
         default=None, metadata={'serialize': lambda x: None, 'deserialize': lambda x: None}
@@ -579,6 +616,9 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
     _ref_lookup: Optional[RefableLookup] = field(
         default=None, metadata={'serialize': lambda x: None, 'deserialize': lambda x: None}
     )
+    _disabled_lookup: Optional[DisabledLookup] = field(
+        default=None, metadata={'serialize': lambda x: None, 'deserialize': lambda x: None}
+    )
     _analysis_lookup: Optional[AnalysisLookup] = field(
         default=None, metadata={'serialize': lambda x: None, 'deserialize': lambda x: None}
     )
@@ -652,6 +692,15 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
             }
         }
 
+    def build_disabled_by_file_id(self):
+        disabled_by_file_id = {}
+        for node in self.disabled:
+            disabled_by_file_id[node.file_id] = node
+        for node_list in self._disabled.values():
+            for node in node_list:
+                disabled_by_file_id[node.file_id] = node
+        return disabled_by_file_id
+
     def find_disabled_by_name(
         self, name: str, package: Optional[str] = None
     ) -> Optional[ManifestNode]:
@@ -822,6 +871,15 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
     def rebuild_ref_lookup(self):
         self._ref_lookup = RefableLookup(self)
 
+    @property
+    def disabled_lookup(self) -> DisabledLookup:
+        if self._disabled_lookup is None:
+            self._disabled_lookup = DisabledLookup(self)
+        return self._disabled_lookup
+
+    def rebuild_disabled_lookup(self):
+        self._disabled_lookup = DisabledLookup(self)
+
     @property
     def analysis_lookup(self) -> AnalysisLookup:
         if self._analysis_lookup is None:
@@ -1054,6 +1112,8 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin):
             self._doc_lookup,
             self._source_lookup,
             self._ref_lookup,
+            self._disabled_lookup,
+            self._analysis_lookup
         )
         return self.__class__, args
 
@@ -1071,7 +1131,7 @@ AnyManifest = Union[Manifest, MacroManifest]
 
 
 @dataclass
-@schema_version('manifest', 2)
+@schema_version('manifest', 3)
 class WritableManifest(ArtifactMixin):
     nodes: Mapping[UniqueID, ManifestNode] = field(
         metadata=dict(description=(

core/dbt/contracts/results.py

@@ -185,7 +185,7 @@ class RunExecutionResult(
 
 
 @dataclass
-@schema_version('run-results', 2)
+@schema_version('run-results', 3)
 class RunResultsArtifact(ExecutionResult, ArtifactMixin):
     results: Sequence[RunResultOutput]
     args: Dict[str, Any] = field(default_factory=dict)
@@ -369,7 +369,7 @@ class FreshnessResult(ExecutionResult):
 
 
 @dataclass
-@schema_version('sources', 1)
+@schema_version('sources', 2)
 class FreshnessExecutionResultArtifact(
     ArtifactMixin,
     VersionedSchema,

core/dbt/exceptions.py

@@ -3,6 +3,7 @@ import functools
 from typing import NoReturn, Optional, Mapping, Any
 
 from dbt.logger import GLOBAL_LOGGER as logger
+from dbt.logger import get_secret_env
 from dbt.node_types import NodeType
 from dbt import flags
 from dbt.ui import line_wrap_message
@@ -390,6 +391,8 @@ class CommandError(RuntimeException):
         super().__init__(message)
         self.cwd = cwd
         self.cmd = cmd
+        for secret in get_secret_env():
+            self.cmd = str(self.cmd).replace(secret, "*****")
         self.args = (cwd, cmd, message)
 
     def __str__(self):
@@ -466,6 +469,15 @@ def invalid_type_error(method_name, arg_name, got_value, expected_type,
                          got_value=got_value, got_type=got_type))
 
 
+def invalid_bool_error(got_value, macro_name) -> NoReturn:
+    """Raise a CompilationException when an macro expects a boolean but gets some
+    other value.
+    """
+    msg = ("Macro '{macro_name}' returns '{got_value}'.  It is not type 'bool' "
+           "and cannot not be converted reliably to a bool.")
+    raise_compiler_error(msg.format(macro_name=macro_name, got_value=got_value))
+
+
 def ref_invalid_args(model, args) -> NoReturn:
     raise_compiler_error(
         "ref() takes at most two arguments ({} given)".format(len(args)),

core/dbt/flags.py

@@ -18,6 +18,7 @@ WRITE_JSON = None
 PARTIAL_PARSE = None
 USE_COLORS = None
 STORE_FAILURES = None
+GREEDY = None
 
 
 def env_set_truthy(key: str) -> Optional[str]:
@@ -56,7 +57,7 @@ MP_CONTEXT = _get_context()
 def reset():
     global STRICT_MODE, FULL_REFRESH, USE_CACHE, WARN_ERROR, TEST_NEW_PARSER, \
         USE_EXPERIMENTAL_PARSER, WRITE_JSON, PARTIAL_PARSE, MP_CONTEXT, USE_COLORS, \
-        STORE_FAILURES
+        STORE_FAILURES, GREEDY
 
     STRICT_MODE = False
     FULL_REFRESH = False
@@ -69,12 +70,13 @@ def reset():
     MP_CONTEXT = _get_context()
     USE_COLORS = True
     STORE_FAILURES = False
+    GREEDY = False
 
 
 def set_from_args(args):
     global STRICT_MODE, FULL_REFRESH, USE_CACHE, WARN_ERROR, TEST_NEW_PARSER, \
         USE_EXPERIMENTAL_PARSER, WRITE_JSON, PARTIAL_PARSE, MP_CONTEXT, USE_COLORS, \
-        STORE_FAILURES
+        STORE_FAILURES, GREEDY
 
     USE_CACHE = getattr(args, 'use_cache', USE_CACHE)
 
@@ -99,6 +101,7 @@ def set_from_args(args):
         USE_COLORS = use_colors_override
 
     STORE_FAILURES = getattr(args, 'store_failures', STORE_FAILURES)
+    GREEDY = getattr(args, 'greedy', GREEDY)
 
 
 # initialize everything to the defaults on module load

core/dbt/graph/cli.py

@@ -1,4 +1,5 @@
 # special support for CLI argument parsing.
+from dbt import flags
 import itertools
 from dbt.clients.yaml_helper import yaml, Loader, Dumper  # noqa: F401
 
@@ -66,7 +67,7 @@ def parse_union_from_default(
 def parse_difference(
     include: Optional[List[str]], exclude: Optional[List[str]]
 ) -> SelectionDifference:
-    included = parse_union_from_default(include, DEFAULT_INCLUDES)
+    included = parse_union_from_default(include, DEFAULT_INCLUDES, greedy=bool(flags.GREEDY))
     excluded = parse_union_from_default(exclude, DEFAULT_EXCLUDES, greedy=True)
     return SelectionDifference(components=[included, excluded])
 
@@ -180,7 +181,7 @@ def parse_union_definition(definition: Dict[str, Any]) -> SelectionSpec:
     union_def_parts = _get_list_dicts(definition, 'union')
     include, exclude = _parse_include_exclude_subdefs(union_def_parts)
 
-    union = SelectionUnion(components=include)
+    union = SelectionUnion(components=include, greedy_warning=False)
 
     if exclude is None:
         union.raw = definition
@@ -188,7 +189,8 @@ def parse_union_definition(definition: Dict[str, Any]) -> SelectionSpec:
     else:
         return SelectionDifference(
             components=[union, exclude],
-            raw=definition
+            raw=definition,
+            greedy_warning=False
         )
 
 
@@ -197,7 +199,7 @@ def parse_intersection_definition(
 ) -> SelectionSpec:
     intersection_def_parts = _get_list_dicts(definition, 'intersection')
     include, exclude = _parse_include_exclude_subdefs(intersection_def_parts)
-    intersection = SelectionIntersection(components=include)
+    intersection = SelectionIntersection(components=include, greedy_warning=False)
 
     if exclude is None:
         intersection.raw = definition
@@ -205,7 +207,8 @@ def parse_intersection_definition(
     else:
         return SelectionDifference(
             components=[intersection, exclude],
-            raw=definition
+            raw=definition,
+            greedy_warning=False
         )
 
 
@@ -239,7 +242,7 @@ def parse_dict_definition(definition: Dict[str, Any]) -> SelectionSpec:
     if diff_arg is None:
         return base
     else:
-        return SelectionDifference(components=[base, diff_arg])
+        return SelectionDifference(components=[base, diff_arg], greedy_warning=False)
 
 
 def parse_from_definition(

core/dbt/graph/graph.py

@@ -1,6 +1,7 @@
 from typing import (
     Set, Iterable, Iterator, Optional, NewType
 )
+from itertools import product
 import networkx as nx  # type: ignore
 
 from dbt.exceptions import InternalException
@@ -77,17 +78,26 @@ class Graph:
             successors.update(self.graph.successors(node))
         return successors
 
-    def get_subset_graph(self, selected: Iterable[UniqueId]) -> 'Graph':
+    def get_subset_graph(self, selected: Iterable[UniqueId]) -> "Graph":
         """Create and return a new graph that is a shallow copy of the graph,
         but with only the nodes in include_nodes. Transitive edges across
         removed nodes are preserved as explicit new edges.
         """
-        new_graph = nx.algorithms.transitive_closure(self.graph)
 
+        new_graph = self.graph.copy()
         include_nodes = set(selected)
 
         for node in self:
             if node not in include_nodes:
+                source_nodes = [x for x, _ in new_graph.in_edges(node)]
+                target_nodes = [x for _, x in new_graph.out_edges(node)]
+
+                new_edges = product(source_nodes, target_nodes)
+                non_cyclic_new_edges = [
+                    (source, target) for source, target in new_edges if source != target
+                ]  # removes cyclic refs
+
+                new_graph.add_edges_from(non_cyclic_new_edges)
                 new_graph.remove_node(node)
 
         for node in include_nodes:
@@ -96,6 +106,7 @@ class Graph:
                     "Couldn't find model '{}' -- does it exist or is "
                     "it disabled?".format(node)
                 )
+
         return Graph(new_graph)
 
     def subgraph(self, nodes: Iterable[UniqueId]) -> 'Graph':

core/dbt/graph/selector.py

@@ -1,4 +1,3 @@
-
 from typing import Set, List, Optional, Tuple
 
 from .graph import Graph, UniqueId
@@ -30,6 +29,24 @@ def alert_non_existence(raw_spec, nodes):
         )
 
 
+def alert_unused_nodes(raw_spec, node_names):
+    summary_nodes_str = ("\n  - ").join(node_names[:3])
+    debug_nodes_str = ("\n  - ").join(node_names)
+    and_more_str = f"\n  - and {len(node_names) - 3} more" if len(node_names) > 4 else ""
+    summary_msg = (
+        f"\nSome tests were excluded because at least one parent is not selected. "
+        f"Use the --greedy flag to include them."
+        f"\n  - {summary_nodes_str}{and_more_str}"
+    )
+    logger.info(summary_msg)
+    if len(node_names) > 4:
+        debug_msg = (
+            f"Full list of tests that were excluded:"
+            f"\n  - {debug_nodes_str}"
+        )
+        logger.debug(debug_msg)
+
+
 def can_select_indirectly(node):
     """If a node is not selected itself, but its parent(s) are, it may qualify
     for indirect selection.
@@ -151,16 +168,16 @@ class NodeSelector(MethodManager):
 
         return direct_nodes, indirect_nodes
 
-    def select_nodes(self, spec: SelectionSpec) -> Set[UniqueId]:
+    def select_nodes(self, spec: SelectionSpec) -> Tuple[Set[UniqueId], Set[UniqueId]]:
         """Select the nodes in the graph according to the spec.
 
         This is the main point of entry for turning a spec into a set of nodes:
         - Recurse through spec, select by criteria, combine by set operation
         - Return final (unfiltered) selection set
         """
-
         direct_nodes, indirect_nodes = self.select_nodes_recursively(spec)
-        return direct_nodes
+        indirect_only = indirect_nodes.difference(direct_nodes)
+        return direct_nodes, indirect_only
 
     def _is_graph_member(self, unique_id: UniqueId) -> bool:
         if unique_id in self.manifest.sources:
@@ -213,6 +230,8 @@ class NodeSelector(MethodManager):
         #  - If ANY parent is missing, return it separately. We'll keep it around
         #    for later and see if its other parents show up.
         # We use this for INCLUSION.
+        # Users can also opt in to inclusive GREEDY mode by passing --greedy flag,
+        # or by specifying `greedy: true` in a yaml selector
 
         direct_nodes = set(selected)
         indirect_nodes = set()
@@ -251,15 +270,24 @@ class NodeSelector(MethodManager):
 
             - node selection. Based on the include/exclude sets, the set
                 of matched unique IDs is returned
-                - expand the graph at each leaf node, before combination
-                    - selectors might override this. for example, this is where
-                        tests are added
+                - includes direct + indirect selection (for tests)
             - filtering:
                 - selectors can filter the nodes after all of them have been
                   selected
         """
-        selected_nodes = self.select_nodes(spec)
+        selected_nodes, indirect_only = self.select_nodes(spec)
         filtered_nodes = self.filter_selection(selected_nodes)
+
+        if indirect_only:
+            filtered_unused_nodes = self.filter_selection(indirect_only)
+            if filtered_unused_nodes and spec.greedy_warning:
+                # log anything that didn't make the cut
+                unused_node_names = []
+                for unique_id in filtered_unused_nodes:
+                    name = self.manifest.nodes[unique_id].name
+                    unused_node_names.append(name)
+                alert_unused_nodes(spec, unused_node_names)
+
         return filtered_nodes
 
     def get_graph_queue(self, spec: SelectionSpec) -> GraphQueue:

core/dbt/graph/selector_methods.py

@@ -405,27 +405,38 @@ class StateSelectorMethod(SelectorMethod):
 
         return modified
 
-    def recursively_check_macros_modified(self, node):
-        # check if there are any changes in macros the first time
-        if self.modified_macros is None:
-            self.modified_macros = self._macros_modified()
-
+    def recursively_check_macros_modified(self, node, previous_macros):
         # loop through all macros that this node depends on
         for macro_uid in node.depends_on.macros:
+            # avoid infinite recursion if we've already seen this macro
+            if macro_uid in previous_macros:
+                continue
+            previous_macros.append(macro_uid)
             # is this macro one of the modified macros?
             if macro_uid in self.modified_macros:
                 return True
             # if not, and this macro depends on other macros, keep looping
-            macro = self.manifest.macros[macro_uid]
-            if len(macro.depends_on.macros) > 0:
-                return self.recursively_check_macros_modified(macro)
+            macro_node = self.manifest.macros[macro_uid]
+            if len(macro_node.depends_on.macros) > 0:
+                return self.recursively_check_macros_modified(macro_node, previous_macros)
             else:
                 return False
-        return False
+
+    def check_macros_modified(self, node):
+        # check if there are any changes in macros the first time
+        if self.modified_macros is None:
+            self.modified_macros = self._macros_modified()
+        # no macros have been modified, skip looping entirely
+        if not self.modified_macros:
+            return False
+        # recursively loop through upstream macros to see if any is modified
+        else:
+            previous_macros = []
+            return self.recursively_check_macros_modified(node, previous_macros)
 
     def check_modified(self, old: Optional[SelectorTarget], new: SelectorTarget) -> bool:
         different_contents = not new.same_contents(old)  # type: ignore
-        upstream_macro_change = self.recursively_check_macros_modified(new)
+        upstream_macro_change = self.check_macros_modified(new)
         return different_contents or upstream_macro_change
 
     def check_modified_body(self, old: Optional[SelectorTarget], new: SelectorTarget) -> bool:
@@ -457,7 +468,7 @@ class StateSelectorMethod(SelectorMethod):
             return False
 
     def check_modified_macros(self, _, new: SelectorTarget) -> bool:
-        return self.recursively_check_macros_modified(new)
+        return self.check_macros_modified(new)
 
     def check_new(self, old: Optional[SelectorTarget], new: SelectorTarget) -> bool:
         return old is None

core/dbt/graph/selector_spec.py

@@ -67,6 +67,7 @@ class SelectionCriteria:
     children: bool
     children_depth: Optional[int]
     greedy: bool = False
+    greedy_warning: bool = False  # do not raise warning for yaml selectors
 
     def __post_init__(self):
         if self.children and self.childrens_parents:
@@ -124,11 +125,11 @@ class SelectionCriteria:
             parents_depth=parents_depth,
             children=bool(dct.get('children')),
             children_depth=children_depth,
-            greedy=greedy
+            greedy=(greedy or bool(dct.get('greedy'))),
         )
 
     @classmethod
-    def dict_from_single_spec(cls, raw: str, greedy: bool = False):
+    def dict_from_single_spec(cls, raw: str):
         result = RAW_SELECTOR_PATTERN.match(raw)
         if result is None:
             return {'error': 'Invalid selector spec'}
@@ -145,6 +146,8 @@ class SelectionCriteria:
             dct['parents'] = bool(dct.get('parents'))
         if 'children' in dct:
             dct['children'] = bool(dct.get('children'))
+        if 'greedy' in dct:
+            dct['greedy'] = bool(dct.get('greedy'))
         return dct
 
     @classmethod
@@ -162,10 +165,12 @@ class BaseSelectionGroup(Iterable[SelectionSpec], metaclass=ABCMeta):
         self,
         components: Iterable[SelectionSpec],
         expect_exists: bool = False,
+        greedy_warning: bool = True,
         raw: Any = None,
     ):
         self.components: List[SelectionSpec] = list(components)
         self.expect_exists = expect_exists
+        self.greedy_warning = greedy_warning
         self.raw = raw
 
     def __iter__(self) -> Iterator[SelectionSpec]:

core/dbt/include/global_project/macros/adapters/common.sql

@@ -331,7 +331,7 @@
        
             {% for column in add_columns %}
                add column {{ column.name }} {{ column.data_type }}{{ ',' if not loop.last }}
-            {% endfor %}{{ ',' if remove_columns | length > 0 }}
+            {% endfor %}{{ ',' if add_columns and remove_columns }}
             
             {% for column in remove_columns %}
                 drop column {{ column.name }}{{ ',' if not loop.last }}

core/dbt/include/global_project/macros/etc/where_subquery.sql

@@ -1,5 +1,5 @@
 {% macro get_where_subquery(relation) -%}
-    {% do return(adapter.dispatch('get_where_subquery')(relation)) %}
+    {% do return(adapter.dispatch('get_where_subquery', 'dbt')(relation)) %}
 {%- endmacro %}
 
 {% macro default__get_where_subquery(relation) -%}

core/dbt/include/global_project/macros/materializations/seed/seed.sql

@@ -51,7 +51,7 @@
 {% endmacro %}
 
 {% macro get_batch_size() -%}
-  {{ adapter.dispatch('get_batch_size', 'dbt')() }}
+  {{ return(adapter.dispatch('get_batch_size', 'dbt')()) }}
 {%- endmacro %}
 
 {% macro default__get_batch_size() %}

core/dbt/logger.py

@@ -345,7 +345,7 @@ class TimestampNamed(logbook.Processor):
 class ScrubSecrets(logbook.Processor):
     def process(self, record):
         for secret in get_secret_env():
-            record.message = record.message.replace(secret, "*****")
+            record.message = str(record.message).replace(secret, "*****")
 
 
 logger = logbook.Logger('dbt')

core/dbt/main.py

@@ -406,6 +406,14 @@ def _build_build_subparser(subparsers, base_subparser):
         Store test results (failing rows) in the database
         '''
     )
+    sub.add_argument(
+        '--greedy',
+        action='store_true',
+        help='''
+        Select all tests that touch the selected resources,
+        even if they also depend on unselected resources
+        '''
+    )
     resource_values: List[str] = [
         str(s) for s in build_task.BuildTask.ALL_RESOURCE_VALUES
     ] + ['all']
@@ -637,7 +645,7 @@ def _add_table_mutability_arguments(*subparsers):
             '--full-refresh',
             action='store_true',
             help='''
-            If specified, DBT will drop incremental models and
+            If specified, dbt will drop incremental models and
             fully-recalculate the incremental table from the model definition.
             '''
         )
@@ -753,6 +761,14 @@ def _build_test_subparser(subparsers, base_subparser):
         Store test results (failing rows) in the database
         '''
     )
+    sub.add_argument(
+        '--greedy',
+        action='store_true',
+        help='''
+        Select all tests that touch the selected resources,
+        even if they also depend on unselected resources
+        '''
+    )
 
     sub.set_defaults(cls=test_task.TestTask, which='test', rpc_method='test')
     return sub
@@ -878,6 +894,14 @@ def _build_list_subparser(subparsers, base_subparser):
         metavar='SELECTOR',
         required=False,
     )
+    sub.add_argument(
+        '--greedy',
+        action='store_true',
+        help='''
+        Select all tests that touch the selected resources,
+        even if they also depend on unselected resources
+        '''
+    )
     _add_common_selector_arguments(sub)
 
     return sub

core/dbt/parser/manifest.py

@@ -247,12 +247,17 @@ class ManifestLoader:
                     # get file info for local logs
                     parse_file_type = None
                     file_id = partial_parsing.processing_file
-                    if file_id and file_id in self.manifest.files:
-                        old_file = self.manifest.files[file_id]
-                        parse_file_type = old_file.parse_file_type
-                        logger.debug(f"Partial parsing exception processing file {file_id}")
-                        file_dict = old_file.to_dict()
-                        logger.debug(f"PP file: {file_dict}")
+                    if file_id:
+                        source_file = None
+                        if file_id in self.saved_manifest.files:
+                            source_file = self.saved_manifest.files[file_id]
+                        elif file_id in self.manifest.files:
+                            source_file = self.manifest.files[file_id]
+                        if source_file:
+                            parse_file_type = source_file.parse_file_type
+                            logger.debug(f"Partial parsing exception processing file {file_id}")
+                            file_dict = source_file.to_dict()
+                            logger.debug(f"PP file: {file_dict}")
                     exc_info['parse_file_type'] = parse_file_type
                     logger.debug(f"PP exception info: {exc_info}")
 
@@ -310,6 +315,7 @@ class ManifestLoader:
             # aren't in place yet
             self.manifest.rebuild_ref_lookup()
             self.manifest.rebuild_doc_lookup()
+            self.manifest.rebuild_disabled_lookup()
 
             # Load yaml files
             parser_types = [SchemaParser]

core/dbt/parser/models.py

@@ -1,3 +1,5 @@
+
+from copy import deepcopy
 from dbt.context.context_config import ContextConfig
 from dbt.contracts.graph.parsed import ParsedModelNode
 import dbt.flags as flags
@@ -11,7 +13,11 @@ from dbt_extractor import ExtractionError, py_extract_from_source  # type: ignor
 from functools import reduce
 from itertools import chain
 import random
-from typing import Any, Dict, Iterator, List, Optional, Union
+from typing import Any, Dict, Iterator, List, Optional, Tuple, Union
+
+# debug loglines are used for integration testing. If you change
+# the code at the beginning of the debug line, change the tests in
+# test/integration/072_experimental_parser_tests/test_all_experimental_parser.py
 
 
 class ModelParser(SimpleSQLParser[ParsedModelNode]):
@@ -40,16 +46,11 @@ class ModelParser(SimpleSQLParser[ParsedModelNode]):
         experimentally_parsed: Optional[Union[str, Dict[str, List[Any]]]] = None
         config_call_dict: Dict[str, Any] = {}
         source_calls: List[List[str]] = []
+        result: List[str] = []
 
         # run the experimental parser if the flag is on or if we're sampling
         if flags.USE_EXPERIMENTAL_PARSER or sample:
             if self._has_banned_macro(node):
-                # this log line is used for integration testing. If you change
-                # the code at the beginning of the line change the tests in
-                # test/integration/072_experimental_parser_tests/test_all_experimental_parser.py
-                logger.debug(
-                    f"1601: parser fallback to jinja because of macro override for {node.path}"
-                )
                 experimentally_parsed = "has_banned_macro"
             else:
                 # run the experimental parser and return the results
@@ -75,69 +76,76 @@ class ModelParser(SimpleSQLParser[ParsedModelNode]):
                 source_calls.append([s[0], s[1]])
             experimentally_parsed['sources'] = source_calls
 
-        # normal dbt run
+        # if we're sampling during a normal dbt run, populate an entirely new node to compare
         if not flags.USE_EXPERIMENTAL_PARSER:
-            # normal rendering
-            super().render_update(node, config)
-            # if we're sampling, compare for correctness
-            if sample:
-                result = _get_sample_result(
-                    experimentally_parsed,
-                    config_call_dict,
-                    source_calls,
-                    node,
-                    config
+            if sample and isinstance(experimentally_parsed, dict):
+                # if this will _never_ mutate anything `self` we could avoid these deep copies,
+                # but we can't really guarantee that going forward.
+                model_parser_copy = self.partial_deepcopy()
+                exp_sample_node = deepcopy(node)
+                exp_sample_config = deepcopy(config)
+
+                model_parser_copy.populate(
+                    exp_sample_node,
+                    exp_sample_config,
+                    experimentally_parsed
                 )
-                # fire a tracking event. this fires one event for every sample
-                # so that we have data on a per file basis. Not only can we expect
-                # no false positives or misses, we can expect the number model
-                # files parseable by the experimental parser to match our internal
-                # testing.
-                if tracking.active_user is not None:  # None in some tests
-                    tracking.track_experimental_parser_sample({
-                        "project_id": self.root_project.hashed_name(),
-                        "file_id": utils.get_hash(node),
-                        "status": result
-                    })
+
+            super().render_update(node, config)
 
         # if the --use-experimental-parser flag was set, and the experimental parser succeeded
-        elif isinstance(experimentally_parsed, Dict):
-            # since it doesn't need python jinja, fit the refs, sources, and configs
-            # into the node. Down the line the rest of the node will be updated with
-            # this information. (e.g. depends_on etc.)
-            config._config_call_dict = config_call_dict
-
-            # this uses the updated config to set all the right things in the node.
-            # if there are hooks present, it WILL render jinja. Will need to change
-            # when the experimental parser supports hooks
-            self.update_parsed_node_config(node, config)
-
-            # update the unrendered config with values from the file.
+        elif isinstance(experimentally_parsed, dict):
+            # update the unrendered config with values from the static parser.
             # values from yaml files are in there already
-            node.unrendered_config.update(dict(experimentally_parsed['configs']))
-
-            # set refs and sources on the node object
-            node.refs += experimentally_parsed['refs']
-            node.sources += experimentally_parsed['sources']
-
-            # configs don't need to be merged into the node
-            # setting them in config._config_call_dict is sufficient
+            self.populate(
+                node,
+                config,
+                experimentally_parsed
+            )
 
             self.manifest._parsing_info.static_analysis_parsed_path_count += 1
 
         # the experimental parser didn't run on this model.
         # fall back to python jinja rendering.
-        elif experimentally_parsed in ["has_banned_macro"]:
-            # not logging here since the reason should have been logged above
+        elif isinstance(experimentally_parsed, str):
+            if experimentally_parsed == "cannot_parse":
+                result += ["01_stable_parser_cannot_parse"]
+                logger.debug(
+                    f"1602: parser fallback to jinja for {node.path}"
+                )
+            elif experimentally_parsed == "has_banned_macro":
+                result += ["08_has_banned_macro"]
+                logger.debug(
+                    f"1601: parser fallback to jinja because of macro override for {node.path}"
+                )
+
             super().render_update(node, config)
-        # the experimental parser ran on this model and failed.
-        # fall back to python jinja rendering.
+        # otherwise jinja rendering.
         else:
-            logger.debug(
-                f"1602: parser fallback to jinja because of extractor failure for {node.path}"
-            )
             super().render_update(node, config)
 
+        if sample and isinstance(experimentally_parsed, dict):
+            # now that the sample succeeded, is populated and the current
+            # values are rendered, compare the two and collect the tracking messages
+            result += _get_exp_sample_result(
+                exp_sample_node,
+                exp_sample_config,
+                node,
+                config,
+            )
+
+        # fire a tracking event. this fires one event for every sample
+        # so that we have data on a per file basis. Not only can we expect
+        # no false positives or misses, we can expect the number model
+        # files parseable by the experimental parser to match our internal
+        # testing.
+        if result and tracking.active_user is not None:  # None in some tests
+            tracking.track_experimental_parser_sample({
+                "project_id": self.root_project.hashed_name(),
+                "file_id": utils.get_hash(node),
+                "status": result
+            })
+
     # checks for banned macros
     def _has_banned_macro(
         self, node: ParsedModelNode
@@ -163,64 +171,117 @@ class ModelParser(SimpleSQLParser[ParsedModelNode]):
             False
         )
 
+    # this method updates the model note rendered and unrendered config as well
+    # as the node object. Used to populate these values when circumventing jinja
+    # rendering like the static parser.
+    def populate(
+        self,
+        node: ParsedModelNode,
+        config: ContextConfig,
+        experimentally_parsed: Dict[str, Any]
+    ):
+        # manually fit configs in
+        config._config_call_dict = _get_config_call_dict(experimentally_parsed)
+
+        # if there are hooks present this, it WILL render jinja. Will need to change
+        # when the experimental parser supports hooks
+        self.update_parsed_node_config(node, config)
+
+        # update the unrendered config with values from the file.
+        # values from yaml files are in there already
+        node.unrendered_config.update(experimentally_parsed['configs'])
+
+        # set refs and sources on the node object
+        node.refs += experimentally_parsed['refs']
+        node.sources += experimentally_parsed['sources']
+
+        # configs don't need to be merged into the node because they
+        # are read from config._config_call_dict
+
+    # the manifest is often huge so this method avoids deepcopying it
+    def partial_deepcopy(self):
+        return ModelParser(
+            deepcopy(self.project),
+            self.manifest,
+            deepcopy(self.root_project)
+        )
+
+
+# pure function. safe to use elsewhere, but unlikely to be useful outside this file.
+def _get_config_call_dict(
+    static_parser_result: Dict[str, List[Any]]
+) -> Dict[str, Any]:
+    config_call_dict: Dict[str, Any] = {}
+
+    for c in static_parser_result['configs']:
+        ContextConfig._add_config_call(config_call_dict, {c[0]: c[1]})
+
+    return config_call_dict
+
 
 # returns a list of string codes to be sent as a tracking event
-def _get_sample_result(
-    sample_output: Optional[Union[str, Dict[str, Any]]],
-    config_call_dict: Dict[str, Any],
-    source_calls: List[List[str]],
+def _get_exp_sample_result(
+    sample_node: ParsedModelNode,
+    sample_config: ContextConfig,
     node: ParsedModelNode,
     config: ContextConfig
 ) -> List[str]:
-    result: List[str] = []
-    # experimental parser didn't run
-    if sample_output is None:
-        result += ["09_experimental_parser_skipped"]
-    # experimental parser couldn't parse
-    elif (isinstance(sample_output, str)):
-        if sample_output == "cannot_parse":
-            result += ["01_experimental_parser_cannot_parse"]
-        elif sample_output == "has_banned_macro":
-            result += ["08_has_banned_macro"]
-    else:
-        # look for false positive configs
-        for k in config_call_dict.keys():
-            if k not in config._config_call_dict:
-                result += ["02_false_positive_config_value"]
-                break
+    result: List[Tuple[int, str]] = _get_sample_result(sample_node, sample_config, node, config)
 
-        # look for missed configs
-        for k in config._config_call_dict.keys():
-            if k not in config_call_dict:
-                result += ["03_missed_config_value"]
-                break
+    def process(codemsg):
+        code, msg = codemsg
+        return f"0{code}_experimental_{msg}"
 
-        # look for false positive sources
-        for s in sample_output['sources']:
-            if s not in node.sources:
-                result += ["04_false_positive_source_value"]
-                break
+    return list(map(process, result))
 
-        # look for missed sources
-        for s in node.sources:
-            if s not in sample_output['sources']:
-                result += ["05_missed_source_value"]
-                break
 
-        # look for false positive refs
-        for r in sample_output['refs']:
-            if r not in node.refs:
-                result += ["06_false_positive_ref_value"]
-                break
+# returns a list of messages and int codes and messages that need a single digit
+# prefix to be prepended before being sent as a tracking event
+def _get_sample_result(
+    sample_node: ParsedModelNode,
+    sample_config: ContextConfig,
+    node: ParsedModelNode,
+    config: ContextConfig
+) -> List[Tuple[int, str]]:
+    result: List[Tuple[int, str]] = []
+    # look for false positive configs
+    for k in sample_config._config_call_dict:
+        if k not in config._config_call_dict:
+            result += [(2, "false_positive_config_value")]
+            break
 
-        # look for missed refs
-        for r in node.refs:
-            if r not in sample_output['refs']:
-                result += ["07_missed_ref_value"]
-                break
+    # look for missed configs
+    for k in config._config_call_dict.keys():
+        if k not in sample_config._config_call_dict.keys():
+            result += [(3, "missed_config_value")]
+            break
 
-        # if there are no errors, return a success value
-        if not result:
-            result = ["00_exact_match"]
+    # look for false positive sources
+    for s in sample_node.sources:
+        if s not in node.sources:
+            result += [(4, "false_positive_source_value")]
+            break
+
+    # look for missed sources
+    for s in node.sources:
+        if s not in sample_node.sources:
+            result += [(5, "missed_source_value")]
+            break
+
+    # look for false positive refs
+    for r in sample_node.refs:
+        if r not in node.refs:
+            result += [(6, "false_positive_ref_value")]
+            break
+
+    # look for missed refs
+    for r in node.refs:
+        if r not in sample_node.refs:
+            result += [(7, "missed_ref_value")]
+            break
+
+    # if there are no errors, return a success value
+    if not result:
+        result = [(0, "exact_match")]
 
     return result

core/dbt/parser/partial.py

@@ -47,6 +47,7 @@ class PartialParsing:
         self.macro_child_map: Dict[str, List[str]] = {}
         self.build_file_diff()
         self.processing_file = None
+        self.disabled_by_file_id = self.saved_manifest.build_disabled_by_file_id()
 
     def skip_parsing(self):
         return (
@@ -233,24 +234,38 @@ class PartialParsing:
             return
 
         # These files only have one node.
-        unique_id = old_source_file.nodes[0]
+        unique_id = None
+        if old_source_file.nodes:
+            unique_id = old_source_file.nodes[0]
+        else:
+            # It's not clear when this would actually happen.
+            # Logging in case there are other associated errors.
+            logger.debug(f"Partial parsing: node not found for source_file {old_source_file}")
 
         # replace source_file in saved and add to parsing list
         file_id = new_source_file.file_id
         self.deleted_manifest.files[file_id] = old_source_file
         self.saved_files[file_id] = new_source_file
         self.add_to_pp_files(new_source_file)
-        self.remove_node_in_saved(new_source_file, unique_id)
+        if unique_id:
+            self.remove_node_in_saved(new_source_file, unique_id)
 
     def remove_node_in_saved(self, source_file, unique_id):
-        # Has already been deleted by another action
-        if unique_id not in self.saved_manifest.nodes:
+        if unique_id in self.saved_manifest.nodes:
+            # delete node in saved
+            node = self.saved_manifest.nodes.pop(unique_id)
+            self.deleted_manifest.nodes[unique_id] = node
+        elif source_file.file_id in self.disabled_by_file_id:
+            for dis_index, dis_node in self.saved_manifest.disabled:
+                if dis_node.file_id == source_file.file_id:
+                    node = dis_node
+                    break
+            if dis_node:
+                del self.saved_manifest.disabled[dis_index]
+        else:
+            # Has already been deleted by another action
             return
 
-        # delete node in saved
-        node = self.saved_manifest.nodes.pop(unique_id)
-        self.deleted_manifest.nodes[unique_id] = node
-
         # look at patch_path in model node to see if we need
         # to reapply a patch from a schema_file.
         if node.patch_path:
@@ -261,15 +276,22 @@ class PartialParsing:
                 schema_file = self.saved_files[file_id]
                 dict_key = parse_file_type_to_key[source_file.parse_file_type]
                 # look for a matching list dictionary
-                for elem in schema_file.dict_from_yaml[dict_key]:
-                    if elem['name'] == node.name:
-                        elem_patch = elem
-                        break
+                elem_patch = None
+                if dict_key in schema_file.dict_from_yaml:
+                    for elem in schema_file.dict_from_yaml[dict_key]:
+                        if elem['name'] == node.name:
+                            elem_patch = elem
+                            break
                 if elem_patch:
                     self.delete_schema_mssa_links(schema_file, dict_key, elem_patch)
                     self.merge_patch(schema_file, dict_key, elem_patch)
                     if unique_id in schema_file.node_patches:
                         schema_file.node_patches.remove(unique_id)
+            if unique_id in self.saved_manifest.disabled:
+                # We have a patch_path in disabled nodes with a patch so
+                # that we can connect the patch to the node
+                for node in self.saved_manifest.disabled[unique_id]:
+                    node.patch_path = None
 
     def update_macro_in_saved(self, new_source_file, old_source_file):
         if self.already_scheduled_for_parsing(old_source_file):
@@ -290,7 +312,8 @@ class PartialParsing:
         # nodes [unique_ids] -- SQL files
         # There should always be a node for a SQL file
         if not source_file.nodes:
-            raise Exception(f"No nodes found for source file {source_file.file_id}")
+            logger.debug(f"No nodes found for source file {source_file.file_id}")
+            return
         # There is generally only 1 node for SQL files, except for macros
         for unique_id in source_file.nodes:
             self.remove_node_in_saved(source_file, unique_id)
@@ -299,7 +322,10 @@ class PartialParsing:
     # We need to re-parse nodes that reference another removed node
     def schedule_referencing_nodes_for_parsing(self, unique_id):
         # Look at "children", i.e. nodes that reference this node
-        self.schedule_nodes_for_parsing(self.saved_manifest.child_map[unique_id])
+        if unique_id in self.saved_manifest.child_map:
+            self.schedule_nodes_for_parsing(self.saved_manifest.child_map[unique_id])
+        else:
+            logger.debug(f"Partial parsing: {unique_id} not found in child_map")
 
     def schedule_nodes_for_parsing(self, unique_ids):
         for unique_id in unique_ids:

core/dbt/parser/schema_test_builders.py

@@ -358,7 +358,7 @@ class TestBuilder(Generic[Testable]):
         if self.warn_if is not None:
             config['warn_if'] = self.warn_if
         if self.error_if is not None:
-            config['error_id'] = self.error_if
+            config['error_if'] = self.error_if
         if self.fail_calc is not None:
             config['fail_calc'] = self.fail_calc
         if self.store_failures is not None:
@@ -369,8 +369,6 @@ class TestBuilder(Generic[Testable]):
             config['database'] = self.database
         if self.schema is not None:
             config['schema'] = self.schema
-        if self.alias is not None:
-            config['alias'] = self.alias
         return config
 
     def tags(self) -> List[str]:

core/dbt/parser/schemas.py

@@ -825,6 +825,10 @@ class NodePatchParser(
             )
         if unique_id is None:
             # This will usually happen when a node is disabled
+            disabled_nodes = self.manifest.disabled_lookup.find(patch.name, patch.package_name)
+            if disabled_nodes:
+                for node in disabled_nodes:
+                    node.patch_path = source_file.file_id
             return
 
         # patches can't be overwritten

core/dbt/task/build.py

@@ -3,6 +3,7 @@ from .snapshot import SnapshotRunner as snapshot_model_runner
 from .seed import SeedRunner as seed_runner
 from .test import TestRunner as test_runner
 
+from dbt.adapters.factory import get_adapter
 from dbt.contracts.results import NodeStatus
 from dbt.exceptions import InternalException
 from dbt.graph import ResourceTypeSelector
@@ -64,3 +65,12 @@ class BuildTask(RunTask):
 
     def get_runner_type(self, node):
         return self.RUNNER_MAP.get(node.resource_type)
+
+    def compile_manifest(self):
+        if self.manifest is None:
+            raise InternalException(
+                'compile_manifest called before manifest was loaded'
+            )
+        adapter = get_adapter(self.config)
+        compiler = adapter.get_compiler()
+        self.graph = compiler.compile(self.manifest, add_test_edges=True)

core/dbt/task/run.py

@@ -414,13 +414,13 @@ class RunTask(CompileTask):
 
     def after_run(self, adapter, results):
         # in on-run-end hooks, provide the value 'database_schemas', which is a
-        # list  of unique database, schema pairs that successfully executed
-        # models  were in. for backwards compatibility, include the old
+        # list of unique (database, schema) pairs that successfully executed
+        # models were in. For backwards compatibility, include the old
         # 'schemas', which did not include database information.
 
         database_schema_set: Set[Tuple[Optional[str], str]] = {
             (r.node.database, r.node.schema) for r in results
-            if r.status not in (
+            if r.node.is_relational and r.status not in (
                 NodeStatus.Error,
                 NodeStatus.Fail,
                 NodeStatus.Skipped

core/dbt/task/runnable.py

@@ -1,5 +1,6 @@
 import os
 import time
+import json
 from abc import abstractmethod
 from concurrent.futures import as_completed
 from datetime import datetime
@@ -14,6 +15,7 @@ from .printer import (
 )
 
 from dbt import ui
+from dbt.clients.system import write_file
 from dbt.task.base import ConfiguredTask
 from dbt.adapters.base import BaseRelation
 from dbt.adapters.factory import get_adapter
@@ -69,6 +71,9 @@ class ManifestTask(ConfiguredTask):
         if flags.WRITE_JSON:
             path = os.path.join(self.config.target_path, MANIFEST_FILE_NAME)
             self.manifest.write(path)
+        if os.getenv('DBT_WRITE_FILES'):
+            path = os.path.join(self.config.target_path, 'files.json')
+            write_file(path, json.dumps(self.manifest.files, cls=dbt.utils.JSONEncoder, indent=4))
 
     def load_manifest(self):
         self.manifest = ManifestLoader.get_full_manifest(self.config)
@@ -438,7 +443,7 @@ class GraphRunnableTask(ManifestTask):
             )
 
         if len(self._flattened_nodes) == 0:
-            logger.warning("WARNING: Nothing to do. Try checking your model "
+            logger.warning("\nWARNING: Nothing to do. Try checking your model "
                            "configs and model specification args")
             result = self.get_result(
                 results=[],

core/dbt/task/test.py

@@ -1,3 +1,5 @@
+from distutils.util import strtobool
+
 from dataclasses import dataclass
 from dbt import utils
 from dbt.dataclass_schema import dbtClassMixin
@@ -19,6 +21,7 @@ from dbt.context.providers import generate_runtime_model
 from dbt.clients.jinja import MacroGenerator
 from dbt.exceptions import (
     InternalException,
+    invalid_bool_error,
     missing_materialization
 )
 from dbt.graph import (
@@ -36,6 +39,23 @@ class TestResultData(dbtClassMixin):
     should_warn: bool
     should_error: bool
 
+    @classmethod
+    def validate(cls, data):
+        data['should_warn'] = cls.convert_bool_type(data['should_warn'])
+        data['should_error'] = cls.convert_bool_type(data['should_error'])
+        super().validate(data)
+
+    def convert_bool_type(field) -> bool:
+        # if it's type string let python decide if it's a valid value to convert to bool
+        if isinstance(field, str):
+            try:
+                return bool(strtobool(field))  # type: ignore
+            except ValueError:
+                raise invalid_bool_error(field, 'get_test_sql')
+
+        # need this so we catch both true bools and 0/1
+        return bool(field)
+
 
 class TestRunner(CompileRunner):
     def describe_node(self):

core/dbt/version.py

@@ -96,5 +96,5 @@ def _get_dbt_plugins_info():
         yield plugin_name, mod.version
 
 
-__version__ = '0.21.0b2'
+__version__ = '0.21.1'
 installed = get_installed_version()

core/scripts/create_adapter_plugins.py

@@ -284,12 +284,12 @@ def parse_args(argv=None):
     parser.add_argument('adapter')
     parser.add_argument('--title-case', '-t', default=None)
     parser.add_argument('--dependency', action='append')
-    parser.add_argument('--dbt-core-version', default='0.21.0b2')
+    parser.add_argument('--dbt-core-version', default='0.21.1')
     parser.add_argument('--email')
     parser.add_argument('--author')
     parser.add_argument('--url')
     parser.add_argument('--sql', action='store_true')
-    parser.add_argument('--package-version', default='0.21.0b2')
+    parser.add_argument('--package-version', default='0.21.1')
     parser.add_argument('--project-version', default='1.0')
     parser.add_argument(
         '--no-dependency', action='store_false', dest='set_dependency'

core/setup.py

@@ -24,7 +24,7 @@ def read(fname):
 
 
 package_name = "dbt-core"
-package_version = "0.21.0b2"
+package_version = "0.21.1"
 description = """dbt (data build tool) is a command line tool that helps \
 analysts and engineers transform data in their warehouse more effectively"""
 

docker/requirements/requirements.0.21.0.txt

@@ -0,0 +1,75 @@
+agate==1.6.1
+asn1crypto==1.4.0
+attrs==21.2.0
+azure-common==1.1.27
+azure-core==1.19.0
+azure-storage-blob==12.9.0
+Babel==2.9.1
+boto3==1.18.53
+botocore==1.21.53
+cachetools==4.2.4
+certifi==2021.5.30
+cffi==1.14.6
+chardet==4.0.0
+charset-normalizer==2.0.6
+colorama==0.4.4
+cryptography==3.4.8
+google-api-core==1.31.3
+google-auth==1.35.0
+google-cloud-bigquery==2.28.0
+google-cloud-core==1.7.2
+google-crc32c==1.2.0
+google-resumable-media==2.0.3
+googleapis-common-protos==1.53.0
+grpcio==1.41.0
+hologram==0.0.14
+idna==3.2
+importlib-metadata==4.8.1
+isodate==0.6.0
+jeepney==0.7.1
+Jinja2==2.11.3
+jmespath==0.10.0
+json-rpc==1.13.0
+jsonschema==3.1.1
+keyring==21.8.0
+leather==0.3.3
+Logbook==1.5.3
+MarkupSafe==2.0.1
+mashumaro==2.5
+minimal-snowplow-tracker==0.0.2
+msgpack==1.0.2
+msrest==0.6.21
+networkx==2.6.3
+oauthlib==3.1.1
+oscrypto==1.2.1
+packaging==20.9
+parsedatetime==2.6
+proto-plus==1.19.2
+protobuf==3.17.3
+psycopg2-binary==2.9.1
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pycparser==2.20
+pycryptodomex==3.10.4
+PyJWT==2.1.0
+pyOpenSSL==20.0.1
+pyparsing==2.4.7
+pyrsistent==0.18.0
+python-dateutil==2.8.2
+python-slugify==5.0.2
+pytimeparse==1.1.8
+pytz==2021.3
+PyYAML==5.4.1
+requests==2.26.0
+requests-oauthlib==1.3.0
+rsa==4.7.2
+s3transfer==0.5.0
+SecretStorage==3.3.1
+six==1.16.0
+snowflake-connector-python==2.5.1
+sqlparse==0.4.2
+text-unidecode==1.3
+typing-extensions==3.10.0.2
+urllib3==1.26.7
+Werkzeug==2.0.1
+zipp==3.6.0

docker/requirements/requirements.0.21.0rc1.txt

@@ -0,0 +1,75 @@
+agate==1.6.1
+asn1crypto==1.4.0
+attrs==21.2.0
+azure-common==1.1.27
+azure-core==1.18.0
+azure-storage-blob==12.8.1
+Babel==2.9.1
+boto3==1.18.44
+botocore==1.21.44
+cachetools==4.2.2
+certifi==2021.5.30
+cffi==1.14.6
+chardet==4.0.0
+charset-normalizer==2.0.6
+colorama==0.4.4
+cryptography==3.4.8
+google-api-core==1.31.2
+google-auth==1.35.0
+google-cloud-bigquery==2.26.0
+google-cloud-core==1.7.2
+google-crc32c==1.1.2
+google-resumable-media==2.0.2
+googleapis-common-protos==1.53.0
+grpcio==1.40.0
+hologram==0.0.14
+idna==3.2
+importlib-metadata==4.8.1
+isodate==0.6.0
+jeepney==0.7.1
+Jinja2==2.11.3
+jmespath==0.10.0
+json-rpc==1.13.0
+jsonschema==3.1.1
+keyring==21.8.0
+leather==0.3.3
+Logbook==1.5.3
+MarkupSafe==2.0.1
+mashumaro==2.5
+minimal-snowplow-tracker==0.0.2
+msgpack==1.0.2
+msrest==0.6.21
+networkx==2.6.3
+oauthlib==3.1.1
+oscrypto==1.2.1
+packaging==20.9
+parsedatetime==2.6
+proto-plus==1.19.0
+protobuf==3.18.0
+psycopg2-binary==2.9.1
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pycparser==2.20
+pycryptodomex==3.10.1
+PyJWT==2.1.0
+pyOpenSSL==20.0.1
+pyparsing==2.4.7
+pyrsistent==0.18.0
+python-dateutil==2.8.2
+python-slugify==5.0.2
+pytimeparse==1.1.8
+pytz==2021.1
+PyYAML==5.4.1
+requests==2.26.0
+requests-oauthlib==1.3.0
+rsa==4.7.2
+s3transfer==0.5.0
+SecretStorage==3.3.1
+six==1.16.0
+snowflake-connector-python==2.5.1
+sqlparse==0.4.2
+text-unidecode==1.3
+typing-extensions==3.10.0.2
+urllib3==1.26.6
+Werkzeug==2.0.1
+zipp==3.5.0

docker/requirements/requirements.0.21.0rc2.txt

@@ -0,0 +1,75 @@
+agate==1.6.1
+asn1crypto==1.4.0
+attrs==21.2.0
+azure-common==1.1.27
+azure-core==1.18.0
+azure-storage-blob==12.9.0
+Babel==2.9.1
+boto3==1.18.48
+botocore==1.21.48
+cachetools==4.2.2
+certifi==2021.5.30
+cffi==1.14.6
+chardet==4.0.0
+charset-normalizer==2.0.6
+colorama==0.4.4
+cryptography==3.4.8
+google-api-core==1.31.3
+google-auth==1.35.0
+google-cloud-bigquery==2.27.0
+google-cloud-core==1.7.2
+google-crc32c==1.2.0
+google-resumable-media==2.0.3
+googleapis-common-protos==1.53.0
+grpcio==1.40.0
+hologram==0.0.14
+idna==3.2
+importlib-metadata==4.8.1
+isodate==0.6.0
+jeepney==0.7.1
+Jinja2==2.11.3
+jmespath==0.10.0
+json-rpc==1.13.0
+jsonschema==3.1.1
+keyring==21.8.0
+leather==0.3.3
+Logbook==1.5.3
+MarkupSafe==2.0.1
+mashumaro==2.5
+minimal-snowplow-tracker==0.0.2
+msgpack==1.0.2
+msrest==0.6.21
+networkx==2.6.3
+oauthlib==3.1.1
+oscrypto==1.2.1
+packaging==20.9
+parsedatetime==2.6
+proto-plus==1.19.0
+protobuf==3.17.3
+psycopg2-binary==2.9.1
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pycparser==2.20
+pycryptodomex==3.10.4
+PyJWT==2.1.0
+pyOpenSSL==20.0.1
+pyparsing==2.4.7
+pyrsistent==0.18.0
+python-dateutil==2.8.2
+python-slugify==5.0.2
+pytimeparse==1.1.8
+pytz==2021.1
+PyYAML==5.4.1
+requests==2.26.0
+requests-oauthlib==1.3.0
+rsa==4.7.2
+s3transfer==0.5.0
+SecretStorage==3.3.1
+six==1.16.0
+snowflake-connector-python==2.5.1
+sqlparse==0.4.2
+text-unidecode==1.3
+typing-extensions==3.10.0.2
+urllib3==1.26.7
+Werkzeug==2.0.1
+zipp==3.5.0

docker/requirements/requirements.0.21.1.txt

@@ -0,0 +1,75 @@
+agate==1.6.1
+asn1crypto==1.4.0
+attrs==21.2.0
+azure-common==1.1.27
+azure-core==1.20.1
+azure-storage-blob==12.9.0
+Babel==2.9.1
+boto3==1.20.15
+botocore==1.23.15
+cachetools==4.2.4
+certifi==2021.10.8
+cffi==1.15.0
+chardet==4.0.0
+charset-normalizer==2.0.8
+colorama==0.4.4
+cryptography==3.4.8
+google-api-core==1.31.2
+google-auth==1.35.0
+google-cloud-bigquery==2.30.1
+google-cloud-core==1.7.2
+google-crc32c==1.3.0
+google-resumable-media==2.1.0
+googleapis-common-protos==1.53.0
+grpcio==1.42.0
+hologram==0.0.14
+idna==3.3
+importlib-metadata==4.8.2
+isodate==0.6.0
+jeepney==0.7.1
+Jinja2==2.11.3
+jmespath==0.10.0
+json-rpc==1.13.0
+jsonschema==3.1.1
+keyring==21.8.0
+leather==0.3.4
+Logbook==1.5.3
+MarkupSafe==2.0.1
+mashumaro==2.5
+minimal-snowplow-tracker==0.0.2
+msgpack==1.0.3
+msrest==0.6.21
+networkx==2.6.3
+oauthlib==3.1.1
+oscrypto==1.2.1
+packaging==20.9
+parsedatetime==2.6
+proto-plus==1.19.8
+protobuf==3.19.1
+psycopg2-binary==2.9.2
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pycparser==2.21
+pycryptodomex==3.11.0
+PyJWT==2.3.0
+pyOpenSSL==20.0.1
+pyparsing==3.0.6
+pyrsistent==0.18.0
+python-dateutil==2.8.2
+python-slugify==5.0.2
+pytimeparse==1.1.8
+pytz==2021.3
+PyYAML==6.0
+requests==2.26.0
+requests-oauthlib==1.3.0
+rsa==4.8
+s3transfer==0.5.0
+SecretStorage==3.3.1
+six==1.16.0
+snowflake-connector-python==2.5.1
+sqlparse==0.4.2
+text-unidecode==1.3
+typing-extensions==3.10.0.2
+urllib3==1.26.7
+Werkzeug==2.0.2
+zipp==3.6.0

docker/requirements/requirements.0.21.1rc1.txt

@@ -0,0 +1,75 @@
+agate==1.6.1
+asn1crypto==1.4.0
+attrs==21.2.0
+azure-common==1.1.27
+azure-core==1.19.1
+azure-storage-blob==12.9.0
+Babel==2.9.1
+boto3==1.19.9
+botocore==1.22.9
+cachetools==4.2.4
+certifi==2021.10.8
+cffi==1.15.0
+chardet==4.0.0
+charset-normalizer==2.0.7
+colorama==0.4.4
+cryptography==3.4.8
+google-api-core==1.31.2
+google-auth==1.35.0
+google-cloud-bigquery==2.29.0
+google-cloud-core==1.7.2
+google-crc32c==1.3.0
+google-resumable-media==2.1.0
+googleapis-common-protos==1.53.0
+grpcio==1.41.1
+hologram==0.0.14
+idna==3.3
+importlib-metadata==4.8.1
+isodate==0.6.0
+jeepney==0.7.1
+Jinja2==2.11.3
+jmespath==0.10.0
+json-rpc==1.13.0
+jsonschema==3.1.1
+keyring==21.8.0
+leather==0.3.4
+Logbook==1.5.3
+MarkupSafe==2.0.1
+mashumaro==2.5
+minimal-snowplow-tracker==0.0.2
+msgpack==1.0.2
+msrest==0.6.21
+networkx==2.6.3
+oauthlib==3.1.1
+oscrypto==1.2.1
+packaging==20.9
+parsedatetime==2.6
+proto-plus==1.19.7
+protobuf==3.19.1
+psycopg2-binary==2.9.1
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pycparser==2.20
+pycryptodomex==3.11.0
+PyJWT==2.3.0
+pyOpenSSL==20.0.1
+pyparsing==3.0.4
+pyrsistent==0.18.0
+python-dateutil==2.8.2
+python-slugify==5.0.2
+pytimeparse==1.1.8
+pytz==2021.3
+PyYAML==6.0
+requests==2.26.0
+requests-oauthlib==1.3.0
+rsa==4.7.2
+s3transfer==0.5.0
+SecretStorage==3.3.1
+six==1.16.0
+snowflake-connector-python==2.5.1
+sqlparse==0.4.2
+text-unidecode==1.3
+typing-extensions==3.10.0.2
+urllib3==1.26.7
+Werkzeug==2.0.2
+zipp==3.6.0

docker/requirements/requirements.0.21.1rc2.txt

@@ -0,0 +1,75 @@
+agate==1.6.1
+asn1crypto==1.4.0
+attrs==21.2.0
+azure-common==1.1.27
+azure-core==1.20.1
+azure-storage-blob==12.9.0
+Babel==2.9.1
+boto3==1.20.5
+botocore==1.23.5
+cachetools==4.2.4
+certifi==2021.10.8
+cffi==1.15.0
+chardet==4.0.0
+charset-normalizer==2.0.7
+colorama==0.4.4
+cryptography==3.4.8
+google-api-core==1.31.2
+google-auth==1.35.0
+google-cloud-bigquery==2.30.1
+google-cloud-core==1.7.2
+google-crc32c==1.3.0
+google-resumable-media==2.1.0
+googleapis-common-protos==1.53.0
+grpcio==1.41.1
+hologram==0.0.14
+idna==3.3
+importlib-metadata==4.8.2
+isodate==0.6.0
+jeepney==0.7.1
+Jinja2==2.11.3
+jmespath==0.10.0
+json-rpc==1.13.0
+jsonschema==3.1.1
+keyring==21.8.0
+leather==0.3.4
+Logbook==1.5.3
+MarkupSafe==2.0.1
+mashumaro==2.5
+minimal-snowplow-tracker==0.0.2
+msgpack==1.0.2
+msrest==0.6.21
+networkx==2.6.3
+oauthlib==3.1.1
+oscrypto==1.2.1
+packaging==20.9
+parsedatetime==2.6
+proto-plus==1.19.8
+protobuf==3.19.1
+psycopg2-binary==2.9.2
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pycparser==2.21
+pycryptodomex==3.11.0
+PyJWT==2.3.0
+pyOpenSSL==20.0.1
+pyparsing==3.0.6
+pyrsistent==0.18.0
+python-dateutil==2.8.2
+python-slugify==5.0.2
+pytimeparse==1.1.8
+pytz==2021.3
+PyYAML==6.0
+requests==2.26.0
+requests-oauthlib==1.3.0
+rsa==4.7.2
+s3transfer==0.5.0
+SecretStorage==3.3.1
+six==1.16.0
+snowflake-connector-python==2.5.1
+sqlparse==0.4.2
+text-unidecode==1.3
+typing-extensions==3.10.0.2
+urllib3==1.26.7
+Werkzeug==2.0.2
+zipp==3.6.0

plugins/bigquery/dbt/adapters/bigquery/__version__.py

@@ -1 +1 @@
-version = '0.21.0b2'
+version = '0.21.1'

plugins/bigquery/setup.py

@@ -20,7 +20,7 @@ except ImportError:
 
 
 package_name = "dbt-bigquery"
-package_version = "0.21.0b2"
+package_version = "0.21.1"
 description = """The bigquery adapter plugin for dbt (data build tool)"""
 
 this_directory = os.path.abspath(os.path.dirname(__file__))
@@ -50,7 +50,7 @@ setup(
         'protobuf>=3.13.0,<4',
         'google-cloud-core>=1.3.0,<2',
         'google-cloud-bigquery>=1.25.0,<3',
-        'google-api-core>=1.16.0,<2',
+        'google-api-core>=1.16.0,<1.31.3',
         'googleapis-common-protos>=1.6.0,<2',
         'six>=1.14.0',
     ],

plugins/postgres/dbt/adapters/postgres/__version__.py

@@ -1 +1 @@
-version = '0.21.0b2'
+version = '0.21.1'

plugins/postgres/setup.py

@@ -41,7 +41,7 @@ def _dbt_psycopg2_name():
 
 
 package_name = "dbt-postgres"
-package_version = "0.21.0b2"
+package_version = "0.21.1"
 description = """The postgres adpter plugin for dbt (data build tool)"""
 
 this_directory = os.path.abspath(os.path.dirname(__file__))

plugins/redshift/dbt/adapters/redshift/__version__.py

@@ -1 +1 @@
-version = '0.21.0b2'
+version = '0.21.1'

plugins/redshift/setup.py

@@ -20,7 +20,7 @@ except ImportError:
 
 
 package_name = "dbt-redshift"
-package_version = "0.21.0b2"
+package_version = "0.21.1"
 description = """The redshift adapter plugin for dbt (data build tool)"""
 
 this_directory = os.path.abspath(os.path.dirname(__file__))

plugins/snowflake/dbt/adapters/snowflake/__version__.py

@@ -1 +1 @@
-version = '0.21.0b2'
+version = '0.21.1'

plugins/snowflake/dbt/include/snowflake/macros/materializations/seed.sql

@@ -1,4 +1,5 @@
 {% macro snowflake__load_csv_rows(model, agate_table) %}
+    {% set batch_size = get_batch_size() %}
     {% set cols_sql = get_seed_column_quoted_csv(model, agate_table.column_names) %}
     {% set bindings = [] %}
 

plugins/snowflake/setup.py

@@ -20,7 +20,7 @@ except ImportError:
 
 
 package_name = "dbt-snowflake"
-package_version = "0.21.0b2"
+package_version = "0.21.1"
 description = """The snowflake adapter plugin for dbt (data build tool)"""
 
 this_directory = os.path.abspath(os.path.dirname(__file__))

setup.py

@@ -24,7 +24,7 @@ with open(os.path.join(this_directory, 'README.md')) as f:
 
 
 package_name = "dbt"
-package_version = "0.21.0b2"
+package_version = "0.21.1"
 description = """With dbt, data analysts and engineers can build analytics \
 the way engineers build applications."""
 

test/integration/005_simple_seed_test/data-big/.gitignore

@@ -0,0 +1 @@
+*.csv

test/integration/005_simple_seed_test/test_simple_seed.py

@@ -1,5 +1,5 @@
 import os
-
+import csv
 from test.integration.base import DBTIntegrationTest, use_profile
 
 
@@ -311,4 +311,43 @@ class TestSimpleSeedWithDots(DBTIntegrationTest):
     @use_profile('postgres')
     def test_postgres_simple_seed(self):
         results = self.run_dbt(["seed"])
-        self.assertEqual(len(results),  1)
+        self.assertEqual(len(results),  1)
+        
+class TestSimpleBigSeedBatched(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "simple_seed_005"
+
+    @property
+    def models(self):
+        return "models"
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            "data-paths": ['data-big'],
+            'seeds': {
+                'quote_columns': False,
+            }
+        }
+
+    def test_big_batched_seed(self):
+        with open('data-big/my_seed.csv', 'w') as f:
+            writer = csv.writer(f)
+            writer.writerow(['id'])
+            for i in range(0, 20000):
+                writer.writerow([i])
+            
+        results = self.run_dbt(["seed"])
+        self.assertEqual(len(results),  1)
+
+
+    @use_profile('postgres')
+    def test_postgres_big_batched_seed(self):
+        self.test_big_batched_seed()
+    
+    @use_profile('snowflake')
+    def test_snowflake_big_batched_seed(self):
+        self.test_big_batched_seed()
+    

test/integration/008_schema_tests_test/macros-v2/override_get_test_macros/get_test_sql.sql

@@ -0,0 +1,10 @@
+{% macro get_test_sql(main_sql, fail_calc, warn_if, error_if, limit) -%}
+    select
+      {{ fail_calc }} as failures,
+      case when {{ fail_calc }} {{ warn_if }} then 1 else 0 end as should_warn,
+      case when {{ fail_calc }} {{ error_if }} then 1 else 0 end as should_error
+    from (
+      {{ main_sql }}
+      {{ "limit " ~ limit if limit != none }}
+    ) dbt_internal_test
+{%- endmacro %}

test/integration/008_schema_tests_test/macros-v2/override_get_test_macros_fail/get_test_sql.sql

@@ -0,0 +1,10 @@
+{% macro get_test_sql(main_sql, fail_calc, warn_if, error_if, limit) -%}
+    select
+      {{ fail_calc }} as failures,
+      case when {{ fail_calc }} {{ warn_if }} then 'x' else 'y' end as should_warn,
+      case when {{ fail_calc }} {{ error_if }} then 'x' else 'y' end as should_error
+    from (
+      {{ main_sql }}
+      {{ "limit " ~ limit if limit != none }}
+    ) dbt_internal_test
+{% endmacro %}

test/integration/008_schema_tests_test/models-v2/limit_null/schema.yml

@@ -0,0 +1,37 @@
+version: 2
+
+models:
+    - name: table_limit_null
+      description: "The table has 1 null value, and we're okay with that, until it's more than 1."
+      columns:
+        - name: favorite_color_full_list
+          description: "The favorite color"
+        - name: count
+          description: "The number of responses for this favorite color - purple will be null"
+          tests:
+            - not_null:
+                error_if: '>1'
+                warn_if: '>1'
+
+    - name: table_warning_limit_null
+      description: "The table has 1 null value, and we're okay with 1, but want to know of any."
+      columns:
+        - name: favorite_color_full_list
+          description: "The favorite color"
+        - name: count
+          description: "The number of responses for this favorite color - purple will be null"
+          tests:
+            - not_null:
+                error_if: '>1'
+
+    - name: table_failure_limit_null
+      description: "The table has 2 null values and that's not ok.  Fails."
+      columns:
+        - name: favorite_color_full_list
+          description: "The favorite color"
+        - name: count
+          description: "The number of responses for this favorite color - purple will be null"
+          tests:
+            - not_null:
+                error_if: '>1'
+          

test/integration/008_schema_tests_test/models-v2/limit_null/table_failure_limit_null.sql

@@ -0,0 +1,11 @@
+{{
+    config(
+        materialized='table'
+    )
+}}
+
+select * from {{ref('table_limit_null')}}
+
+UNION ALL
+
+select 'magenta' as favorite_color_full_list, null as count

test/integration/008_schema_tests_test/models-v2/limit_null/table_limit_null.sql

@@ -0,0 +1,13 @@
+{{
+    config(
+        materialized='table'
+    )
+}}
+
+select favorite_color as favorite_color_full_list, count(*) as count
+from {{ this.schema }}.seed
+group by 1
+
+UNION ALL
+
+select 'purple' as favorite_color_full_list, null as count

test/integration/008_schema_tests_test/models-v2/limit_null/table_warning_limit_null.sql

@@ -0,0 +1,7 @@
+{{
+    config(
+        materialized='table'
+    )
+}}
+
+select * from {{ref('table_limit_null')}}

test/integration/008_schema_tests_test/models-v2/override_get_test_models/my_model_failure.sql

@@ -0,0 +1,3 @@
+select * from {{ ref('my_model_pass') }}
+UNION ALL
+select null as id

test/integration/008_schema_tests_test/models-v2/override_get_test_models/my_model_pass.sql

@@ -0,0 +1,3 @@
+select 1 as id
+UNION ALL
+select null as id

test/integration/008_schema_tests_test/models-v2/override_get_test_models/my_model_warning.sql

@@ -0,0 +1 @@
+select * from {{ ref('my_model_pass') }}

test/integration/008_schema_tests_test/models-v2/override_get_test_models/schema.yml

@@ -0,0 +1,31 @@
+version: 2
+
+models:
+    - name: my_model_pass
+      description: "The table has 1 null values, and we're okay with that, until it's more than 1."
+      columns:
+        - name: id
+          description: "The number of responses for this favorite color - purple will be null"
+          tests:
+            - not_null:
+                error_if: '>1'
+                warn_if: '>1'
+
+    - name: my_model_warning
+      description: "The table has 1 null values, and we're okay with that, but let us know"
+      columns:
+        - name: id
+          description: "The number of responses for this favorite color - purple will be null"
+          tests:
+            - not_null:
+                error_if: '>1'
+
+    - name: my_model_failure
+      description: "The table has 2 null values, and we're not okay with that"
+      columns:
+        - name: id
+          description: "The number of responses for this favorite color - purple will be null"
+          tests:
+            - not_null:
+                error_if: '>1'
+

test/integration/008_schema_tests_test/models-v2/override_get_test_models_fail/my_model.sql

@@ -0,0 +1,3 @@
+select 1 as id
+UNION ALL
+select null as id

test/integration/008_schema_tests_test/models-v2/override_get_test_models_fail/schema.yml

@@ -0,0 +1,12 @@
+version: 2
+
+models:
+    - name: my_model
+      description: "The table has 1 null values, and we're not okay with that."
+      columns:
+        - name: id
+          description: "The number of responses for this favorite color - purple will be null"
+          tests:
+            - not_null
+
+

test/integration/008_schema_tests_test/test-context-where-subq-macros/custom_generic_test.sql

@@ -0,0 +1,12 @@
+/*{# This test will fail if get_where_subquery() is missing from TestContext + TestMacroNamespace #}*/
+
+{% test self_referential(model) %}
+
+    {%- set relation = api.Relation.create(schema=model.schema, identifier=model.table) -%}
+    {%- set columns = adapter.get_columns_in_relation(relation) -%}
+    {%- set columns_csv = columns | map(attribute='name') | list | join(', ') -%}
+
+    select {{ columns_csv }} from {{ model }}
+    limit 0
+
+{% endtest %}

test/integration/008_schema_tests_test/test-context-where-subq-models/model_a.sql

@@ -0,0 +1 @@
+select 1 as fun

test/integration/008_schema_tests_test/test-context-where-subq-models/schema.yml

@@ -0,0 +1,6 @@
+version: 2
+
+models:
+  - name: model_a
+    tests:
+      - self_referential

test/integration/008_schema_tests_test/test_schema_v2_tests.py

@@ -95,6 +95,241 @@ class TestSchemaTests(DBTIntegrationTest):
         for result in test_results:
             self.assertTestFailed(result)
 
+class TestLimitedSchemaTests(DBTIntegrationTest):
+
+    def setUp(self):
+        DBTIntegrationTest.setUp(self)
+        self.run_sql_file("seed.sql")
+
+    @property
+    def schema(self):
+        return "schema_tests_008"
+
+    @property
+    def models(self):
+        return "models-v2/limit_null"
+
+    def run_schema_validations(self):
+        args = FakeArgs()
+        test_task = TestTask(args, self.config)
+        return test_task.run()
+    
+    def assertTestFailed(self, result):
+        self.assertEqual(result.status, "fail")
+        self.assertFalse(result.skipped)
+        self.assertTrue(
+            result.failures > 0,
+            'test {} did not fail'.format(result.node.name)
+        )
+
+    def assertTestWarn(self, result):
+        self.assertEqual(result.status, "warn")
+        self.assertFalse(result.skipped)
+        self.assertTrue(
+            result.failures > 0,
+            'test {} passed without expected warning'.format(result.node.name)
+        )
+
+    def assertTestPassed(self, result):
+        self.assertEqual(result.status, "pass")
+        self.assertFalse(result.skipped)
+        self.assertEqual(
+            result.failures, 0,
+            'test {} failed'.format(result.node.name)
+        )
+
+    @use_profile('postgres')
+    def test_postgres_limit_schema_tests(self):
+        results = self.run_dbt(strict=False)
+        self.assertEqual(len(results), 3)
+        test_results = self.run_schema_validations()
+        self.assertEqual(len(test_results), 3)
+
+        for result in test_results:
+            # assert that all deliberately failing tests actually fail
+            if 'failure' in result.node.name:
+                self.assertTestFailed(result)
+            # assert that tests with warnings have them
+            elif 'warning' in result.node.name:
+                self.assertTestWarn(result)
+            # assert that actual tests pass
+            else:
+                self.assertTestPassed(result)
+        # warnings are also marked as failures
+        self.assertEqual(sum(x.failures for x in test_results), 3)
+
+
+class TestDefaultBoolType(DBTIntegrationTest):
+    # test with default True/False in get_test_sql macro
+
+    def setUp(self):
+        DBTIntegrationTest.setUp(self)
+
+    @property
+    def schema(self):
+        return "schema_tests_008"
+
+    @property
+    def models(self):
+        return "models-v2/override_get_test_models"
+
+    def run_schema_validations(self):
+        args = FakeArgs()
+        test_task = TestTask(args, self.config)
+        return test_task.run()
+
+    def assertTestFailed(self, result):
+        self.assertEqual(result.status, "fail")
+        self.assertFalse(result.skipped)
+        self.assertTrue(
+            result.failures > 0,
+            'test {} did not fail'.format(result.node.name)
+        )
+
+    def assertTestWarn(self, result):
+        self.assertEqual(result.status, "warn")
+        self.assertFalse(result.skipped)
+        self.assertTrue(
+            result.failures > 0,
+            'test {} passed without expected warning'.format(result.node.name)
+        )
+
+    def assertTestPassed(self, result):
+        self.assertEqual(result.status, "pass")
+        self.assertFalse(result.skipped)
+        self.assertEqual(
+            result.failures, 0,
+            'test {} failed'.format(result.node.name)
+        )
+
+    @use_profile('postgres')
+    def test_postgres_limit_schema_tests(self):
+        results = self.run_dbt(strict=False)
+        self.assertEqual(len(results), 3)
+        test_results = self.run_schema_validations()
+        self.assertEqual(len(test_results), 3)
+
+        for result in test_results:
+            # assert that all deliberately failing tests actually fail
+            if 'failure' in result.node.name:
+                self.assertTestFailed(result)
+            # assert that tests with warnings have them
+            elif 'warning' in result.node.name:
+                self.assertTestWarn(result)
+            # assert that actual tests pass
+            else:
+                self.assertTestPassed(result)
+        # warnings are also marked as failures
+        self.assertEqual(sum(x.failures for x in test_results), 3)
+
+
+class TestOtherBoolType(DBTIntegrationTest):
+    # test with expected 0/1 in custom get_test_sql macro
+
+    def setUp(self):
+        DBTIntegrationTest.setUp(self)
+
+    @property
+    def schema(self):
+        return "schema_tests_008"
+
+    @property
+    def models(self):
+        return "models-v2/override_get_test_models"
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            "macro-paths": ["macros-v2/override_get_test_macros"],
+        }
+
+    def run_schema_validations(self):
+        args = FakeArgs()
+        test_task = TestTask(args, self.config)
+        return test_task.run()
+
+    def assertTestFailed(self, result):
+        self.assertEqual(result.status, "fail")
+        self.assertFalse(result.skipped)
+        self.assertTrue(
+            result.failures > 0,
+            'test {} did not fail'.format(result.node.name)
+        )
+
+    def assertTestWarn(self, result):
+        self.assertEqual(result.status, "warn")
+        self.assertFalse(result.skipped)
+        self.assertTrue(
+            result.failures > 0,
+            'test {} passed without expected warning'.format(result.node.name)
+        )
+
+    def assertTestPassed(self, result):
+        self.assertEqual(result.status, "pass")
+        self.assertFalse(result.skipped)
+        self.assertEqual(
+            result.failures, 0,
+            'test {} failed'.format(result.node.name)
+        )
+
+    @use_profile('postgres')
+    def test_postgres_limit_schema_tests(self):
+        results = self.run_dbt(strict=False)
+        self.assertEqual(len(results), 3)
+        test_results = self.run_schema_validations()
+        self.assertEqual(len(test_results), 3)
+
+        for result in test_results:
+            # assert that all deliberately failing tests actually fail
+            if 'failure' in result.node.name:
+                self.assertTestFailed(result)
+            # assert that tests with warnings have them
+            elif 'warning' in result.node.name:
+                self.assertTestWarn(result)
+            # assert that actual tests pass
+            else:
+                self.assertTestPassed(result)
+        # warnings are also marked as failures
+        self.assertEqual(sum(x.failures for x in test_results), 3)
+
+
+class TestNonBoolType(DBTIntegrationTest):
+    # test with invalid 'x'/'y' in custom get_test_sql macro
+    def setUp(self):
+        DBTIntegrationTest.setUp(self)
+
+    @property
+    def schema(self):
+        return "schema_tests_008"
+
+    @property
+    def models(self):
+        return "models-v2/override_get_test_models_fail"
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            "macro-paths": ["macros-v2/override_get_test_macros_fail"],
+        }
+
+    def run_schema_validations(self):
+        args = FakeArgs()
+
+        test_task = TestTask(args, self.config)
+        return test_task.run()
+
+    @use_profile('postgres')
+    def test_postgres_limit_schema_tests(self):
+        results = self.run_dbt()
+        self.assertEqual(len(results), 1)
+        run_result = self.run_dbt(['test'], expect_pass=False)
+        results = run_result.results
+        self.assertEqual(len(results), 1)
+        self.assertEqual(results[0].status, TestStatus.Error)
+        self.assertRegex(results[0].message, r"'get_test_sql' returns 'x'")
+
 
 class TestMalformedSchemaTests(DBTIntegrationTest):
 
@@ -566,3 +801,28 @@ class TestInvalidSchema(DBTIntegrationTest):
             results = self.run_dbt()
         self.assertRegex(str(exc.exception), r"'models' is not a list")
 
+class TestSchemaTestContextWhereSubq(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "schema_tests_008"
+
+    @property
+    def models(self):
+        return "test-context-where-subq-models"
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            "macro-paths": ["test-context-where-subq-macros"],
+        }
+
+    @use_profile('postgres')
+    def test_postgres_test_context_tests(self):
+        # This test tests that get_where_subquery() is included in TestContext + TestMacroNamespace,
+        # otherwise api.Relation.create() will return an error
+        results = self.run_dbt()
+        self.assertEqual(len(results), 1)
+
+        results = self.run_dbt(['test'])
+        self.assertEqual(len(results), 1)

test/integration/016_macro_tests/override-postgres-get-columns-macros/macros.sql

@@ -0,0 +1,3 @@
+{% macro postgres__get_columns_in_relation(relation) %}
+	{{ return('a string') }}
+{% endmacro %}

test/integration/016_macro_tests/test_macros.py

@@ -133,6 +133,64 @@ class TestMacroOverrideBuiltin(DBTIntegrationTest):
         self.run_dbt()
 
 
+class TestMacroOverridePackage(DBTIntegrationTest):
+    """
+    The macro in `override-postgres-get-columns-macros` should override the
+    `get_columns_in_relation` macro by default.
+    """
+
+    @property
+    def schema(self):
+        return "test_macros_016"
+
+    @property
+    def models(self):
+        return 'override-get-columns-models'
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            'macro-paths': ['override-postgres-get-columns-macros'],
+        }
+
+    @use_profile('postgres')
+    def test_postgres_overrides(self):
+        # the first time, the model doesn't exist
+        self.run_dbt()
+        self.run_dbt()
+
+
+class TestMacroNotOverridePackage(DBTIntegrationTest):
+    """
+    The macro in `override-postgres-get-columns-macros` does NOT override the
+    `get_columns_in_relation` macro because we tell dispatch to not look at the
+    postgres macros.
+    """
+
+    @property
+    def schema(self):
+        return "test_macros_016"
+
+    @property
+    def models(self):
+        return 'override-get-columns-models'
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            'macro-paths': ['override-postgres-get-columns-macros'],
+            'dispatch': [{'macro_namespace': 'dbt', 'search_order': ['dbt']}],
+        }
+
+    @use_profile('postgres')
+    def test_postgres_overrides(self):
+        # the first time, the model doesn't exist
+        self.run_dbt(expect_pass=False)
+        self.run_dbt(expect_pass=False)
+
+
 class TestDispatchMacroOverrideBuiltin(TestMacroOverrideBuiltin):
     # test the same functionality as above, but this time,
     # dbt.get_columns_in_relation will dispatch to a default__ macro

test/integration/029_docs_generate_tests/test_docs_generate.py

@@ -1093,7 +1093,7 @@ class TestDocsGenerate(DBTIntegrationTest):
         )
 
         return {
-            'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v2.json',
+            'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v3.json',
             'dbt_version': dbt.version.__version__,
             'nodes': {
                 'model.test.model': {
@@ -1680,7 +1680,7 @@ class TestDocsGenerate(DBTIntegrationTest):
         snapshot_path = self.dir('snapshot/snapshot_seed.sql')
 
         return {
-            'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v2.json',
+            'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v3.json',
             'dbt_version': dbt.version.__version__,
             'nodes': {
                 'model.test.ephemeral_copy': {
@@ -2203,7 +2203,7 @@ class TestDocsGenerate(DBTIntegrationTest):
         my_schema_name = self.unique_schema()
 
         return {
-            'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v2.json',
+            'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v3.json',
             'dbt_version': dbt.version.__version__,
             'nodes': {
                 'model.test.clustered': {
@@ -2695,7 +2695,7 @@ class TestDocsGenerate(DBTIntegrationTest):
         snapshot_path = self.dir('snapshot/snapshot_seed.sql')
 
         return {
-            'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v2.json',
+            'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v3.json',
             'dbt_version': dbt.version.__version__,
             'nodes': {
                 'model.test.model': {
@@ -2959,7 +2959,7 @@ class TestDocsGenerate(DBTIntegrationTest):
             elif key == 'metadata':
                 metadata = manifest['metadata']
                 self.verify_metadata(
-                    metadata, 'https://schemas.getdbt.com/dbt/manifest/v2.json')
+                    metadata, 'https://schemas.getdbt.com/dbt/manifest/v3.json')
                 assert 'project_id' in metadata and metadata[
                     'project_id'] == '098f6bcd4621d373cade4e832627b4f6'
                 assert 'send_anonymous_usage_stats' in metadata and metadata[
@@ -3100,7 +3100,7 @@ class TestDocsGenerate(DBTIntegrationTest):
         run_results = _read_json('./target/run_results.json')
         assert 'metadata' in run_results
         self.verify_metadata(
-            run_results['metadata'], 'https://schemas.getdbt.com/dbt/run-results/v2.json')
+            run_results['metadata'], 'https://schemas.getdbt.com/dbt/run-results/v3.json')
         self.assertIn('elapsed_time', run_results)
         self.assertGreater(run_results['elapsed_time'], 0)
         self.assertTrue(

test/integration/042_sources_test/test_sources.py

@@ -248,7 +248,7 @@ class TestSourceFreshness(SuccessfulSourcesTest):
         assert isinstance(data['elapsed_time'], float)
         self.assertBetween(data['metadata']['generated_at'],
                            self.freshness_start_time)
-        assert data['metadata']['dbt_schema_version'] == 'https://schemas.getdbt.com/dbt/sources/v1.json'
+        assert data['metadata']['dbt_schema_version'] == 'https://schemas.getdbt.com/dbt/sources/v2.json'
         assert data['metadata']['dbt_version'] == dbt.version.__version__
         assert data['metadata']['invocation_id'] == dbt.tracking.active_user.invocation_id
         key = 'key'

test/integration/062_defer_state_test/macros/infinite_macros.sql

@@ -0,0 +1,13 @@
+{# trigger infinite recursion if not handled #}
+
+{% macro my_infinitely_recursive_macro() %}
+  {{ return(adapter.dispatch('my_infinitely_recursive_macro')()) }}
+{% endmacro %}
+
+{% macro default__my_infinitely_recursive_macro() %}
+    {% if unmet_condition %}
+        {{ my_infinitely_recursive_macro() }}
+    {% else %}
+        {{ return('') }}
+    {% endif %}
+{% endmacro %}

test/integration/062_defer_state_test/models/view_model.sql

@@ -1 +1,4 @@
 select * from {{ ref('seed') }}
+
+-- establish a macro dependency that trips infinite recursion if not handled
+-- depends on: {{ my_infinitely_recursive_macro() }}

test/integration/066_test_selection_tests/test_selection_expansion.py

@@ -1,4 +1,5 @@
 from test.integration.base import DBTIntegrationTest, FakeArgs, use_profile
+import yaml
 
 from dbt.task.test import TestTask
 from dbt.task.list import ListTask
@@ -20,12 +21,18 @@ class TestSelectionExpansion(DBTIntegrationTest):
             "test-paths": ["tests"]
         }
 
-    def list_tests_and_assert(self, include, exclude, expected_tests):
+    def list_tests_and_assert(self, include, exclude, expected_tests, greedy=False, selector_name=None):
         list_args = [ 'ls', '--resource-type', 'test']
         if include:
             list_args.extend(('--select', include))
         if exclude:
             list_args.extend(('--exclude', exclude))
+        if exclude:
+            list_args.extend(('--exclude', exclude))
+        if greedy:
+            list_args.append('--greedy')
+        if selector_name:
+            list_args.extend(('--selector', selector_name))
         
         listed = self.run_dbt(list_args)
         assert len(listed) == len(expected_tests)
@@ -34,7 +41,7 @@ class TestSelectionExpansion(DBTIntegrationTest):
         assert sorted(test_names) == sorted(expected_tests)
 
     def run_tests_and_assert(
-        self, include, exclude, expected_tests, schema = False, data = False
+        self, include, exclude, expected_tests, schema=False, data=False, greedy=False, selector_name=None
     ):
         results = self.run_dbt(['run'])
         self.assertEqual(len(results), 2)
@@ -48,6 +55,10 @@ class TestSelectionExpansion(DBTIntegrationTest):
             test_args.append('--schema')
         if data:
             test_args.append('--data')
+        if greedy:
+            test_args.append('--greedy')
+        if selector_name:
+            test_args.extend(('--selector', selector_name))
         
         results = self.run_dbt(test_args)
         tests_run = [r.node.name for r in results]
@@ -228,3 +239,80 @@ class TestSelectionExpansion(DBTIntegrationTest):
         
         self.list_tests_and_assert(select, exclude, expected)
         self.run_tests_and_assert(select, exclude, expected)
+
+    @use_profile('postgres')
+    def test__postgres__model_a_greedy(self):
+        select = 'model_a'
+        exclude = None
+        greedy = True
+        expected = [
+            'cf_a_b', 'cf_a_src', 'just_a',
+            'relationships_model_a_fun__fun__ref_model_b_',
+            'relationships_model_a_fun__fun__source_my_src_my_tbl_',
+            'unique_model_a_fun'
+        ]
+            
+        self.list_tests_and_assert(select, exclude, expected, greedy)
+        self.run_tests_and_assert(select, exclude, expected, greedy=greedy)
+
+    @use_profile('postgres')
+    def test__postgres__model_a_greedy_exclude_unique_tests(self):
+        select = 'model_a'
+        exclude = 'test_name:unique'
+        greedy = True
+        expected = [
+            'cf_a_b', 'cf_a_src', 'just_a',
+            'relationships_model_a_fun__fun__ref_model_b_',
+            'relationships_model_a_fun__fun__source_my_src_my_tbl_',
+        ]
+            
+        self.list_tests_and_assert(select, exclude, expected, greedy)
+        self.run_tests_and_assert(select, exclude, expected, greedy=greedy)
+
+class TestExpansionWithSelectors(TestSelectionExpansion):
+
+    @property
+    def selectors_config(self):
+        return yaml.safe_load('''
+            selectors:
+            - name: model_a_greedy_none
+              definition:
+                method: fqn
+                value: model_a
+            - name: model_a_greedy_false
+              definition:
+                method: fqn
+                value: model_a
+                greedy: false
+            - name: model_a_greedy_true
+              definition:
+                method: fqn
+                value: model_a
+                greedy: true
+        ''')
+
+    @use_profile('postgres')
+    def test__postgres__selector_model_a_not_greedy(self):
+        expected = ['just_a','unique_model_a_fun']
+        
+        # when greedy is not specified, so implicitly False
+        self.list_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_none')
+        self.run_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_none')
+
+        # when greedy is explicitly False
+        self.list_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_false')
+        self.run_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_false')
+
+
+    @use_profile('postgres')
+    def test__postgres__selector_model_a_yes_greedy(self):
+        expected = [
+            'cf_a_b', 'cf_a_src', 'just_a',
+            'relationships_model_a_fun__fun__ref_model_b_',
+            'relationships_model_a_fun__fun__source_my_src_my_tbl_',
+            'unique_model_a_fun'
+        ]
+
+        # when greedy is explicitly False
+        self.list_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_true')
+        self.run_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_true')

test/integration/068_partial_parsing_tests/extra-files/model_three_disabled.sql

@@ -0,0 +1,12 @@
+{{ config(materialized='table', enabled=False) }}
+ 
+with source_data as (
+ 
+    select 1 as id
+    union all
+    select null as id
+ 
+)
+ 
+select *
+from source_data

test/integration/068_partial_parsing_tests/extra-files/model_three_disabled2.sql

@@ -0,0 +1,13 @@
+- Disabled model
+{{ config(materialized='table', enabled=False) }}
+ 
+with source_data as (
+ 
+    select 1 as id
+    union all
+    select null as id
+ 
+)
+ 
+select *
+from source_data

test/integration/068_partial_parsing_tests/extra-files/model_three_modified.sql

@@ -0,0 +1,14 @@
+{{ config(materialized='table') }}
+ 
+with source_data as (
+
+    {#- This is model three #}
+ 
+    select 1 as id
+    union all
+    select null as id
+ 
+)
+ 
+select *
+from source_data

test/integration/068_partial_parsing_tests/extra-files/snapshot.sql

@@ -0,0 +1,15 @@
+{% snapshot orders_snapshot %}
+
+{{
+    config(
+      target_schema=schema,
+      strategy='check',
+      unique_key='id',
+      check_cols=['status'],
+    )
+}}
+
+select * from {{ ref('orders') }}
+
+{% endsnapshot %}
+

test/integration/068_partial_parsing_tests/extra-files/snapshot2.sql

@@ -0,0 +1,14 @@
+{% snapshot orders_snapshot %}
+
+{{
+    config(
+      target_schema=schema,
+      strategy='check',
+      unique_key='id',
+      check_cols=['status'],
+    )
+}}
+select * from {{ ref('orders') }}
+
+{% endsnapshot %}
+

test/integration/068_partial_parsing_tests/models-d/orders.sql

@@ -0,0 +1 @@
+select 1 as id, 101 as user_id, 'pending' as status

test/integration/068_partial_parsing_tests/test_partial_parsing.py

@@ -2,7 +2,7 @@ from dbt.exceptions import CompilationException
 from dbt.contracts.graph.manifest import Manifest
 from dbt.contracts.files import ParseFileType
 from dbt.contracts.results import TestStatus
-from test.integration.base import DBTIntegrationTest, use_profile, normalize
+from test.integration.base import DBTIntegrationTest, use_profile, normalize, get_manifest
 import shutil
 import os
 
@@ -10,16 +10,6 @@ import os
 # Note: every test case needs to have separate directories, otherwise
 # they will interfere with each other when tests are multi-threaded
 
-def get_manifest():
-    path = './target/partial_parse.msgpack'
-    if os.path.exists(path):
-        with open(path, 'rb') as fp:
-            manifest_mp = fp.read()
-        manifest: Manifest = Manifest.from_msgpack(manifest_mp)
-        return manifest
-    else:
-        return None
-
 class TestModels(DBTIntegrationTest):
 
     @property
@@ -78,6 +68,15 @@ class TestModels(DBTIntegrationTest):
         unique_test_id = tests[0]
         self.assertIn(unique_test_id, manifest.nodes)
 
+        # modify model sql file, ensure description still there
+        shutil.copyfile('extra-files/model_three_modified.sql', 'models-a/model_three.sql')
+        results = self.run_dbt(["--partial-parse", "run"])
+        manifest = get_manifest()
+        model_id = 'model.test.model_three'
+        self.assertIn(model_id, manifest.nodes)
+        model_three_node = manifest.nodes[model_id]
+        self.assertEqual(model_three_node.description, 'The third model')
+
         # Change the model 3 test from unique to not_null
         shutil.copyfile('extra-files/models-schema2b.yml', 'models-a/schema.yml')
         results = self.run_dbt(["--partial-parse", "test"], expect_pass=False)
@@ -187,6 +186,41 @@ class TestModels(DBTIntegrationTest):
         results = self.run_dbt(["--partial-parse", "run"])
         self.assertEqual(len(results), 3)
 
+        # Disable model_three
+        shutil.copyfile('extra-files/model_three_disabled.sql', 'models-a/model_three.sql')
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 2)
+        manifest = get_manifest()
+        model_id = 'model.test.model_three'
+        found_in_disabled = False
+        for model in manifest.disabled:
+            if model_id == model.unique_id:
+                found_in_disabled = True
+        self.assertTrue(found_in_disabled)
+        self.assertNotIn(model_id, manifest.nodes)
+
+        # Edit disabled model three
+        shutil.copyfile('extra-files/model_three_disabled2.sql', 'models-a/model_three.sql')
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 2)
+        manifest = get_manifest()
+        model_id = 'model.test.model_three'
+        found_in_disabled = False
+        for model in manifest.disabled:
+            if model_id == model.unique_id:
+                found_in_disabled = True
+        self.assertTrue(found_in_disabled)
+        self.assertNotIn(model_id, manifest.nodes)
+
+        # Remove disabled from model three
+        shutil.copyfile('extra-files/model_three.sql', 'models-a/model_three.sql')
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 3)
+        manifest = get_manifest()
+        model_id = 'model.test.model_three'
+        self.assertIn(model_id, manifest.nodes)
+        self.assertNotIn(model_id, manifest.disabled)
+
     def tearDown(self):
         if os.path.exists(normalize('models-a/model_two.sql')):
             os.remove(normalize('models-a/model_two.sql'))
@@ -468,3 +502,53 @@ class TestMacros(DBTIntegrationTest):
         self.assertEqual(results[1].status, TestStatus.Fail)
         self.assertEqual(results[1].node.config.severity, 'ERROR')
 
+
+class TestSnapshots(DBTIntegrationTest):
+
+    @property
+    def schema(self):
+        return "test_068A"
+
+    @property
+    def models(self):
+        return "models-d"
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            'snapshot-paths': ['snapshots-d'],
+        }
+
+    def tearDown(self):
+        if os.path.exists(normalize('snapshots-d/snapshot.sql')):
+            os.remove(normalize('snapshots-d/snapshot.sql'))
+
+    @use_profile('postgres')
+    def test_postgres_pp_snapshots(self):
+
+        # initial run 
+        results = self.run_dbt() 
+        self.assertEqual(len(results), 1)
+
+        # add snapshot
+        shutil.copyfile('extra-files/snapshot.sql', 'snapshots-d/snapshot.sql')
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 1)
+        manifest = get_manifest()
+        snapshot_id = 'snapshot.test.orders_snapshot'
+        self.assertIn(snapshot_id, manifest.nodes)
+
+        # run snapshot
+        results = self.run_dbt(["--partial-parse", "snapshot"])
+        self.assertEqual(len(results), 1)
+
+        # modify snapshot
+        shutil.copyfile('extra-files/snapshot2.sql', 'snapshots-d/snapshot.sql')
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 1)
+
+        # delete snapshot
+        os.remove(normalize('snapshots-d/snapshot.sql'))
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 1)

test/integration/069_build_test/models-interdependent/model_a.sql

@@ -0,0 +1 @@
+select 1 as id

test/integration/069_build_test/models-interdependent/model_c.sql

@@ -0,0 +1 @@
+select * from {{ ref('model_b') }}

test/integration/069_build_test/models-interdependent/schema.yml

@@ -0,0 +1,41 @@
+version: 2
+
+models:
+  - name: model_a
+    columns:
+      - name: id
+        tests:
+          - unique
+          - not_null
+          - relationships:
+              to: ref('model_b')
+              field: id
+          - relationships:
+              to: ref('model_c')
+              field: id
+
+  - name: model_b
+    columns:
+      - name: id
+        tests:
+          - unique
+          - not_null
+          - relationships:
+              to: ref('model_a')
+              field: id
+          - relationships:
+              to: ref('model_c')
+              field: id
+
+  - name: model_c
+    columns:
+      - name: id
+        tests:
+          - unique
+          - not_null
+          - relationships:
+              to: ref('model_a')
+              field: id
+          - relationships:
+              to: ref('model_b')
+              field: id

test/integration/069_build_test/models-simple-blocking/model_a.sql

@@ -0,0 +1 @@
+select null as id

test/integration/069_build_test/models-simple-blocking/model_b.sql

@@ -0,0 +1 @@
+select * from {{ ref('model_a') }}

test/integration/069_build_test/models-simple-blocking/schema.yml

@@ -0,0 +1,8 @@
+version: 2
+
+models:
+  - name: model_a
+    columns:
+      - name: id
+        tests:
+          - not_null

test/integration/069_build_test/test-files/model_b.sql

@@ -0,0 +1 @@
+select * from {{ ref('model_a') }} 

test/integration/069_build_test/test-files/model_b_null.sql

@@ -0,0 +1 @@
+select null from {{ ref('model_a') }} 

test/integration/069_build_test/test_build.py

@@ -1,5 +1,7 @@
-from test.integration.base import DBTIntegrationTest, use_profile
+from test.integration.base import DBTIntegrationTest, use_profile, normalize
 import yaml
+import shutil
+import os
 
 
 class TestBuildBase(DBTIntegrationTest):
@@ -79,3 +81,63 @@ class TestCircularRelationshipTestsBuild(TestBuildBase):
         actual = [r.status for r in results]
         expected = ['success']*7 + ['pass']*2
         self.assertEqual(sorted(actual), sorted(expected))
+
+
+class TestSimpleBlockingTest(TestBuildBase):
+    @property
+    def models(self):
+        return "models-simple-blocking"
+        
+    @property
+    def project_config(self):
+        return {
+            "config-version": 2,
+            "snapshot-paths": ["does-not-exist"],
+            "data-paths": ["does-not-exist"],
+        }
+
+    @use_profile("postgres")
+    def test__postgres_simple_blocking_test(self):
+        """ Ensure that a failed test on model_a always blocks model_b """
+        results = self.build(expect_pass=False)
+        actual = [r.status for r in results]
+        expected = ['success', 'fail', 'skipped']
+        self.assertEqual(sorted(actual), sorted(expected))
+
+
+class TestInterdependentModels(TestBuildBase):
+
+    @property
+    def project_config(self):
+        return {
+            "config-version": 2,
+            "snapshot-paths": ["snapshots-none"],
+            "seeds": {
+                "quote_columns": False,
+            },
+        }
+
+    @property
+    def models(self):
+        return "models-interdependent"
+
+    def tearDown(self):
+        if os.path.exists(normalize('models-interdependent/model_b.sql')):
+            os.remove(normalize('models-interdependent/model_b.sql'))
+
+
+    @use_profile("postgres")
+    def test__postgres_interdependent_models(self):
+        # check that basic build works
+        shutil.copyfile('test-files/model_b.sql', 'models-interdependent/model_b.sql')
+        results = self.build()
+        self.assertEqual(len(results), 16)
+
+        # return null from model_b
+        shutil.copyfile('test-files/model_b_null.sql', 'models-interdependent/model_b.sql')
+        results = self.build(expect_pass=False)
+        self.assertEqual(len(results), 16)
+        actual = [str(r.status) for r in results]
+        expected = ['error']*4 + ['skipped']*7 + ['pass']*2 + ['success']*3
+        self.assertEqual(sorted(actual), sorted(expected))
+

test/integration/070_incremental_schema_tests/models/incremental_sync_remove_only.sql

@@ -0,0 +1,29 @@
+{{
+    config(
+        materialized='incremental',
+        unique_key='id',
+        on_schema_change='sync_all_columns'
+        
+    )
+}}
+
+WITH source_data AS (SELECT * FROM {{ ref('model_a') }} )
+
+{% set string_type = 'string' if target.type == 'bigquery' else 'varchar(10)' %}
+
+{% if is_incremental() %}
+
+SELECT id, 
+       cast(field1 as {{string_type}}) as field1
+
+FROM source_data WHERE id NOT IN (SELECT id from {{ this }} )
+
+{% else %}
+
+select id, 
+       cast(field1 as {{string_type}}) as field1, 
+       cast(field2 as {{string_type}}) as field2
+
+from source_data where id <= 3
+
+{% endif %}

test/integration/070_incremental_schema_tests/models/incremental_sync_remove_only_target.sql

@@ -0,0 +1,17 @@
+{{ 
+    config(materialized='table') 
+}}
+
+with source_data as (
+
+    select * from {{ ref('model_a') }}
+
+)
+
+{% set string_type = 'string' if target.type == 'bigquery' else 'varchar(10)' %}
+
+select id
+       ,cast(field1 as {{string_type}}) as field1
+
+from source_data
+order by id 

test/integration/070_incremental_schema_tests/test_incremental_schema.py

@@ -1,7 +1,7 @@
 from test.integration.base import DBTIntegrationTest, FakeArgs, use_profile
 
 
-class TestSelectionExpansion(DBTIntegrationTest):
+class TestIncrementalSchemaChange(DBTIntegrationTest):
     @property
     def schema(self):
         return "test_incremental_schema_069"
@@ -17,24 +17,11 @@ class TestSelectionExpansion(DBTIntegrationTest):
             "test-paths": ["tests"]
         }
 
-    def list_tests_and_assert(self, include, exclude, expected_tests):
-        list_args = [ 'ls', '--resource-type', 'test']
-        if include:
-            list_args.extend(('--select', include))
-        if exclude:
-            list_args.extend(('--exclude', exclude))
-        
-        listed = self.run_dbt(list_args)
-        print(listed)
-        assert len(listed) == len(expected_tests)
-        
-        test_names = [name.split('.')[2] for name in listed]
-        assert sorted(test_names) == sorted(expected_tests)
-
-    def run_tests_and_assert(
-        self, include, exclude, expected_tests, compare_source, compare_target, schema = False, data = False
-    ):  
+    def run_twice_and_assert(
+        self, include, compare_source, compare_target
+    ):
 
+        # dbt run (twice)
         run_args = ['run']
         if include:
             run_args.extend(('--models', include))
@@ -44,74 +31,33 @@ class TestSelectionExpansion(DBTIntegrationTest):
 
         self.assertEqual(len(results_one), 3)
         self.assertEqual(len(results_two), 3)
-         
-        test_args = ['test']
-        if include:
-            test_args.extend(('--models', include))
-        if exclude:
-            test_args.extend(('--exclude', exclude))
-        if schema:
-            test_args.append('--schema')
-        if data:
-            test_args.append('--data')
         
-        results = self.run_dbt(test_args)
-        tests_run = [r.node.name for r in results]
-        assert len(tests_run) == len(expected_tests)
-        assert sorted(tests_run) == sorted(expected_tests)
         self.assertTablesEqual(compare_source, compare_target)
 
     def run_incremental_ignore(self):
         select = 'model_a incremental_ignore incremental_ignore_target'
         compare_source = 'incremental_ignore'
         compare_target = 'incremental_ignore_target'
-        exclude = None
-        expected = [
-            'select_from_a',
-            'select_from_incremental_ignore',
-            'select_from_incremental_ignore_target',
-            'unique_model_a_id',
-            'unique_incremental_ignore_id',
-            'unique_incremental_ignore_target_id'
-        ]
-            
-        self.list_tests_and_assert(select, exclude, expected)
-        self.run_tests_and_assert(select, exclude, expected, compare_source, compare_target)
-    
+        self.run_twice_and_assert(select, compare_source, compare_target)
+
     def run_incremental_append_new_columns(self):
         select = 'model_a incremental_append_new_columns incremental_append_new_columns_target'
         compare_source = 'incremental_append_new_columns'
         compare_target = 'incremental_append_new_columns_target'
-        exclude = None
-        expected = [
-            'select_from_a',
-            'select_from_incremental_append_new_columns',
-            'select_from_incremental_append_new_columns_target',
-            'unique_model_a_id',
-            'unique_incremental_append_new_columns_id',
-            'unique_incremental_append_new_columns_target_id'
-        ]
-            
-        self.list_tests_and_assert(select, exclude, expected)
-        self.run_tests_and_assert(select, exclude, expected, compare_source, compare_target)
-    
+        self.run_twice_and_assert(select, compare_source, compare_target)
+
     def run_incremental_sync_all_columns(self):
         select = 'model_a incremental_sync_all_columns incremental_sync_all_columns_target'
         compare_source = 'incremental_sync_all_columns'
         compare_target = 'incremental_sync_all_columns_target'
-        exclude = None
-        expected = [
-            'select_from_a',
-            'select_from_incremental_sync_all_columns',
-            'select_from_incremental_sync_all_columns_target',
-            'unique_model_a_id',
-            'unique_incremental_sync_all_columns_id',
-            'unique_incremental_sync_all_columns_target_id'
-        ]
-            
-        self.list_tests_and_assert(select, exclude, expected)
-        self.run_tests_and_assert(select, exclude, expected, compare_source, compare_target)
+        self.run_twice_and_assert(select, compare_source, compare_target)
         
+    def run_incremental_sync_remove_only(self):
+        select = 'model_a incremental_sync_remove_only incremental_sync_remove_only_target'
+        compare_source = 'incremental_sync_remove_only'
+        compare_target = 'incremental_sync_remove_only_target'
+        self.run_twice_and_assert(select, compare_source, compare_target)
+
     def run_incremental_fail_on_schema_change(self):
         select = 'model_a incremental_fail'
         results_one = self.run_dbt(['run', '--models', select, '--full-refresh'])
@@ -130,7 +76,8 @@ class TestSelectionExpansion(DBTIntegrationTest):
     @use_profile('postgres')
     def test__postgres__run_incremental_sync_all_columns(self):
         self.run_incremental_sync_all_columns()
-        
+        self.run_incremental_sync_remove_only()
+
     @use_profile('postgres')
     def test__postgres__run_incremental_fail_on_schema_change(self):
         self.run_incremental_fail_on_schema_change()
@@ -147,6 +94,7 @@ class TestSelectionExpansion(DBTIntegrationTest):
     @use_profile('redshift')
     def test__redshift__run_incremental_sync_all_columns(self):
         self.run_incremental_sync_all_columns()
+        self.run_incremental_sync_remove_only()
 
     @use_profile('redshift')
     def test__redshift__run_incremental_fail_on_schema_change(self):
@@ -164,6 +112,7 @@ class TestSelectionExpansion(DBTIntegrationTest):
     @use_profile('snowflake')
     def test__snowflake__run_incremental_sync_all_columns(self):
         self.run_incremental_sync_all_columns()
+        self.run_incremental_sync_remove_only()
 
     @use_profile('snowflake')
     def test__snowflake__run_incremental_fail_on_schema_change(self):
@@ -181,6 +130,7 @@ class TestSelectionExpansion(DBTIntegrationTest):
     @use_profile('bigquery')
     def test__bigquery__run_incremental_sync_all_columns(self):
         self.run_incremental_sync_all_columns()
+        self.run_incremental_sync_remove_only()
         
     @use_profile('bigquery')
     def test__bigquery__run_incremental_fail_on_schema_change(self):

test/unit/test_graph_selection.py

@@ -120,7 +120,7 @@ def test_run_specs(include, exclude, expected):
     manifest = _get_manifest(graph)
     selector = graph_selector.NodeSelector(graph, manifest)
     spec = graph_cli.parse_difference(include, exclude)
-    selected = selector.select_nodes(spec)
+    selected, _ = selector.select_nodes(spec)
 
     assert selected == expected
 

test/unit/test_manifest.py

@@ -273,7 +273,7 @@ class ManifestTest(unittest.TestCase):
                 'child_map': {},
                 'metadata': {
                     'generated_at': '2018-02-14T09:15:13Z',
-                    'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v2.json',
+                    'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v3.json',
                     'dbt_version': dbt.version.__version__,
                     'env': {ENV_KEY_NAME: 'value'},
                     # invocation_id is None, so it will not be present
@@ -419,7 +419,7 @@ class ManifestTest(unittest.TestCase):
                 'docs': {},
                 'metadata': {
                     'generated_at': '2018-02-14T09:15:13Z',
-                    'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v2.json',
+                    'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v3.json',
                     'dbt_version': dbt.version.__version__,
                     'project_id': '098f6bcd4621d373cade4e832627b4f6',
                     'user_id': 'cfc9500f-dc7f-4c83-9ea7-2c581c1b38cf',
@@ -662,7 +662,7 @@ class MixedManifestTest(unittest.TestCase):
                 'child_map': {},
                 'metadata': {
                     'generated_at': '2018-02-14T09:15:13Z',
-                    'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v2.json',
+                    'dbt_schema_version': 'https://schemas.getdbt.com/dbt/manifest/v3.json',
                     'dbt_version': dbt.version.__version__,
                     'invocation_id': '01234567-0123-0123-0123-0123456789ab',
                     'env': {ENV_KEY_NAME: 'value'},

test/unit/test_parser.py

@@ -5,9 +5,11 @@ from unittest import mock
 import os
 import yaml
 
+from copy import deepcopy
 import dbt.flags
 import dbt.parser
 from dbt import tracking
+from dbt.context.context_config import ContextConfig
 from dbt.exceptions import CompilationException
 from dbt.parser import (
     ModelParser, MacroParser, DataTestParser, SchemaParser,
@@ -32,6 +34,9 @@ from dbt.contracts.graph.parsed import (
     UnpatchedSourceDefinition
 )
 from dbt.contracts.graph.unparsed import Docs
+from dbt.parser.models import (
+    _get_config_call_dict, _get_exp_sample_result, _get_sample_result
+)
 import itertools
 from .utils import config_from_parts_or_dicts, normalize, generate_name_macros, MockNode, MockSource, MockDocumentation
 
@@ -573,6 +578,144 @@ class StaticModelParserTest(BaseParserTest):
         assert(self.parser._has_banned_macro(node))
 
 
+class StaticModelParserUnitTest(BaseParserTest):
+
+    def setUp(self):
+        super().setUp()
+        self.parser = ModelParser(
+            project=self.snowplow_project_config,
+            manifest=self.manifest,
+            root_project=self.root_project_config,
+        )
+        self.example_node = ParsedModelNode(
+            alias='model_1',
+            name='model_1',
+            database='test',
+            schema='analytics',
+            resource_type=NodeType.Model,
+            unique_id='model.snowplow.model_1',
+            fqn=['snowplow', 'nested', 'model_1'],
+            package_name='snowplow',
+            original_file_path=normalize('models/nested/model_1.sql'),
+            root_path=get_abs_os_path('./dbt_packages/snowplow'),
+            config=NodeConfig(materialized='table'),
+            path=normalize('nested/model_1.sql'),
+            raw_sql='{{ config(materialized="table") }}select 1 as id',
+            checksum=None,
+            unrendered_config={'materialized': 'table'},
+        )
+        self.example_config = ContextConfig(
+            self.root_project_config,
+            self.example_node.fqn,
+            self.example_node.resource_type,
+            self.snowplow_project_config,
+        )
+
+    def file_block_for(self, data, filename):
+        return super().file_block_for(data, filename, 'models')
+
+    # tests that configs get extracted properly. the function should respect merge behavior,
+    # but becuase it's only reading from one dictionary it won't matter except in edge cases
+    # like this example with tags changing type to a list.
+    def test_config_shifting(self):
+        static_parser_result = {
+            'configs': [
+                ('hello', 'world'),
+                ('flag', True),
+                ('tags', 'tag1'),
+                ('tags', 'tag2')
+            ]
+        }
+        expected = {
+            'hello': 'world',
+            'flag': True,
+            'tags': ['tag1', 'tag2']
+        }
+        got = _get_config_call_dict(static_parser_result)
+        self.assertEqual(expected, got)
+
+    def test_sample_results(self):
+        # --- missed ref --- #
+        node = deepcopy(self.example_node)
+        config = deepcopy(self.example_config)
+        sample_node = deepcopy(self.example_node)
+        sample_config = deepcopy(self.example_config)
+
+        sample_node.refs = []
+        node.refs = ['myref']
+
+        result = _get_sample_result(sample_node, sample_config, node, config)
+        self.assertEqual([(7, "missed_ref_value")], result)
+
+        # --- false positive ref --- #
+        node = deepcopy(self.example_node)
+        config = deepcopy(self.example_config)
+        sample_node = deepcopy(self.example_node)
+        sample_config = deepcopy(self.example_config)
+
+        sample_node.refs = ['myref']
+        node.refs = []
+
+        result = _get_sample_result(sample_node, sample_config, node, config)
+        self.assertEqual([(6, "false_positive_ref_value")], result)
+
+        # --- missed source --- #
+        node = deepcopy(self.example_node)
+        config = deepcopy(self.example_config)
+        sample_node = deepcopy(self.example_node)
+        sample_config = deepcopy(self.example_config)
+
+        sample_node.sources = []
+        node.sources = [['abc', 'def']]
+
+        result = _get_sample_result(sample_node, sample_config, node, config)
+        self.assertEqual([(5, 'missed_source_value')], result)
+
+        # --- false positive source --- #
+        node = deepcopy(self.example_node)
+        config = deepcopy(self.example_config)
+        sample_node = deepcopy(self.example_node)
+        sample_config = deepcopy(self.example_config)
+
+        sample_node.sources = [['abc', 'def']]
+        node.sources = []
+
+        result = _get_sample_result(sample_node, sample_config, node, config)
+        self.assertEqual([(4, 'false_positive_source_value')], result)
+
+        # --- missed config --- #
+        node = deepcopy(self.example_node)
+        config = deepcopy(self.example_config)
+        sample_node = deepcopy(self.example_node)
+        sample_config = deepcopy(self.example_config)
+
+        sample_config._config_call_dict = {}
+        config._config_call_dict = {'key': 'value'}
+
+        result = _get_sample_result(sample_node, sample_config, node, config)
+        self.assertEqual([(3, 'missed_config_value')], result)
+
+        # --- false positive config --- #
+        node = deepcopy(self.example_node)
+        config = deepcopy(self.example_config)
+        sample_node = deepcopy(self.example_node)
+        sample_config = deepcopy(self.example_config)
+
+        sample_config._config_call_dict = {'key': 'value'}
+        config._config_call_dict = {}
+
+        result = _get_sample_result(sample_node, sample_config, node, config)
+        self.assertEqual([(2, "false_positive_config_value")], result)
+
+    def test_exp_sample_results(self):
+        node = deepcopy(self.example_node)
+        config = deepcopy(self.example_config)
+        sample_node = deepcopy(self.example_node)
+        sample_config = deepcopy(self.example_config)
+        result = _get_exp_sample_result(sample_node, sample_config, node, config)
+        self.assertEqual(["00_experimental_exact_match"], result)
+
+
 class SnapshotParserTest(BaseParserTest):
     def setUp(self):
         super().setUp()