fmt

remove comment nonsense
make scale more useful
2025-12-19 02:51:27 +00:00 · 2021-09-22 10:33:46 -04:00 · 2021-09-22 10:30:29 -04:00 · 2021-09-22 10:30:29 -04:00 · 2021-09-22 10:30:29 -04:00 · 2021-09-22 10:30:29 -04:00
17 changed files with 1108 additions and 34 deletions
--- a/.github/workflows/performance.yml
+++ b/.github/workflows/performance.yml
@@ -164,11 +164,13 @@ jobs:
          name: runner
      - name: change permissions
        run: chmod +x ./runner
+      - name: make results directory
+        run: mkdir ./final-output/
      - name: run calculation
-        run: ./runner calculate -r ./
+        run: ./runner calculate -r ./ -o ./final-output/
        # always attempt to upload the results even if there were regressions found
      - uses: actions/upload-artifact@v2
        if: ${{ always() }}
        with:
          name: final-calculations
-          path: ./final_calculations.json
+          path: ./final-output/*
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,6 +16,7 @@
 - Added default field in the `selectors.yml` to allow user to define default selector ([#3448](https://github.com/dbt-labs/dbt/issues/3448), [#3875](https://github.com/dbt-labs/dbt/issues/3875), [#3892](https://github.com/dbt-labs/dbt/issues/3892))
 - Added timing and thread information to sources.json artifact ([#3804](https://github.com/dbt-labs/dbt/issues/3804), [#3894](https://github.com/dbt-labs/dbt/pull/3894))
 - Update cli and rpc flags for the `build` task to align with other commands (`--resource-type`, `--store-failures`) ([#3596](https://github.com/dbt-labs/dbt/issues/3596), [#3884](https://github.com/dbt-labs/dbt/pull/3884))
+- Log tests that are not indirectly selected. Add `--greedy` flag to `test`, `list`, `build` and `greedy` property in yaml selectors ([#3723](https://github.com/dbt-labs/dbt/pull/3723), [#3833](https://github.com/dbt-labs/dbt/pull/3833))

 ### Fixes

@@ -44,7 +45,7 @@ Contributors:
 - [@dbrtly](https://github.com/dbrtly) ([#3834](https://github.com/dbt-labs/dbt/pull/3834))
 - [@swanderz](https://github.com/swanderz) [#3623](https://github.com/dbt-labs/dbt/pull/3623)
 - [@JasonGluck](https://github.com/JasonGluck) ([#3582](https://github.com/dbt-labs/dbt/pull/3582))
- [@joellabes](https://github.com/joellabes) ([#3669](https://github.com/dbt-labs/dbt/pull/3669))
+- [@joellabes](https://github.com/joellabes) ([#3669](https://github.com/dbt-labs/dbt/pull/3669), [#3833](https://github.com/dbt-labs/dbt/pull/3833))
 - [@juma-adoreme](https://github.com/juma-adoreme) ([#3838](https://github.com/dbt-labs/dbt/pull/3838))
 - [@annafil](https://github.com/annafil) ([#3825](https://github.com/dbt-labs/dbt/pull/3825))
 - [@AndreasTA-AW](https://github.com/AndreasTA-AW) ([#3691](https://github.com/dbt-labs/dbt/pull/3691))
--- a/core/dbt/flags.py
+++ b/core/dbt/flags.py
@@ -18,6 +18,7 @@ WRITE_JSON = None
 PARTIAL_PARSE = None
 USE_COLORS = None
 STORE_FAILURES = None
+GREEDY = None


 def env_set_truthy(key: str) -> Optional[str]:
@@ -56,7 +57,7 @@ MP_CONTEXT = _get_context()
 def reset():
    global STRICT_MODE, FULL_REFRESH, USE_CACHE, WARN_ERROR, TEST_NEW_PARSER, \
        USE_EXPERIMENTAL_PARSER, WRITE_JSON, PARTIAL_PARSE, MP_CONTEXT, USE_COLORS, \
-        STORE_FAILURES
+        STORE_FAILURES, GREEDY

    STRICT_MODE = False
    FULL_REFRESH = False
@@ -69,12 +70,13 @@ def reset():
    MP_CONTEXT = _get_context()
    USE_COLORS = True
    STORE_FAILURES = False
+    GREEDY = False


 def set_from_args(args):
    global STRICT_MODE, FULL_REFRESH, USE_CACHE, WARN_ERROR, TEST_NEW_PARSER, \
        USE_EXPERIMENTAL_PARSER, WRITE_JSON, PARTIAL_PARSE, MP_CONTEXT, USE_COLORS, \
-        STORE_FAILURES
+        STORE_FAILURES, GREEDY

    USE_CACHE = getattr(args, 'use_cache', USE_CACHE)

@@ -99,6 +101,7 @@ def set_from_args(args):
        USE_COLORS = use_colors_override

    STORE_FAILURES = getattr(args, 'store_failures', STORE_FAILURES)
+    GREEDY = getattr(args, 'greedy', GREEDY)


 # initialize everything to the defaults on module load
--- a/core/dbt/graph/cli.py
+++ b/core/dbt/graph/cli.py
@@ -1,4 +1,5 @@
 # special support for CLI argument parsing.
+from dbt import flags
 import itertools
 from dbt.clients.yaml_helper import yaml, Loader, Dumper  # noqa: F401

@@ -66,7 +67,7 @@ def parse_union_from_default(
 def parse_difference(
    include: Optional[List[str]], exclude: Optional[List[str]]
 ) -> SelectionDifference:
-    included = parse_union_from_default(include, DEFAULT_INCLUDES)
+    included = parse_union_from_default(include, DEFAULT_INCLUDES, greedy=bool(flags.GREEDY))
    excluded = parse_union_from_default(exclude, DEFAULT_EXCLUDES, greedy=True)
    return SelectionDifference(components=[included, excluded])

@@ -180,7 +181,7 @@ def parse_union_definition(definition: Dict[str, Any]) -> SelectionSpec:
    union_def_parts = _get_list_dicts(definition, 'union')
    include, exclude = _parse_include_exclude_subdefs(union_def_parts)

-    union = SelectionUnion(components=include)
+    union = SelectionUnion(components=include, greedy_warning=False)

    if exclude is None:
        union.raw = definition
@@ -188,7 +189,8 @@ def parse_union_definition(definition: Dict[str, Any]) -> SelectionSpec:
    else:
        return SelectionDifference(
            components=[union, exclude],
-            raw=definition
+            raw=definition,
+            greedy_warning=False
        )


@@ -197,7 +199,7 @@ def parse_intersection_definition(
 ) -> SelectionSpec:
    intersection_def_parts = _get_list_dicts(definition, 'intersection')
    include, exclude = _parse_include_exclude_subdefs(intersection_def_parts)
-    intersection = SelectionIntersection(components=include)
+    intersection = SelectionIntersection(components=include, greedy_warning=False)

    if exclude is None:
        intersection.raw = definition
@@ -205,7 +207,8 @@ def parse_intersection_definition(
    else:
        return SelectionDifference(
            components=[intersection, exclude],
-            raw=definition
+            raw=definition,
+            greedy_warning=False
        )


@@ -239,7 +242,7 @@ def parse_dict_definition(definition: Dict[str, Any]) -> SelectionSpec:
    if diff_arg is None:
        return base
    else:
-        return SelectionDifference(components=[base, diff_arg])
+        return SelectionDifference(components=[base, diff_arg], greedy_warning=False)


 def parse_from_definition(
--- a/core/dbt/graph/selector.py
+++ b/core/dbt/graph/selector.py
@@ -1,4 +1,3 @@
-
 from typing import Set, List, Optional, Tuple

 from .graph import Graph, UniqueId
@@ -30,6 +29,24 @@ def alert_non_existence(raw_spec, nodes):
        )


+def alert_unused_nodes(raw_spec, node_names):
+    summary_nodes_str = ("\n  - ").join(node_names[:3])
+    debug_nodes_str = ("\n  - ").join(node_names)
+    and_more_str = f"\n  - and {len(node_names) - 3} more" if len(node_names) > 4 else ""
+    summary_msg = (
+        f"\nSome tests were excluded because at least one parent is not selected. "
+        f"Use the --greedy flag to include them."
+        f"\n  - {summary_nodes_str}{and_more_str}"
+    )
+    logger.info(summary_msg)
+    if len(node_names) > 4:
+        debug_msg = (
+            f"Full list of tests that were excluded:"
+            f"\n  - {debug_nodes_str}"
+        )
+        logger.debug(debug_msg)
+
+
 def can_select_indirectly(node):
    """If a node is not selected itself, but its parent(s) are, it may qualify
    for indirect selection.
@@ -151,16 +168,16 @@ class NodeSelector(MethodManager):

        return direct_nodes, indirect_nodes

-    def select_nodes(self, spec: SelectionSpec) -> Set[UniqueId]:
+    def select_nodes(self, spec: SelectionSpec) -> Tuple[Set[UniqueId], Set[UniqueId]]:
        """Select the nodes in the graph according to the spec.

        This is the main point of entry for turning a spec into a set of nodes:
        - Recurse through spec, select by criteria, combine by set operation
        - Return final (unfiltered) selection set
        """
-
        direct_nodes, indirect_nodes = self.select_nodes_recursively(spec)
-        return direct_nodes
+        indirect_only = indirect_nodes.difference(direct_nodes)
+        return direct_nodes, indirect_only

    def _is_graph_member(self, unique_id: UniqueId) -> bool:
        if unique_id in self.manifest.sources:
@@ -213,6 +230,8 @@ class NodeSelector(MethodManager):
        #  - If ANY parent is missing, return it separately. We'll keep it around
        #    for later and see if its other parents show up.
        # We use this for INCLUSION.
+        # Users can also opt in to inclusive GREEDY mode by passing --greedy flag,
+        # or by specifying `greedy: true` in a yaml selector

        direct_nodes = set(selected)
        indirect_nodes = set()
@@ -251,15 +270,24 @@ class NodeSelector(MethodManager):

            - node selection. Based on the include/exclude sets, the set
                of matched unique IDs is returned
-                - expand the graph at each leaf node, before combination
-                    - selectors might override this. for example, this is where
-                        tests are added
+                - includes direct + indirect selection (for tests)
            - filtering:
                - selectors can filter the nodes after all of them have been
                  selected
        """
-        selected_nodes = self.select_nodes(spec)
+        selected_nodes, indirect_only = self.select_nodes(spec)
        filtered_nodes = self.filter_selection(selected_nodes)
+
+        if indirect_only:
+            filtered_unused_nodes = self.filter_selection(indirect_only)
+            if filtered_unused_nodes and spec.greedy_warning:
+                # log anything that didn't make the cut
+                unused_node_names = []
+                for unique_id in filtered_unused_nodes:
+                    name = self.manifest.nodes[unique_id].name
+                    unused_node_names.append(name)
+                alert_unused_nodes(spec, unused_node_names)
+
        return filtered_nodes

    def get_graph_queue(self, spec: SelectionSpec) -> GraphQueue:
--- a/core/dbt/graph/selector_spec.py
+++ b/core/dbt/graph/selector_spec.py
@@ -67,6 +67,7 @@ class SelectionCriteria:
    children: bool
    children_depth: Optional[int]
    greedy: bool = False
+    greedy_warning: bool = False  # do not raise warning for yaml selectors

    def __post_init__(self):
        if self.children and self.childrens_parents:
@@ -124,11 +125,11 @@ class SelectionCriteria:
            parents_depth=parents_depth,
            children=bool(dct.get('children')),
            children_depth=children_depth,
-            greedy=greedy
+            greedy=(greedy or bool(dct.get('greedy'))),
        )

    @classmethod
-    def dict_from_single_spec(cls, raw: str, greedy: bool = False):
+    def dict_from_single_spec(cls, raw: str):
        result = RAW_SELECTOR_PATTERN.match(raw)
        if result is None:
            return {'error': 'Invalid selector spec'}
@@ -145,6 +146,8 @@ class SelectionCriteria:
            dct['parents'] = bool(dct.get('parents'))
        if 'children' in dct:
            dct['children'] = bool(dct.get('children'))
+        if 'greedy' in dct:
+            dct['greedy'] = bool(dct.get('greedy'))
        return dct

    @classmethod
@@ -162,10 +165,12 @@ class BaseSelectionGroup(Iterable[SelectionSpec], metaclass=ABCMeta):
        self,
        components: Iterable[SelectionSpec],
        expect_exists: bool = False,
+        greedy_warning: bool = True,
        raw: Any = None,
    ):
        self.components: List[SelectionSpec] = list(components)
        self.expect_exists = expect_exists
+        self.greedy_warning = greedy_warning
        self.raw = raw

    def __iter__(self) -> Iterator[SelectionSpec]:
--- a/core/dbt/main.py
+++ b/core/dbt/main.py
@@ -406,6 +406,14 @@ def _build_build_subparser(subparsers, base_subparser):
        Store test results (failing rows) in the database
        '''
    )
+    sub.add_argument(
+        '--greedy',
+        action='store_true',
+        help='''
+        Select all tests that touch the selected resources,
+        even if they also depend on unselected resources
+        '''
+    )
    resource_values: List[str] = [
        str(s) for s in build_task.BuildTask.ALL_RESOURCE_VALUES
    ] + ['all']
@@ -637,7 +645,7 @@ def _add_table_mutability_arguments(*subparsers):
            '--full-refresh',
            action='store_true',
            help='''
-            If specified, DBT will drop incremental models and
+            If specified, dbt will drop incremental models and
            fully-recalculate the incremental table from the model definition.
            '''
        )
@@ -753,6 +761,14 @@ def _build_test_subparser(subparsers, base_subparser):
        Store test results (failing rows) in the database
        '''
    )
+    sub.add_argument(
+        '--greedy',
+        action='store_true',
+        help='''
+        Select all tests that touch the selected resources,
+        even if they also depend on unselected resources
+        '''
+    )

    sub.set_defaults(cls=test_task.TestTask, which='test', rpc_method='test')
    return sub
@@ -878,6 +894,14 @@ def _build_list_subparser(subparsers, base_subparser):
        metavar='SELECTOR',
        required=False,
    )
+    sub.add_argument(
+        '--greedy',
+        action='store_true',
+        help='''
+        Select all tests that touch the selected resources,
+        even if they also depend on unselected resources
+        '''
+    )
    _add_common_selector_arguments(sub)

    return sub
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -438,7 +438,7 @@ class GraphRunnableTask(ManifestTask):
            )

        if len(self._flattened_nodes) == 0:
-            logger.warning("WARNING: Nothing to do. Try checking your model "
+            logger.warning("\nWARNING: Nothing to do. Try checking your model "
                           "configs and model specification args")
            result = self.get_result(
                results=[],
--- a/performance/runner/.gitignore
+++ b/performance/runner/.gitignore
@@ -1,2 +1,3 @@
 target/
 projects/*/logs
+plots/
--- a/performance/runner/Cargo.lock
+++ b/performance/runner/Cargo.lock
@@ -2,6 +2,12 @@
 # It is not intended for manual editing.
 version = 3

+[[package]]
+name = "adler32"
+version = "1.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "aae1277d39aeec15cb388266ecc24b11c80469deae6067e17a1a7aa9e5c1f234"
+
 [[package]]
 name = "ansi_term"
 version = "0.11.0"
@@ -22,12 +28,62 @@ dependencies = [
 "winapi",
 ]

+[[package]]
+name = "autocfg"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a"
+
 [[package]]
 name = "bitflags"
 version = "1.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"

+[[package]]
+name = "bumpalo"
+version = "3.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9c59e7af012c713f529e7a3ee57ce9b31ddd858d4b512923602f74608b009631"
+
+[[package]]
+name = "bytemuck"
+version = "1.7.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "72957246c41db82b8ef88a5486143830adeb8227ef9837740bdec67724cf2c5b"
+
+[[package]]
+name = "byteorder"
+version = "1.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
+
+[[package]]
+name = "cc"
+version = "1.0.70"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d26a6ce4b6a484fa3edb70f7efa6fc430fd2b87285fe8b84304fd0936faa0dc0"
+
+[[package]]
+name = "cfg-if"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+
+[[package]]
+name = "chrono"
+version = "0.4.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "670ad68c9088c2a963aaa298cb369688cf3f9465ce5e2d4ca10e6e0098a1ce73"
+dependencies = [
+ "libc",
+ "num-integer",
+ "num-traits",
+ "serde",
+ "time",
+ "winapi",
+]
+
 [[package]]
 name = "clap"
 version = "2.33.3"
@@ -43,12 +99,230 @@ dependencies = [
 "vec_map",
 ]

+[[package]]
+name = "cmake"
+version = "0.1.45"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "eb6210b637171dfba4cda12e579ac6dc73f5165ad56133e5d72ef3131f320855"
+dependencies = [
+ "cc",
+]
+
+[[package]]
+name = "color_quant"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3d7b894f5411737b7867f4827955924d7c254fc9f4d91a6aad6b097804b1018b"
+
+[[package]]
+name = "core-foundation"
+version = "0.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0a89e2ae426ea83155dccf10c0fa6b1463ef6d5fcb44cee0b224a408fa640a62"
+dependencies = [
+ "core-foundation-sys",
+ "libc",
+]
+
+[[package]]
+name = "core-foundation-sys"
+version = "0.8.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ea221b5284a47e40033bf9b66f35f984ec0ea2931eb03505246cd27a963f981b"
+
+[[package]]
+name = "core-graphics"
+version = "0.22.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "269f35f69b542b80e736a20a89a05215c0ce80c2c03c514abb2e318b78379d86"
+dependencies = [
+ "bitflags",
+ "core-foundation",
+ "core-graphics-types",
+ "foreign-types",
+ "libc",
+]
+
+[[package]]
+name = "core-graphics-types"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3a68b68b3446082644c91ac778bf50cd4104bfb002b5a6a7c44cca5a2c70788b"
+dependencies = [
+ "bitflags",
+ "core-foundation",
+ "foreign-types",
+ "libc",
+]
+
+[[package]]
+name = "core-text"
+version = "19.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "99d74ada66e07c1cefa18f8abfba765b486f250de2e4a999e5727fc0dd4b4a25"
+dependencies = [
+ "core-foundation",
+ "core-graphics",
+ "foreign-types",
+ "libc",
+]
+
+[[package]]
+name = "crc32fast"
+version = "1.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "81156fece84ab6a9f2afdb109ce3ae577e42b1228441eded99bd77f627953b1a"
+dependencies = [
+ "cfg-if",
+]
+
+[[package]]
+name = "deflate"
+version = "0.8.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "73770f8e1fe7d64df17ca66ad28994a0a623ea497fa69486e14984e715c5d174"
+dependencies = [
+ "adler32",
+ "byteorder",
+]
+
+[[package]]
+name = "dirs-next"
+version = "2.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b98cf8ebf19c3d1b223e151f99a4f9f0690dca41414773390fc824184ac833e1"
+dependencies = [
+ "cfg-if",
+ "dirs-sys-next",
+]
+
+[[package]]
+name = "dirs-sys-next"
+version = "0.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4ebda144c4fe02d1f7ea1a7d9641b6fc6b580adcfa024ae48797ecdeb6825b4d"
+dependencies = [
+ "libc",
+ "redox_users",
+ "winapi",
+]
+
+[[package]]
+name = "dwrote"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "439a1c2ba5611ad3ed731280541d36d2e9c4ac5e7fb818a27b604bdc5a6aa65b"
+dependencies = [
+ "lazy_static",
+ "libc",
+ "winapi",
+ "wio",
+]
+
 [[package]]
 name = "either"
 version = "1.6.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e78d4f1cc4ae33bbfc157ed5d5a5ef3bc29227303d595861deb238fcec4e9457"

+[[package]]
+name = "expat-sys"
+version = "2.1.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "658f19728920138342f68408b7cf7644d90d4784353d8ebc32e7e8663dbe45fa"
+dependencies = [
+ "cmake",
+ "pkg-config",
+]
+
+[[package]]
+name = "float-ord"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7bad48618fdb549078c333a7a8528acb57af271d0433bdecd523eb620628364e"
+
+[[package]]
+name = "font-kit"
+version = "0.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "46c9a156ec38864999bc9c4156e5f3b50224d4a5578028a64e5a3875caa9ee28"
+dependencies = [
+ "bitflags",
+ "byteorder",
+ "core-foundation",
+ "core-graphics",
+ "core-text",
+ "dirs-next",
+ "dwrote",
+ "float-ord",
+ "freetype",
+ "lazy_static",
+ "libc",
+ "log",
+ "pathfinder_geometry",
+ "pathfinder_simd",
+ "servo-fontconfig",
+ "walkdir",
+ "winapi",
+]
+
+[[package]]
+name = "foreign-types"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1"
+dependencies = [
+ "foreign-types-shared",
+]
+
+[[package]]
+name = "foreign-types-shared"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b"
+
+[[package]]
+name = "freetype"
+version = "0.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bee38378a9e3db1cc693b4f88d166ae375338a0ff75cb8263e1c601d51f35dc6"
+dependencies = [
+ "freetype-sys",
+ "libc",
+]
+
+[[package]]
+name = "freetype-sys"
+version = "0.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a37d4011c0cc628dfa766fcc195454f4b068d7afdc2adfd28861191d866e731a"
+dependencies = [
+ "cmake",
+ "libc",
+ "pkg-config",
+]
+
+[[package]]
+name = "getrandom"
+version = "0.2.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7fcd999463524c52659517fe2cea98493cfe485d10565e7b0fb07dbba7ad2753"
+dependencies = [
+ "cfg-if",
+ "libc",
+ "wasi",
+]
+
+[[package]]
+name = "gif"
+version = "0.11.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5a668f699973d0f573d15749b7002a9ac9e1f9c6b220e7b165601334c173d8de"
+dependencies = [
+ "color_quant",
+ "weezl",
+]
+
 [[package]]
 name = "heck"
 version = "0.3.3"
@@ -67,6 +341,22 @@ dependencies = [
 "libc",
 ]

+[[package]]
+name = "image"
+version = "0.23.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "24ffcb7e7244a9bf19d35bf2883b9c080c4ced3c07a9895572178cdb8f13f6a1"
+dependencies = [
+ "bytemuck",
+ "byteorder",
+ "color_quant",
+ "jpeg-decoder",
+ "num-iter",
+ "num-rational",
+ "num-traits",
+ "png",
+]
+
 [[package]]
 name = "itertools"
 version = "0.10.1"
@@ -82,6 +372,21 @@ version = "0.4.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dd25036021b0de88a0aff6b850051563c6516d0bf53f8638938edbb9de732736"

+[[package]]
+name = "jpeg-decoder"
+version = "0.1.22"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "229d53d58899083193af11e15917b5640cd40b29ff475a1fe4ef725deb02d0f2"
+
+[[package]]
+name = "js-sys"
+version = "0.3.54"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1866b355d9c878e5e607473cbe3f63282c0b7aad2db1dbebf55076c686918254"
+dependencies = [
+ "wasm-bindgen",
+]
+
 [[package]]
 name = "lazy_static"
 version = "1.4.0"
@@ -94,6 +399,157 @@ version = "0.2.98"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "320cfe77175da3a483efed4bc0adc1968ca050b098ce4f2f1c13a56626128790"

+[[package]]
+name = "log"
+version = "0.4.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "51b9bbe6c47d51fc3e1a9b945965946b4c44142ab8792c50835a980d362c2710"
+dependencies = [
+ "cfg-if",
+]
+
+[[package]]
+name = "miniz_oxide"
+version = "0.3.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "791daaae1ed6889560f8c4359194f56648355540573244a5448a83ba1ecc7435"
+dependencies = [
+ "adler32",
+]
+
+[[package]]
+name = "num-integer"
+version = "0.1.44"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d2cc698a63b549a70bc047073d2949cce27cd1c7b0a4a862d08a8031bc2801db"
+dependencies = [
+ "autocfg",
+ "num-traits",
+]
+
+[[package]]
+name = "num-iter"
+version = "0.1.42"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b2021c8337a54d21aca0d59a92577a029af9431cb59b909b03252b9c164fad59"
+dependencies = [
+ "autocfg",
+ "num-integer",
+ "num-traits",
+]
+
+[[package]]
+name = "num-rational"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "12ac428b1cb17fce6f731001d307d351ec70a6d202fc2e60f7d4c5e42d8f4f07"
+dependencies = [
+ "autocfg",
+ "num-integer",
+ "num-traits",
+]
+
+[[package]]
+name = "num-traits"
+version = "0.2.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9a64b1ec5cda2586e284722486d802acf1f7dbdc623e2bfc57e65ca1cd099290"
+dependencies = [
+ "autocfg",
+]
+
+[[package]]
+name = "pathfinder_geometry"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b7b7e7b4ea703700ce73ebf128e1450eb69c3a8329199ffbfb9b2a0418e5ad3"
+dependencies = [
+ "log",
+ "pathfinder_simd",
+]
+
+[[package]]
+name = "pathfinder_simd"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "39fe46acc5503595e5949c17b818714d26fdf9b4920eacf3b2947f0199f4a6ff"
+dependencies = [
+ "rustc_version",
+]
+
+[[package]]
+name = "pest"
+version = "2.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "10f4872ae94d7b90ae48754df22fd42ad52ce740b8f370b03da4835417403e53"
+dependencies = [
+ "ucd-trie",
+]
+
+[[package]]
+name = "pkg-config"
+version = "0.3.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3831453b3449ceb48b6d9c7ad7c96d5ea673e9b470a1dc578c2ce6521230884c"
+
+[[package]]
+name = "plotters"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32a3fd9ec30b9749ce28cd91f255d569591cdf937fe280c312143e3c4bad6f2a"
+dependencies = [
+ "chrono",
+ "font-kit",
+ "image",
+ "lazy_static",
+ "num-traits",
+ "pathfinder_geometry",
+ "plotters-backend",
+ "plotters-bitmap",
+ "plotters-svg",
+ "ttf-parser",
+ "wasm-bindgen",
+ "web-sys",
+]
+
+[[package]]
+name = "plotters-backend"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d88417318da0eaf0fdcdb51a0ee6c3bed624333bff8f946733049380be67ac1c"
+
+[[package]]
+name = "plotters-bitmap"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "21362fa905695e5618aefd169358f52e0e8bc4a8e05333cf780fda8cddc00b54"
+dependencies = [
+ "gif",
+ "image",
+ "plotters-backend",
+]
+
+[[package]]
+name = "plotters-svg"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "521fa9638fa597e1dc53e9412a4f9cefb01187ee1f7413076f9e6749e2885ba9"
+dependencies = [
+ "plotters-backend",
+]
+
+[[package]]
+name = "png"
+version = "0.16.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3c3287920cb847dee3de33d301c463fba14dda99db24214ddf93f83d3021f4c6"
+dependencies = [
+ "bitflags",
+ "crc32fast",
+ "deflate",
+ "miniz_oxide",
+]
+
 [[package]]
 name = "proc-macro-error"
 version = "1.0.4"
@@ -136,23 +592,80 @@ dependencies = [
 "proc-macro2",
 ]

+[[package]]
+name = "redox_syscall"
+version = "0.2.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8383f39639269cde97d255a32bdb68c047337295414940c68bdd30c2e13203ff"
+dependencies = [
+ "bitflags",
+]
+
+[[package]]
+name = "redox_users"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "528532f3d801c87aec9def2add9ca802fe569e44a544afe633765267840abe64"
+dependencies = [
+ "getrandom",
+ "redox_syscall",
+]
+
 [[package]]
 name = "runner"
 version = "0.1.0"
 dependencies = [
+ "chrono",
 "itertools",
+ "plotters",
 "serde",
 "serde_json",
 "structopt",
 "thiserror",
 ]

+[[package]]
+name = "rustc_version"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f0dfe2087c51c460008730de8b57e6a320782fbfb312e1f4d520e6c6fae155ee"
+dependencies = [
+ "semver",
+]
+
 [[package]]
 name = "ryu"
 version = "1.0.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "71d301d4193d031abdd79ff7e3dd721168a9572ef3fe51a1517aba235bd8f86e"

+[[package]]
+name = "same-file"
+version = "1.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
+dependencies = [
+ "winapi-util",
+]
+
+[[package]]
+name = "semver"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f301af10236f6df4160f7c3f04eec6dbc70ace82d23326abad5edee88801c6b6"
+dependencies = [
+ "semver-parser",
+]
+
+[[package]]
+name = "semver-parser"
+version = "0.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "00b0bef5b7f9e0df16536d3961cfb6e84331c065b4066afb39768d0e319411f7"
+dependencies = [
+ "pest",
+]
+
 [[package]]
 name = "serde"
 version = "1.0.127"
@@ -184,6 +697,27 @@ dependencies = [
 "serde",
 ]

+[[package]]
+name = "servo-fontconfig"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c7e3e22fe5fd73d04ebf0daa049d3efe3eae55369ce38ab16d07ddd9ac5c217c"
+dependencies = [
+ "libc",
+ "servo-fontconfig-sys",
+]
+
+[[package]]
+name = "servo-fontconfig-sys"
+version = "5.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e36b879db9892dfa40f95da1c38a835d41634b825fbd8c4c418093d53c24b388"
+dependencies = [
+ "expat-sys",
+ "freetype-sys",
+ "pkg-config",
+]
+
 [[package]]
 name = "strsim"
 version = "0.8.0"
@@ -254,6 +788,29 @@ dependencies = [
 "syn",
 ]

+[[package]]
+name = "time"
+version = "0.1.44"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6db9e6914ab8b1ae1c260a4ae7a49b6c5611b40328a735b21862567685e73255"
+dependencies = [
+ "libc",
+ "wasi",
+ "winapi",
+]
+
+[[package]]
+name = "ttf-parser"
+version = "0.12.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7ae2f58a822f08abdaf668897e96a5656fe72f5a9ce66422423e8849384872e6"
+
+[[package]]
+name = "ucd-trie"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "56dee185309b50d1f11bfedef0fe6d036842e3fb77413abef29f8f8d1c5d4c1c"
+
 [[package]]
 name = "unicode-segmentation"
 version = "1.8.0"
@@ -284,6 +841,93 @@ version = "0.9.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "5fecdca9a5291cc2b8dcf7dc02453fee791a280f3743cb0905f8822ae463b3fe"

+[[package]]
+name = "walkdir"
+version = "2.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "808cf2735cd4b6866113f648b791c6adc5714537bc222d9347bb203386ffda56"
+dependencies = [
+ "same-file",
+ "winapi",
+ "winapi-util",
+]
+
+[[package]]
+name = "wasi"
+version = "0.10.0+wasi-snapshot-preview1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1a143597ca7c7793eff794def352d41792a93c481eb1042423ff7ff72ba2c31f"
+
+[[package]]
+name = "wasm-bindgen"
+version = "0.2.77"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5e68338db6becec24d3c7977b5bf8a48be992c934b5d07177e3931f5dc9b076c"
+dependencies = [
+ "cfg-if",
+ "wasm-bindgen-macro",
+]
+
+[[package]]
+name = "wasm-bindgen-backend"
+version = "0.2.77"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f34c405b4f0658583dba0c1c7c9b694f3cac32655db463b56c254a1c75269523"
+dependencies = [
+ "bumpalo",
+ "lazy_static",
+ "log",
+ "proc-macro2",
+ "quote",
+ "syn",
+ "wasm-bindgen-shared",
+]
+
+[[package]]
+name = "wasm-bindgen-macro"
+version = "0.2.77"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9d5a6580be83b19dc570a8f9c324251687ab2184e57086f71625feb57ec77c8"
+dependencies = [
+ "quote",
+ "wasm-bindgen-macro-support",
+]
+
+[[package]]
+name = "wasm-bindgen-macro-support"
+version = "0.2.77"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e3775a030dc6f5a0afd8a84981a21cc92a781eb429acef9ecce476d0c9113e92"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+ "wasm-bindgen-backend",
+ "wasm-bindgen-shared",
+]
+
+[[package]]
+name = "wasm-bindgen-shared"
+version = "0.2.77"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c279e376c7a8e8752a8f1eaa35b7b0bee6bb9fb0cdacfa97cc3f1f289c87e2b4"
+
+[[package]]
+name = "web-sys"
+version = "0.3.54"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0a84d70d1ec7d2da2d26a5bd78f4bca1b8c3254805363ce743b7a05bc30d195a"
+dependencies = [
+ "js-sys",
+ "wasm-bindgen",
+]
+
+[[package]]
+name = "weezl"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d8b77fdfd5a253be4ab714e4ffa3c49caf146b4de743e97510c0656cf90f1e8e"
+
 [[package]]
 name = "winapi"
 version = "0.3.9"
@@ -300,8 +944,26 @@ version = "0.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"

+[[package]]
+name = "winapi-util"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "70ec6ce85bb158151cae5e5c87f95a8e97d2c0c4b001223f33a334e3ce5de178"
+dependencies = [
+ "winapi",
+]
+
 [[package]]
 name = "winapi-x86_64-pc-windows-gnu"
 version = "0.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
+
+[[package]]
+name = "wio"
+version = "0.2.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5d129932f4644ac2396cb456385cbf9e63b5b30c6e8dc4820bdca4eb082037a5"
+dependencies = [
+ "winapi",
+]
--- a/performance/runner/Cargo.toml
+++ b/performance/runner/Cargo.toml
@@ -4,7 +4,9 @@ version = "0.1.0"
 edition = "2018"

 [dependencies]
+chrono = { version = "0.4.19", features = ["serde"] }
 itertools = "0.10.1"
+plotters = "^0.3.1"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 structopt = "0.3"
--- a/performance/runner/src/calculate.rs
+++ b/performance/runner/src/calculate.rs
@@ -1,4 +1,5 @@
 use crate::exceptions::{CalculateError, IOError};
+use chrono::prelude::*;
 use itertools::Itertools;
 use serde::{Deserialize, Serialize};
 use std::fs;
@@ -45,6 +46,7 @@ pub struct Data {
 pub struct Calculation {
    pub metric: String,
    pub regression: bool,
+    pub ts: DateTime<Utc>,
    pub data: Data,
 }

@@ -60,6 +62,11 @@ pub struct MeasurementGroup {
 // Given two measurements, return all the calculations. Calculations are
 // flagged as regressions or not regressions.
 fn calculate(metric: &str, dev: &Measurement, baseline: &Measurement) -> Vec<Calculation> {
+    // choosing the current timestamp for all calculations to be the same.
+    // this timestamp is not from the time of measurement becuase hyperfine
+    // controls that. Since calculation is run directly after, this is fine.
+    let ts = Utc::now();
+
    let median_threshold = 1.05; // 5% regression threshold
    let median_difference = dev.median / baseline.median;

@@ -70,6 +77,7 @@ fn calculate(metric: &str, dev: &Measurement, baseline: &Measurement) -> Vec<Cal
        Calculation {
            metric: ["median", metric].join("_"),
            regression: median_difference > median_threshold,
+            ts: ts,
            data: Data {
                threshold: median_threshold,
                difference: median_difference,
@@ -80,6 +88,7 @@ fn calculate(metric: &str, dev: &Measurement, baseline: &Measurement) -> Vec<Cal
        Calculation {
            metric: ["stddev", metric].join("_"),
            regression: stddev_difference > stddev_threshold,
+            ts: ts,
            data: Data {
                threshold: stddev_threshold,
                difference: stddev_difference,
--- a/performance/runner/src/exceptions.rs
+++ b/performance/runner/src/exceptions.rs
@@ -42,6 +42,28 @@ pub enum CalculateError {
    BadBranchNameErr(String, String),
 }

+// Parent exception type for the different sub commands of the runner app.
+#[derive(Debug, Error)]
+pub enum PlotError {
+    #[error("{}", .0)]
+    PlotIOErr(IOError),
+    #[error("FilenameNotTimestampErr: {}", .0)]
+    FilenameNotTimestampErr(String),
+    #[error("BadJSONErr: JSON in file cannot be deserialized as expected.\nFilepath: {}\nOriginating Exception: {}", .0.to_string_lossy().into_owned(), .1.as_ref().map_or("None".to_owned(), |e| format!("{}", e)))]
+    BadJSONErr(PathBuf, Option<serde_json::Error>),
+    #[error("ChartErr: {}", .0)]
+    ChartErr(Box<dyn std::error::Error>),
+}
+
+// Parent exception type for the different sub commands of the runner app.
+#[derive(Debug, Error)]
+pub enum RunnerError {
+    #[error("CalculateErr: {}", .0)]
+    CalculateErr(CalculateError),
+    #[error("PlotErr: {}", .0)]
+    PlotErr(PlotError),
+}
+
 // Tests for exceptions
 #[cfg(test)]
 mod tests {
--- a/performance/runner/src/main.rs
+++ b/performance/runner/src/main.rs
@@ -3,10 +3,12 @@ extern crate structopt;
 mod calculate;
 mod exceptions;
 mod measure;
+mod plot;

 use crate::calculate::Calculation;
-use crate::exceptions::CalculateError;
-use std::fs::File;
+use crate::exceptions::{CalculateError, RunnerError};
+use chrono::offset::Utc;
+use std::fs::{metadata, File};
 use std::io::Write;
 use std::path::PathBuf;
 use structopt::StructOpt;
@@ -29,7 +31,11 @@ enum Opt {
        #[structopt(parse(from_os_str))]
        #[structopt(short)]
        results_dir: PathBuf,
+        #[structopt(parse(from_os_str))]
+        #[structopt(short)]
+        out_dir: PathBuf,
    },
+    Plot,
 }

 // enables proper useage of exit() in main.
@@ -37,7 +43,7 @@ enum Opt {
 //
 // This is where all the printing should happen. Exiting happens
 // in main, and module functions should only return values.
-fn run_app() -> Result<i32, CalculateError> {
+fn run_app() -> Result<i32, RunnerError> {
    // match what the user inputs from the cli
    match Opt::from_args() {
        // measure subcommand
@@ -48,7 +54,8 @@ fn run_app() -> Result<i32, CalculateError> {
            // if there are any nonzero exit codes from the hyperfine runs,
            // return the first one. otherwise return zero.
            measure::measure(&projects_dir, &branch_name)
-                .or_else(|e| Err(CalculateError::CalculateIOError(e)))?
+                .or_else(|e| Err(CalculateError::CalculateIOError(e)))
+                .or_else(|e| Err(RunnerError::CalculateErr(e)))?
                .iter()
                .map(|status| status.code())
                .flatten()
@@ -62,9 +69,21 @@ fn run_app() -> Result<i32, CalculateError> {
        }

        // calculate subcommand
-        Opt::Calculate { results_dir } => {
+        Opt::Calculate {
+            results_dir,
+            out_dir,
+        } => {
+            // validate output directory and exit early if it won't work.
+            let md = metadata(&out_dir)
+                .expect("Main: Failed to read specified output directory metadata. Does it exist?");
+            if !md.is_dir() {
+                eprintln!("Main: Output directory is not a directory");
+                return Ok(1);
+            }
+
            // get all the calculations or gracefully show the user an exception
-            let calculations = calculate::regressions(&results_dir)?;
+            let calculations = calculate::regressions(&results_dir)
+                .or_else(|e| Err(RunnerError::CalculateErr(e)))?;

            // print all calculations to stdout so they can be easily debugged
            // via CI.
@@ -77,9 +96,18 @@ fn run_app() -> Result<i32, CalculateError> {
            let json_calcs = serde_json::to_string_pretty(&calculations)
                .expect("Main: Failed to serialize calculations to json");

+            // if there are any calculations, use the first timestamp, if there are none
+            // just use the current time.
+            let ts = calculations
+                .first()
+                .map_or_else(|| Utc::now(), |calc| calc.ts);
+
            // create the empty destination file, and write the json string
-            let outfile = &mut results_dir.into_os_string();
-            outfile.push("/final_calculations.json");
+            let outfile = &mut out_dir.into_os_string();
+            outfile.push("/final_calculations_");
+            outfile.push(ts.timestamp().to_string());
+            outfile.push(".json");
+
            let mut f = File::create(outfile).expect("Main: Unable to create file");
            f.write_all(json_calcs.as_bytes())
                .expect("Main: Unable to write data");
@@ -105,6 +133,13 @@ fn run_app() -> Result<i32, CalculateError> {
                }
            }
        }
+
+        // plot subcommand
+        Opt::Plot => {
+            plot::draw_plot().or_else(|e| Err(RunnerError::PlotErr(e)))?;
+
+            Ok(0)
+        }
    }
 }

--- a/performance/runner/src/plot.rs
+++ b/performance/runner/src/plot.rs
@@ -0,0 +1,189 @@
+use crate::calculate::Calculation;
+use crate::exceptions::{IOError, PlotError};
+use chrono::prelude::*;
+use itertools::Itertools;
+use plotters::prelude::*;
+use std::cmp::Ordering;
+use std::fs;
+use std::fs::DirEntry;
+use std::path::{Path, PathBuf};
+
+struct Graph {
+    title: String,
+    data: Vec<(f32, f32)>,
+}
+
+impl Graph {
+    const DEFAULT_MIN_Y: f32 = -15.0;
+    const DEFAULT_MAX_Y: f32 = 15.0;
+    const DEFAULT_X_PADDING: f32 = 86400.0;
+
+    fn min_x(&self) -> f32 {
+        self.data
+            .clone()
+            .into_iter()
+            .map(|(x, _)| x)
+            .reduce(f32::min)
+            .unwrap()
+            - Graph::DEFAULT_X_PADDING
+    }
+    fn min_y(&self) -> f32 {
+        let min_data_point = self
+            .data
+            .clone()
+            .into_iter()
+            .map(|(_, y)| y)
+            .reduce(f32::min)
+            .unwrap();
+        f32::min(Graph::DEFAULT_MIN_Y, min_data_point)
+    }
+    fn max_x(&self) -> f32 {
+        self.data
+            .clone()
+            .into_iter()
+            .map(|(x, _)| x)
+            .reduce(f32::max)
+            .unwrap()
+            + Graph::DEFAULT_X_PADDING
+    }
+    fn max_y(&self) -> f32 {
+        let max_data_point = self
+            .data
+            .clone()
+            .into_iter()
+            .map(|(_, y)| y)
+            .reduce(f32::max)
+            .unwrap();
+        f32::max(Graph::DEFAULT_MAX_Y, max_data_point)
+    }
+}
+
+pub fn draw_plot() -> Result<(), PlotError> {
+    // TODO `as` type coersion sucks. swap it out for something safer.
+    let mut sorted_data: Vec<(NaiveDateTime, Calculation)> =
+        read_data(Path::new("plots/raw_data/"))?;
+    sorted_data.sort_by(|(ts_x, x), (ts_y, y)| {
+        // sort by calculation type, then by timestamp
+        match (&x.metric).cmp(&y.metric) {
+            Ordering::Equal => (&ts_x).cmp(&ts_y),
+            x => x,
+        }
+    });
+
+    let data_lines: Vec<Graph> = sorted_data
+        .into_iter()
+        .group_by(|(_, calc)| calc.metric.clone())
+        .into_iter()
+        .map(|(title, line)| Graph {
+            title: title.to_owned(),
+            data: line
+                .map(|(ts, calc)| (ts.timestamp() as f32, calc.data.difference as f32))
+                .collect(),
+        })
+        .collect();
+
+    for graph in data_lines {
+        let title = format!("plots/{}.png", graph.title);
+        let root = BitMapBackend::new(&title, (1600, 1200)).into_drawing_area();
+        root.fill(&WHITE)
+            .or_else(|e| Err(PlotError::ChartErr(Box::new(e))))?;
+        let root = root.margin(10, 10, 10, 10);
+
+        // build chart foundation
+        let mut chart = ChartBuilder::on(&root)
+            .caption(&graph.title, ("sans-serif", 40).into_font())
+            .x_label_area_size(20)
+            .y_label_area_size(40)
+            .build_cartesian_2d(graph.min_x()..graph.max_x(), graph.min_y()..graph.max_y())
+            .or_else(|e| Err(PlotError::ChartErr(Box::new(e))))?;
+
+        // Draw Mesh
+        chart
+            .configure_mesh()
+            .x_labels(5)
+            .y_labels(5)
+            .y_label_formatter(&|x| format!("{:.3}", x))
+            .draw()
+            .or_else(|e| Err(PlotError::ChartErr(Box::new(e))))?;
+
+        // Draw Line
+        chart
+            .draw_series(LineSeries::new(graph.data.clone(), &RED))
+            .or_else(|e| Err(PlotError::ChartErr(Box::new(e))))?;
+
+        // Draw Points on Line
+        chart
+            .draw_series(PointSeries::of_element(
+                graph.data.clone(),
+                5,
+                &RED,
+                &|c, s, st| {
+                    return EmptyElement::at(c)
+                        + Circle::new((0, 0), s, st.filled())
+                        + Text::new(format!("{:?}", c), (10, 0), ("sans-serif", 20).into_font());
+                },
+            ))
+            .or_else(|e| Err(PlotError::ChartErr(Box::new(e))))?;
+    }
+
+    Ok(())
+}
+
+fn read_data(results_directory: &Path) -> Result<Vec<(NaiveDateTime, Calculation)>, PlotError> {
+    fs::read_dir(results_directory)
+        .or_else(|e| Err(IOError::ReadErr(results_directory.to_path_buf(), Some(e))))
+        .or_else(|e| Err(PlotError::PlotIOErr(e)))?
+        .into_iter()
+        .map(|entry| {
+            let ent: DirEntry = entry
+                .or_else(|e| Err(IOError::ReadErr(results_directory.to_path_buf(), Some(e))))
+                .or_else(|e| Err(PlotError::PlotIOErr(e)))?;
+
+            Ok(ent.path())
+        })
+        .collect::<Result<Vec<PathBuf>, PlotError>>()?
+        .iter()
+        .filter(|path| {
+            path.extension()
+                .and_then(|ext| ext.to_str())
+                .map_or(false, |ext| ext.ends_with("json"))
+        })
+        .map(|p| {
+            // TODO pull this filename nonsense out into a lib fn
+            let filename = p
+                .file_stem()
+                .ok_or_else(|| IOError::MissingFilenameErr(p.to_path_buf()))
+                .and_then(|name| {
+                    name.to_str()
+                        .ok_or_else(|| IOError::FilenameNotUnicodeErr(p.to_path_buf()))
+                })
+                .or_else(|e| Err(PlotError::PlotIOErr(e)));
+
+            let timestamp: Result<NaiveDateTime, PlotError> = filename.and_then(|fname| {
+                fname
+                    .parse::<i64>()
+                    // not a timestamp because it's not a number
+                    .or_else(|_| Err(PlotError::FilenameNotTimestampErr(fname.to_owned())))
+                    .and_then(|secs| {
+                        // not a timestamp because the number is out of range
+                        NaiveDateTime::from_timestamp_opt(secs, 0)
+                            .ok_or_else(|| PlotError::FilenameNotTimestampErr(fname.to_owned()))
+                    })
+            });
+
+            let x: Result<Vec<(NaiveDateTime, Calculation)>, PlotError> =
+                timestamp.and_then(|ts| {
+                    fs::read_to_string(p)
+                        .or_else(|e| Err(IOError::BadFileContentsErr(p.clone(), Some(e))))
+                        .or_else(|e| Err(PlotError::PlotIOErr(e)))
+                        .and_then(|contents| {
+                            serde_json::from_str::<Vec<Calculation>>(&contents)
+                                .or_else(|e| Err(PlotError::BadJSONErr(p.clone(), Some(e))))
+                                .map(|calcs| calcs.iter().map(|c| (ts, c.clone())).collect())
+                        })
+                });
+            x
+        })
+        .collect::<Result<Vec<Vec<(NaiveDateTime, Calculation)>>, PlotError>>()
+        .map(|x| x.concat())
+}
--- a/test/integration/066_test_selection_tests/test_selection_expansion.py
+++ b/test/integration/066_test_selection_tests/test_selection_expansion.py
@@ -1,4 +1,5 @@
 from test.integration.base import DBTIntegrationTest, FakeArgs, use_profile
+import yaml

 from dbt.task.test import TestTask
 from dbt.task.list import ListTask
@@ -20,12 +21,18 @@ class TestSelectionExpansion(DBTIntegrationTest):
            "test-paths": ["tests"]
        }

-    def list_tests_and_assert(self, include, exclude, expected_tests):
+    def list_tests_and_assert(self, include, exclude, expected_tests, greedy=False, selector_name=None):
        list_args = [ 'ls', '--resource-type', 'test']
        if include:
            list_args.extend(('--select', include))
        if exclude:
            list_args.extend(('--exclude', exclude))
+        if exclude:
+            list_args.extend(('--exclude', exclude))
+        if greedy:
+            list_args.append('--greedy')
+        if selector_name:
+            list_args.extend(('--selector', selector_name))
        
        listed = self.run_dbt(list_args)
        assert len(listed) == len(expected_tests)
@@ -34,7 +41,7 @@ class TestSelectionExpansion(DBTIntegrationTest):
        assert sorted(test_names) == sorted(expected_tests)

    def run_tests_and_assert(
-        self, include, exclude, expected_tests, schema = False, data = False
+        self, include, exclude, expected_tests, schema=False, data=False, greedy=False, selector_name=None
    ):
        results = self.run_dbt(['run'])
        self.assertEqual(len(results), 2)
@@ -48,6 +55,10 @@ class TestSelectionExpansion(DBTIntegrationTest):
            test_args.append('--schema')
        if data:
            test_args.append('--data')
+        if greedy:
+            test_args.append('--greedy')
+        if selector_name:
+            test_args.extend(('--selector', selector_name))
        
        results = self.run_dbt(test_args)
        tests_run = [r.node.name for r in results]
@@ -228,3 +239,80 @@ class TestSelectionExpansion(DBTIntegrationTest):
        
        self.list_tests_and_assert(select, exclude, expected)
        self.run_tests_and_assert(select, exclude, expected)
+
+    @use_profile('postgres')
+    def test__postgres__model_a_greedy(self):
+        select = 'model_a'
+        exclude = None
+        greedy = True
+        expected = [
+            'cf_a_b', 'cf_a_src', 'just_a',
+            'relationships_model_a_fun__fun__ref_model_b_',
+            'relationships_model_a_fun__fun__source_my_src_my_tbl_',
+            'unique_model_a_fun'
+        ]
+            
+        self.list_tests_and_assert(select, exclude, expected, greedy)
+        self.run_tests_and_assert(select, exclude, expected, greedy=greedy)
+
+    @use_profile('postgres')
+    def test__postgres__model_a_greedy_exclude_unique_tests(self):
+        select = 'model_a'
+        exclude = 'test_name:unique'
+        greedy = True
+        expected = [
+            'cf_a_b', 'cf_a_src', 'just_a',
+            'relationships_model_a_fun__fun__ref_model_b_',
+            'relationships_model_a_fun__fun__source_my_src_my_tbl_',
+        ]
+            
+        self.list_tests_and_assert(select, exclude, expected, greedy)
+        self.run_tests_and_assert(select, exclude, expected, greedy=greedy)
+
+class TestExpansionWithSelectors(TestSelectionExpansion):
+
+    @property
+    def selectors_config(self):
+        return yaml.safe_load('''
+            selectors:
+            - name: model_a_greedy_none
+              definition:
+                method: fqn
+                value: model_a
+            - name: model_a_greedy_false
+              definition:
+                method: fqn
+                value: model_a
+                greedy: false
+            - name: model_a_greedy_true
+              definition:
+                method: fqn
+                value: model_a
+                greedy: true
+        ''')
+
+    @use_profile('postgres')
+    def test__postgres__selector_model_a_not_greedy(self):
+        expected = ['just_a','unique_model_a_fun']
+        
+        # when greedy is not specified, so implicitly False
+        self.list_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_none')
+        self.run_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_none')
+
+        # when greedy is explicitly False
+        self.list_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_false')
+        self.run_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_false')
+
+
+    @use_profile('postgres')
+    def test__postgres__selector_model_a_yes_greedy(self):
+        expected = [
+            'cf_a_b', 'cf_a_src', 'just_a',
+            'relationships_model_a_fun__fun__ref_model_b_',
+            'relationships_model_a_fun__fun__source_my_src_my_tbl_',
+            'unique_model_a_fun'
+        ]
+
+        # when greedy is explicitly False
+        self.list_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_true')
+        self.run_tests_and_assert(include=None, exclude=None, expected_tests=expected, selector_name='model_a_greedy_true')
--- a/test/unit/test_graph_selection.py
+++ b/test/unit/test_graph_selection.py
@@ -120,7 +120,7 @@ def test_run_specs(include, exclude, expected):
    manifest = _get_manifest(graph)
    selector = graph_selector.NodeSelector(graph, manifest)
    spec = graph_cli.parse_difference(include, exclude)
-    selected = selector.select_nodes(spec)
+    selected, _ = selector.select_nodes(spec)

    assert selected == expected
Author	SHA1	Message	Date
Nathaniel May	42d71f5a97	fmt	2021-09-22 10:33:46 -04:00
Nathaniel May	58dc3b1829	remove comment nonsense	2021-09-22 10:30:29 -04:00
Nathaniel May	bb9a400d77	make scale more useful	2021-09-22 10:30:29 -04:00
Nathaniel May	01366be246	plots all detected metrics	2021-09-22 10:30:29 -04:00
Nathaniel May	b034e2bc66	it graphs, but poorly for now.	2021-09-22 10:30:29 -04:00
Nathaniel May	3bc9f49f7a	add plot subcommand exceptions to hierarchy	2021-09-22 10:30:29 -04:00
Nathaniel May	09a61177b4	don't commit plots	2021-09-22 10:27:36 -04:00
Nathaniel May	e8d3efef9f	doesn't compile. saving.	2021-09-22 10:27:36 -04:00
Nathaniel May	f9a46c15b9	sample plot runs	2021-09-22 10:27:09 -04:00
Nathaniel May	7dc491b7ba	Merge pull request #3936 from dbt-labs/regression-test-tweaks Performance Regression Testing: Add timestamps to results and make filenames unique.	2021-09-22 10:18:24 -04:00
Nathaniel May	59d131d3ac	add timestamps to results, and make filenames unique	2021-09-21 18:10:37 -04:00
Joel Labes	6563d09ba7	Add logging for skipped resources (#3833 ) Add --greedy flag to subparser Add greedy flag, override default behaviour when parsing union Add greedy support to ls (I think?!) That was suspiciously easy Fix flake issues Try adding tests with greedy support Remove trailing whitespace Fix return type for select_nodes forgot to add greedy arg Remove incorrectly expected test Use named param for greedy Pull alert_unused_nodes out into its own function rename resources -> tests Add expand_selection explanation of --greedy flag Add greedy support to yaml selectors. Update warning, tests Fix failing tests Fix tests. Changelog Fix flake8 Co-authored-by: Joel Labes c/o Jeremy Cohen <jeremy@dbtlabs.com>	2021-09-18 19:58:38 +02:00