44 Commits

Author SHA1 Message Date
colin-rogers-dbt
15dcb9a19d Remove dbt-postgres and dbt-tests-adapter from dbt-core (#9492)
* delete dbt-tests-adapter and dbt-postgres from dbt-core

* update non-code files to reflect change

* add changie

* update build-dist.sh

* update tox.ini

* fix build-dist.sh

* move adapter tests into /functional dir

* remove adapter unit tests

* update code comments and README.md
2024-02-05 12:28:57 -08:00
Jeremy Cohen
3789acc5a7 Move redshift, snowflake, bigquery plugins (#3969)
* Consolidate dbt-redshift

Start moving unit tests

fix typo; rm breakpoints

rm references to redshift

rm all references to redshift except `integration/100_rpc_test` 🙁

cleanup CI

fix tests!

fix docs test bc of schema version change

* Consolidate dbt-snowflake

Continue pulling apart Snowflake tests

Fixups

Rip out even more

* Consolidate dbt-bigquery

* Rm redshift, snowflake, bigquery from plugins/

* Comment out rs, sf, bq profiles in 100_rpc_test

* Add changelog, readme notes

Co-authored-by: Kyle Wigley <kyle@dbtlabs.com>
2021-10-02 10:30:09 -07:00
Jacob Beck
f94a7a73c1 More packaging fixes
- Add both 'dbt' and 'dbt.*' to the namespace package search path
- set zip_safe=False, because I think zipping breaks macros
- Make CI tests use non-editable installs
- make tox use editable mode for explicit-* so development is still possible without rebuilding tox
- remove {toxinidir} stuff that breaks core installation
  - as of tox 1.6.1, requirements executes within toxinidir.
- fix paths to account for installing things
2019-10-21 09:52:29 -06:00
Connor McArthur
c417c2011b remove presto from requirements.txt 2019-02-14 14:19:58 -05:00
Jacob Beck
85389afb3e Presto testo 2019-01-16 16:47:48 -07:00
Jacob Beck
4780c4bb18 Split dbt into core and plugins 2018-12-14 11:17:41 -07:00
Jacob Beck
60c4619862 Make adapters optional 2018-11-12 11:30:28 -07:00
Jacob Beck
228524fa46 Merge pull request #870 from clrcrl/jinja-upgrade
Upgrade version of Jinja2
2018-09-17 08:40:38 -06:00
Connor McArthur
6ef1ef6a3d pin botocore to the associated versions from boto3 2018-09-04 20:01:50 -04:00
Drew Banin
d56800f638 set upper bound on boto3 version 2018-08-29 14:37:10 -04:00
Jacob Beck
f739bd3927 Set google-cloud-bigquery dependency to >=1, <2 2018-08-13 14:40:11 -06:00
Claire Carroll
ff64f8166a Upgrade version of Jinja 2018-07-22 16:05:33 +10:00
Connor McArthur
3794336920 pin requests <3 2018-07-19 09:11:18 -04:00
Connor McArthur
9d8275c7d6 update lots of deps 2018-07-18 15:11:51 -04:00
Connor McArthur
6bf8028b1f switch to minimal-snowplow-tracker 2018-07-12 20:15:21 -04:00
Drew Banin
e20ac4193b Merge branch 'development' into feature/redshift-iam-auth 2018-07-02 13:28:52 -04:00
Drew Banin
a382def2d0 Bump/psycopg2 2.7.5 (#807)
* use bin psycopg2 dep, bump to 2.7.5

* use psycopg-2 source package, bump to 275
2018-06-28 13:58:35 -04:00
Drew Banin
457db9d09e Merge betsy-ross release code into development (#798)
Merge dev/betsy-ross branch into development
2018-06-18 14:43:52 -04:00
Daniel Chalef
bc432f9584 added iam authentication method for redshift adapter 2018-05-10 13:00:52 -07:00
Connor McArthur
5344f54c3c Implement relations api (#727) 2018-04-26 17:38:44 -04:00
Drew Banin
d966ec28aa add requests dep to fix broken install (#732) 2018-04-10 14:59:16 -04:00
Drew Banin
79c60f68c5 rm freezegun deps - this is only needed in the dev reqs (#695) 2018-04-06 13:16:18 -04:00
Drew Banin
4eb75ec5b6 Bq date partitioning (#641)
* first cut of date partitioning

* cleanup, implement partitioning in materialization

* update requirements.txt

* wip for date partitioning with range

* log data

* arg handling, logging, cleanup + view compat for new bq version

* add partitioning tests, compatibility with bq 0.29.0 release

* pep8

* fix for strange error in appveyor

* debug appveyor...

* dumb

* debugging weird bq adapter use in pg test

* do not use read_project in bq tests

* cleanup connections, initialize bq tests

* remove debug lines

* fix integration tests (actually)

* warning for view creation which clobbers tables

* add query timeout example for bq

* no need to release connections in the adapter

* partition_date interface change (wip)

* list of dates for bq dp tables

* tiny fixes for crufty dbt_project.yml files

* rm debug line

* fix tests
2018-02-12 16:09:59 -05:00
Buck Ryan
0372fefae0 Seed rewrite (#618)
* loader for seed data files

* Functioning rework of seed task

* Make CompilerRunner fns private and impl. SeedRunner.compile

Trying to distinguish between the public/private interface for this
class. And the SeedRunner doesn't need the functionality in the compile
function, it just needs a compile function to exist for use in the
compilation process.

* Test changes and fixes

* make the DB setup script usable locally

* convert simple copy test to use seeed

* Fixes to get Snowflake working

* New seed flag and make it non-destructive by default

* Convert update SQL script to another seed

* cleanup

* implement bigquery csv load

* context handling of StringIO

* Better typing

* strip seeder and csvkit dependency

* update bigquery to use new data typing and to fix unicode issue

* update seed test

* fix abstract functions in base adapter

* support time type

* try pinning crypto, pyopenssl versions

* remove unnecessary version pins

* insert all at once, rather than one query per row

* do not quote field names on creation

* bad

* quiet down parsedatetime logger

* pep8

* UI updates + node conformity for seed nodes

* add seed to list of resource types, cleanup

* show option for CSVs

* typo

* pep8

* move agate import to avoid strange warnings

* deprecation warning for --drop-existing

* quote column names in seed files

* revert quoting change (breaks Snowflake). Hush warnings
2018-02-10 11:28:24 -05:00
Drew Banin
850d00ccbe change req for snowflake to fix crypto install issue (#612)
From cffi callback <function _verify_callback at 0x06BF2978>:
Traceback (most recent call last):
  File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: module 'lib' has no attribute 'X509_up_ref'
From cffi callback <function _verify_callback at 0x06B8CF60>:
2017-12-04 15:51:06 -05:00
Drew Banin
2afc202678 Make run_started_at timezone aware (#553) (#556)
* Make run_started_at timezone aware

Set run_started_at timezone to UTC
Enable timezone change in models
Extend requirements
Extend tests

* Address comments from code review

Create modules namespace to context
Move pytz to modules
Add new dependencies to setup.py
2017-12-02 17:21:20 -05:00
Drew Banin
c82d69f115 revert jinja version, implement tojson hack (#572) 2017-10-24 10:00:23 -04:00
Connor McArthur
fdf4a42eba bump snowflake, remove pyasn1 (#570)
* bump snowflake, remove pyasn1

* change requirements.txt
2017-10-23 09:38:27 -04:00
Connor McArthur
3489e90f28 tojson was added in jinja 2.9 (#563)
* tojson was added in jinja 2.9

* requirements
2017-10-20 18:44:20 -04:00
Drew Banin
7a5670b977 Feature/bq improvements (#507)
* request gdrive scope

* re-add missing lines?

* whitespace

* remove junk

* bq tables working

* optionally override query timeout

* bump requirements.txt

* pep8

* don't mutate global state
2017-08-29 15:02:09 -04:00
Connor McArthur
a37374d1a4 better error handling (#494) 2017-07-26 16:34:08 -04:00
Drew Banin
8b8b3aa47f Bigquery (#455)
* wip

* minimum viable bq adapter

* proper bq quoting

* absolute madness

* auto-install bq module

* catch runtime exception for nodes

* cleanup

* more cleanup

* pip critical logging

* add adapter command

* major wip

* refactorin

* closer

* handle model errors + dependent skips

* cleanup + test transactions (for now)

* move model creation to materializers

* fix for ephemeral models

* override materializer for bq

* error handling

* bq tests working

* commit tweaks for models

* service accounts

* service account json for bq

* better error message if adapter is not installed

* fix unit tests

* pep8

* fix integration tests

* codeclimate

* fix typos

* fix circular dep for imports

* catch programming exception for runners

* code review changes

* refactoring for code climate

* selector cleanup

* fix bug for erin

* comment

* handle cancellation on ctrl-c for bq (log warning)

* better bq validation

* test bq validation

* add uninstall flag to adapter task

* remove pip hacking nonsense

* bq integration tests

* remove initialize call for pg

* fix bq integration tests

* pep8

* remove -x opt from toxfile

* handle notimplemented for bq better

* missing import for seed task

* notimplemented for bq
2017-06-19 14:36:38 -04:00
Drew Banin
dfb24fdaf3 colorized dbt output (#441)
colorize dbt output
2017-05-23 14:52:27 -04:00
Connor McArthur
ce5f2f41d0 snowflake in windows (#276) 2017-02-07 15:09:55 -05:00
Connor McArthur
cd35aa80ee upgrade snowflake-connector-python to 1.3.12 2017-02-05 11:25:31 -05:00
Connor McArthur
172c7cef4a add snowflake support (#259) 2017-01-20 14:00:03 -05:00
Drew Banin
2a7d059ebd add celery dep to fix snowplow dep 2016-09-19 16:38:15 -04:00
Drew Banin
9fbe1f40aa add V1 of event tracking 2016-08-21 18:40:09 -04:00
Drew Banin
fef1f40410 working! needs more testing and error handling 2016-08-20 02:35:32 -04:00
Drew Banin
7192550b86 add csvkit dependency 2016-07-14 12:36:57 -04:00
Drew Banin
58a06c7cfc Merge pull request #9 from analyst-collective/dagnabit
DAGnabit
2016-04-03 17:21:21 -04:00
Drew Banin
a4482bde33 add argparse support 2016-04-01 13:35:17 -04:00
Drew Banin
cccbba8081 first cut 2016-03-21 17:29:15 -04:00
Christopher Merrick
b849af25e3 project and profile scaffolding with clean, compile, and run tasks 2016-03-16 00:53:43 -04:00