Dikootje/sqlfluff
1
0
Fork 0
forked from repo-mirrors/sqlfluff
Code Pull Requests Activity
Files
main
sqlfluff/CHANGELOG.md
github-actions[bot] 36ad37b2a3 Prep version 3.2.5 (#6401) 
Co-authored-by: alanmcruickshank <alanmcruickshank@users.noreply.github.com>
Co-authored-by: Alan Cruickshank <alan+git@designingoverload.com>
2024-10-25 10:14:01 +01:00

551 KiB
Raw Permalink Blame History

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[3.2.5] - 2024-10-25

Highlights

This release is mostly bugfixes and dialect improvements. Notably:

  • Whitespace handling improvements to LT01 & LT02.
  • Better error messages around trying to iterate on missing jinja variables.
  • Better case sensitivity for AL09.
  • Improved handling of jinja context in inline config directives.
  • Enabling AM02 for Trino and Snowflake.
  • Handling potential collisions between ST02 & LT01.
  • Preventing false positives in AL05 with arrays.

There's also a bunch of documentation improvements in this release, including guides on how to troubleshoot SQLFluff and how to write custom rules. Check out https://docs.sqlfluff.com for more details.

We also saw five new contributors to the project this month. Welcome to the project, and thanks for taking the time to contribute! 🎉🏆🎉

What’s Changed

New Contributors

[3.2.4] - 2024-10-14

Highlights

This release is almost all dialect fixes and bugfixes. Notably also, this release brings official python 3.13 support too (although most users should not realise any differences).

We also saw two new contributors to the project. Welcome @R3gardless & @brandonschabell! 🎉🎉🎉

What’s Changed

New Contributors

[3.2.3] - 2024-10-10

Highlights

This is another release of dialect improvements and rule bugfixes. Notably:

  • More robust algorithms for the indentation of Jinja template tags in LT02.
  • The github-annotation-native format option now has groups for each filename.

There's also a refactor of where we guides and howtos in the docs. Keep an eye on that section going forward for more information about best practice and troubleshooting for SQLFluff.

Even in this small PR, we've seen two new contributors. Welcome @nspcc-cm & @rogalski to the project!

What’s Changed

New Contributors

[3.2.2] - 2024-10-07

Highlights

This is a hotfix release to resolve an issue with the JJ01 rule when running in parallel mode.

What’s Changed

[3.2.1] - 2024-10-06

Highlights

This release is primarily housekeeping, bugfixes and dialect improvements. More specifically:

  • Resolving regressions regressions in JJ01, filename extension handling and the treatment of unfixable/unparsable files, which have been noticed with recent releases.
  • Resolving bugs in LT07 & LT12 which relate to jinja whitespace control.
  • More robust support for arbitrary methods on the ref and source macros for the dbt templater.

There's also dialect improvements for BigQuery, TSQL, MySQL, MariaDB, Snowflake, DuckDB, Databricks, Postgres, Teradata, Exasol & Vertica.

We also saw six new contributors merge their first pull request as part of this release. Welcome to the project! 🎉🏆🎉

What’s Changed

New Contributors

[3.2.0] - 2024-09-18

Highlights

This release brings a few minor breaking changes, both for the core project and for the dbt templater. For the main project:

  • Resolving an issue with the spacing of functions (LT01), which involved a change to how functions are parsed. If your project relies on the specific parsing of functions, the bracketed arguments are now wrapped in a function_contents object. We recommend that you examine the new parsing structure using this new release in testing first.

  • RF06 (references.quoting) is now case sensitive when removing quotes which are detected as unnecessary. This rule has also been re-enabled by default for Snowflake and Postgres where it had previously been disabled (for the reason that in the past it hadn't been appropriately case sensitive). Treatment for totally case-insensitive dialects like DuckDB and SparkSQL have also been included. Please check the new documentation for this rule (which is much more explicit now), for details related to your dialect.

  • Patterns equivalent to those from .sqlfluffignore can now be included in .sqlfluff and pyproject.toml files.

  • Using the python templater, users now have an option to include variables which include a dot in the path, like {{ foo.bar }} using a special sqlfluff context variable.

  • Significant changes under the hood to the handling of configuration files. Most of these should not be visible to end users, but for anyone integrating SQLFluff into a larger project and relying on native file loading may need to refactor their project for this release. Most notably here, for maintainers of plugins, the ConfigLoader class has been deprecated, and plugins should instead call the config loading functions directly. See the example plugin for details.

  • Documentation, especially for dialects, has been significantly improved. Documentation for CP02 (capitalisation.identifiers) has also been clarified to make it's implication for references and aliases more clear.

  • During testing, to isolate the effect of specific rules, there's a new CLI option --disable-noqa-except which allows all noqa options to be ignored except the ones provided in this option.

For the dbt templater:

  • Support for dbt 1.1-1.3 has been removed. All have been in End of Life (EOL) support by dbtlabs for almost two years. They are also poorly supported by other projects and tools.

  • The dbt templater has been migrated to use pyproject.toml.

  • Handling of errors and exceptions raised within dbt has had an overhaul. Users may see a slightly different presentation of errors, but the overall stability should be more robust.

In addition to those changes, there have been too many dialect contributions and bugfixes to mention specifically. We've also seen six people make their first contributions to the project as part of preparing for this release! 🎉🏆🎉.

What’s Changed

New Contributors

[3.1.1] - 2024-08-20

Highlights

This release brings a bumper lot of bugfixes, dialect improvements and other minor improvements across the board. Most notably:

  • A rework of the structure of the docs. NOTE: This does change the url of some docs pages, but to prevent future moves, we've also provided permalinks to most important pages and rules. See the conf.py file in the docs folder for a full list of permalinks.
  • Solving rule conflicts between LT02 & LT02.
  • Bugfixes to AM07, CV11, ST03, ST05 & RF03,
  • Removes some redundant dependencies in for the dbt templater (which haven't been required for some time, but have been included in the install dependencies). Specifically: markupsafe, ruamel.yaml, pydantic & rich.
  • And too many dialect improvements to summarise!

We've also seen eleven new contributors to the project! Thanks to all of them for taking the time to contribute. 🎉🎉🏆🎉🎉

What’s Changed

New Contributors

[3.1.0] - 2024-07-03

Highlights

This minor release has two breaking changes:

  • The addition of camelCase in the extended capitalisation policy. This change removes the ability to autodetect PascalCase, from now on PascalCase, and camelCase must be explicitly set in the config if desired.
  • The detection method for sqlfluff config has changed. It should now be more consistent, regardless of how deep if the directory structure you run the command from.

This release also brings in support for the MariaDB dialect. As well as this, there are many bugfixes, and dialect improvements.

Thanks also to the twelve new contributors whose work was included in this release! 🎉🎉🏆🎉🎉

What’s Changed

New Contributors

[3.0.7] - 2024-05-23

Highlights

This is primarily a fix for compatibility with dbt 1.8+. Beyond that it also brings several dialect improvements to SQLite, Bigquery, MySQL, Oracle & Clickhouse.

Thanks also to the five new contributors whose work was included in this release! 🎉🎉🏆🎉🎉

What’s Changed

New Contributors

[3.0.6] - 2024-05-06

Highlights

This release primarily fixes an issue introduced by the recent dbt 1.7.14 release, and better support for dbt 1.7+. It also includes a range of dialect improvements and CLI refinements.

This release also includes the groundwork for linting the unrendered sections of Jinja templates. More documentation on this will be released in due course when it's ready for beta testing.

Thanks also to @padraic00 & @burhanyasar who made their first contributions in this release. 🎉🎉🏆🎉🎉

What’s Changed

New Contributors

[3.0.5] - 2024-04-19

Highlights

This release contains one larger change, which is a big upgrade to case sensitivity in the alias use rules. Also allowing the customisation of how SQLFluff uses case sensitivity in rules like AL05. Beyond that, this also includes a handful of dialect improvements.

Thanks especially to @olshak, @MarkPaulin, @mhoogendoorn & @kawashiro who made their first contributions in this release! 🚀

What’s Changed

New Contributors

[3.0.4] - 2024-04-07

Highlights

This is a standard bugfix release bringing a bunch of dialect improvements and bugfixes. Almost every dialect sees some improvements and it also includes quality of life improvements to the CLI, pre-commit hooks, docs and several rules.

Thanks also to the eight new contributors whose first contributions are included in this release. 🎉🎉🏆🎉🎉

What’s Changed

New Contributors

[3.0.3] - 2024-03-22

Highlights

This is a standard minor release fixing a set of dialect issues with Trino, BigQuery, Vertica and Snowflake.

Thanks to @maegan-canva, @rileymcdowell & @paysni who made their first contributions in this release.

What’s Changed

New Contributors

[3.0.2] - 2024-03-17

Highlights

This is primarily another hotfix release for 3.0.0. Specifically making sure the deprecation warnings for -f/--force go to stderr rather than stdout. It also includes two dialect improvements, one for Snowflake and one for T-SQL.

What’s Changed

[3.0.1] - 2024-03-13

Highlights

This minor release is a hotfix to resolve a bug introduced affecting CLI exit codes in the 3.0.0 release.

What’s Changed

[3.0.0] - 2024-03-12

Highlights

This release brings several breaking changes to previous releases. Most notably:

  • It drops support for python 3.7, which reached end of life in June 2023.

  • It migrates to pyproject.toml rather than setup.cfg as the python packaging configuration file (although keeping setuptools as the default backend).

  • The serialised output for sqlfluff lint (and the corresponding API methods) now contains more information about the span of linting issues, initial proposed fixes and several statistics which were previously only accessible via csv export. Beside the new fields, the original fields of line_pos and line_no have been renamed to start_line_pos and start_line_no, to distinguish them from the new fields starting end_*.

  • The default annotation_level set by the --annotation-level option on the sqlfluff lint command has been changed from notice to warning, to better distinguish linting errors from warnings, which always now have the level of notice. This is only relevant when using the github-annotation or github-annotation-native formats.

  • A change in the default behaviour for convention.not_equals. The new default is to be consistent, which is slightly more relaxed than the original behaviour.

  • The --force option has been deprecated on sqlfluff fix as that option is now the default behaviour. This is to enable significant reductions in memory overhead when linting large projects.

  • The long since deprecated --disable_progress_bar option has been removed (which was replaced by the kabab-case --disable-progress-bar more than a year ago).

  • Plugins are now loaded progressively, and with error handling. If a plugin fails to load, SQLFluff will now continue onward and try to run regardless while also showing a more helpful error message.

On top of these changes, there have a been a whole host of dialect improvements and additions, in particular the inclusion of avertica dialect for the first time. There's also:

  • A new rule (aliasing.self_alias.column) which prevents aliasing a column as itself.

  • A change to disables AL01 (aliasing.table) by default for Oracle.

  • A change to allow AL05 to allow aliasing for a VALUES clause.

For more specifics please take a look at the release notes.

Thanks to the community for patience during the release cycle for 3.0.0, which has taken a little longer than expected. Thanks also to the TWENTY SEVEN new contributors whose changes are included in this release. 🎉🎉🏆🎉🎉

What’s Changed

New Contributors

[3.0.0a6] - 2024-03-05

Highlights

This introduces some memory optimisations in the linting operation which prevent a major cause of crashes when linting large projects. As part of that we've also deprecated the --force option on sqlfluff fix and made that the default behaviour (the associated memory optimisations will come shortly).

This also removes the long since deprecated --disable_progress_bar option (which was replaced by the kabab-case --disable-progress-bar more than a year ago).

On top of that this release also introduces the vertica dialect for the first time, and a whole host of bugfixes and improvements to other dialects.

This release should be considered a release candidate for the final 3.0.0 release which will follow shortly in the next few days unless any other major issues are found.

Thanks particularly to the seven new contributors we saw in this release 🏆🎉.

What’s Changed

New Contributors

[3.0.0a5] - 2024-01-30

Highlights

This release primarily brings through a large set of dialect improvements and bugfixes from over the holiday period. Notably also:

  • A change in the default behaviour for convention.not_equals. The new default is to be consistent, which is slightly more relaxed than the original behaviour.
  • A new rule (aliasing.self_alias.column) which prevents aliasing a column as itself.
  • Disables AL01 (aliasing.table) by default for Oracle.

This release also saw ELEVEN new contributors in this release 🎉🎉🏆🏆🎉🎉. Great to see so many new people getting involved with the project. Thank You 🙏.

What’s Changed

New Contributors

[3.0.0a4] - 2023-12-05

Highlights

This release makes a breaking change to the serialized output of the CLI (and by extension, any of the serialized outputs of the API).

  • The serialised output for sqlfluff lint now contains more information about the span of linting issues and initial proposed fixes. Beside the new fields, the original fields of line_pos and line_no have been renamed to start_line_pos and start_line_no, to distinguish them from the new fields starting end_*.

  • The default annotation_level set by the --annotation-level option on the sqlfluff lint command has been changed from notice to warning, to better distinguish linting errors from warnings, which always now have the level of notice. This is only relevant when using the github-annotation or github-annotation-native formats.

What’s Changed

[3.0.0a3] - 2023-11-29

Highlights

This brings no further breaking changes on top of 3.0.0a2, but instead releases a few of the more minor fixes flowing through while other breaking changes are staged in. In particular there are a few dialect improvements for Snowflake, TSQL, SQLite and Databricks alongsids a few further improvements to deployment scripts off the back of earlier changes for 3.x.

What’s Changed

New Contributors

[3.0.0a2] - 2023-11-09

Highlights

The initial 3.0.0a1 release failed to build a docker image, this resolves that issue.

What’s Changed

[3.0.0a1] - 2023-11-08

Highlights

This release makes a couple of potentially breaking changes:

  • It drops support for python 3.7, which reached end of life in June 2023.

  • It migrates to pyproject.toml rather than setup.cfg as the python packaging configuration file (although keeping setuptools as the default backend).

Further breaking changes may be made as part of the full 3.0.0 release, but this alpha release is designed to test the new packaging changes for any issues before releasing a stable version.

What’s Changed

New Contributors

[2.3.5] - 2023-10-27

Highlights

This is a fairly minor release, primarily bugfixes and dialect improvements.

For python API users, there's the addition of a public method on the FluffConfig object allowing the construction of a config object from multiple strings to mimic the effect of nested config files in the CLI.

This release also includes a selection of internal refactoring and reorganisation to support future development work.

This also sees the first contributions by @ShubhamJagtap2000 & @kang8, particularly notable in that both were contributions to SQLFluff documentation! 🎉🎉🏆🎉🎉

What’s Changed

New Contributors

[2.3.4] - 2023-10-17

Highlights

This is a fairly small bugfix release, mostly to resolve a bug introduced in 2.3.3 with commas and LT09. This also includes a couple of additional small performance improvements and some dialect improvements for Oracle, BigQuery and MySQL.

Thanks in particular to @bonnal-enzo who made their first contribution as part of this release 🎉🎉🏆🎉🎉.

What’s Changed

New Contributors

[2.3.3] - 2023-10-13

Highlights

There's a lot in this release. Most of it is under the covers and so shouldn't cause any breaking changes for most users. If your use case depends on some of the internals of SQLFluff, you may find some breaking changes. The bigger changes are:

  • Python 3.12 support is now official (although older releases may also work as only a few changes were required for full 3.12 support).
  • We've done a significant re-write of the parsing engine to remove some unnecessary segment manipulation and get us closer to "single pass" parsing. This changes the internal API being used on any .match() methods, and also removes the parse_grammar attribute on any dialect segments. We are not aware of any 3rd party libraries which rely on these APIs however and so have not triggered a more major release. These lead to significant performance improvements during parsing.
  • Standardisation of terminators in the parser, and the introduction of the ParseMode option has enabled the removal of the StartsWith, GreedyUntil and EphemeralSegment parser classes.
  • Several validation checks have been revised in this release, which should both improve performance (by reducing duplication), but also be more effective in preventing the application of any fixes which would result in unparsable files.

Alongside the big things this also includes a host of bugfixes, dialect improvements and CI/testing improvements.

This release also sees a bumper crop of new contributors, thanks to @dehume, @andychannery, @Kylea650, @robin-alphasophia, @jtbg, @r-petit, @bpfaust & @freewaydev who all made the first contributions in this release! 🎉🎉🎉

What’s Changed

New Contributors

[2.3.2] - 2023-09-10

Highlights

Much of this release is internal optimisations and refactoring. We're in the process of upgrading some quite old code in the parser, most of which should not be visible to end users (apart from perhaps some performance improvements!).

This release also allows missing template variables in the placeholder templater to be automatically filled with the name of the variable rather than raising an error (see: #5101).

Beyond that this includes some dialect improvements for DuckDB, SparkSQL, Snowflake, Redshift & Postgres.

Thanks particularly to @shyaginuma, @Fullcure3, @adilkhanekt & @pilou-komoot who made their first contributions as part of this release. 🎉🎉🎉

What’s Changed

New Contributors

[2.3.1] - 2023-08-29

Highlights

This release is primarily a performance release, with most major changes aimed at the linting and fixing phases of operation. Most of the longest duration rules (excepting the layout rules) should see noticeable speed improvements.

Alongside those changes, there are a selection of bugfixes and dialect improvements for Oracle, PostgreSQL, Snowflake & TSQL.

What’s Changed

[2.3.0] - 2023-08-14

Highlights

This release brings one new dialect, two new rules and some changes to the CLI:

  • We now support the trino dialect. This is a first version of support, so do post any issues on GitHub in the usual way. This was also the first contribution to the project from @efung 🏆.
  • ST09 / structure.join_condition_order: Which checks whether tables referenced in JOIN clauses are referenced in the order of their definition. By default this means that in the ON clause, the column referencing the table in the FROM clause should come before the column referencing the table in the JOIN clause (e.g. ... FROM a JOIN b on a.c = b.c). This rule was also the first contribution to the project from @thibonacci 🏆.
  • AL08 / aliasing.unique.column: Which checks that column aliases and names are not repeated within the same SELECT clause. This is normally an error as it implies the same column has been imported twice, or that two expressions have been given the same alias.
  • The --profiler option on sqlfluff parse has been removed. It was only present on the parse command and not lint or fix, and it is just as simple to invoke the python cProfiler directly.
  • The --recurse cli option and sqlfluff.recurse configuration option have both been removed. They both existed purely for debugging the parser, and were never used in a production setting. The improvement in other debugging messages when unparsable sections are found means that this option is no longer necessary.

Along side these more significant changes this also includes:

  • Performance optimisations for AL04, AL05, AM04, RF01 & ST05 which cumulatively may save up to 30% on the total time spend in the linting phase for some projects.
  • Dialect improvements for Oracle & TSQL.

What’s Changed

New Contributors

[2.2.1] - 2023-08-09

Highlights

This is primarily a bugfix release for 2.2.0 which introduced a bug in the exit_code returned by linting commands which ignored errors while setting processes > 1.

In addition to that this release introduces bugfixes for:

  • Errors raised by two specific dbt exceptions.
  • Issues with unwanted logging output when using -f yaml or -f json alongside the dbt templater.

This also introduces dialect improvements for Oracle and for LIMIT clauses.

Thanks also to @adityapat3l who made their first contribution as part of this release! 🎉🎉🎉

What’s Changed

New Contributors

[2.2.0] - 2023-08-04

Highlights

This release changes some of the interfaces between SQLFluff core and our plugin ecosystem. The only breaking change is in the interface between SQLFluff and templater plugins (which are not common in the ecosystem, hence why this is only a minor and not a major release).

For all plugins, we also recommend a different structure for their imports (especially for rule plugins which are more common in the ecosystem) - for performance and stability reasons. Some users had been experiencing very long import times with previous releases as a result of the layout of plugin imports. Users with affected plugins will begin to see a warning from this release onward, which can be resolved for their plugin by updating to a new version of that plugin which follows the guidelines.

For more details (especially if you're a plugin maintainer) see our release notes.

Additionally this release includes:

  • Some internal performance gains which may cumulatively save roughly 10% of the time spent in the parsing phase of larger files.
  • Improvements to the Simple API, including the ability to pass in a FluffConfig object directly, and better support for parsing config files directly from strings (see the included example).
  • A bugfix for AM06.
  • A new --warn-unused-ignores CLI option (and corresponding config setting) to allow warnings to be shown if any noqa comments in SQL files are unused.
  • Improvements to Redshift, Oracle, Clickhouse, Materialize & MySQL dialects.
  • A selection of internal improvements, documentation and type hints.

Thanks also to @kaiyannameighu, @josef-v, @aglebov & @joaostorrer who made their first contributions as part of this release! 🎉🎉🎉

What’s Changed

New Contributors

[2.1.4] - 2023-07-25

Highlights

This release brings some meaningful performance improvements to the parsing of complex SQL statements. In files with deeply nested expressions, we have seen up to a 50% reduction on time spent in the parsing phase. These changes are all internal optimisations and have minimal implications for the parser. In a few isolated cases they did highlight inconsistencies in the parsing of literals and so if your use case relies on the specific structure of literal and expression parsing you may find some small differences in how some expressions are parsed.

Additionally this release brings new validation steps to configuration. Layout configuration is now validated on load (and so users with invalid layout configurations may see some of these being caught now) and inline configuration statements in files are also now validated for both their layout rules and for any removed or deprecated settings.

On top of both we've seen dialect improvements to Databricks, PostgreSQL, BigQuery, Snowflake & Athena.

What’s Changed

[2.1.3] - 2023-07-19

Highlights

This release is a fairly standard incremental release. Highlights include bugfixes to RF05 and dialect improvements to Snowflake, Teradata, MySQL, TSQL, SparkSQL & Postgres.

Internally, the last few weeks have brought several improvements to developer tooling.

We've also moved over to GitHub sponsorships - so if you previously used the old flattr link, you can find our new profile page at https://github.com/sponsors/sqlfluff.

What’s Changed

New Contributors

[2.1.2] - 2023-07-03

Highlights

This release resolves compatibility issues with a set of dbt-core versions.

  • dbt-core 1.5.2 onwards is now properly supported.
  • support for dbt-core 1.1 to 1.4 has now been re-enabled after support had to be abandoned a few releases ago.

NOTE: We cannot guarantee that SQLFluff will always continue to remain compatible with all dbt versions, particularly as the folks at dbt-labs have often backported breaking changes to their internal APIs to previous versions of dbt-core. This release does at least bring more extensive internal testing to catch when this does occur to allow our community to react.

This release fixes also resolves a potential security issue for when using external libraries (and the library_path config setting), and also contains various dialect improvements.

What’s Changed

[2.1.1] - 2023-05-25

Highlights

This releases fixes a compatibility issue with the latest version of dbt. It also ships various dialect improvements.

What’s Changed

New Contributors

[2.1.0] - 2023-05-03

Highlights

This release brings support for dbt 1.5+. Some internals of dbt mean that SQFluff versions prior to this release may experience errors with dbt versions post 1.5. In addition to that there are some dialect and templating improvements bundled too:

  • Support for custom Jinja filters.
  • An additional configurable indent behaviour within CASE WHEN clauses.
  • Additional support for bracket quoted literals in TSQL and RF06.
  • Dialect improvements to Snowflake, Hive, Redshift, Postgres, Clickhouse, Oracle and SQLite

What’s Changed

New Contributors

[2.0.7] - 2023-04-20

Highlights

This is a bugfix release to resolve two regressions included in 2.0.6 related to implicit indents. This also includes a bugfix for config file on osx, contributed by first time contributor @jpuris 🎉.

What’s Changed

New Contributors

[2.0.6] - 2023-04-19

Highlights

  • Introduction of a --quiet option for the CLI for situations where less output is useful.
  • When using the --force option is used for sqlfluff fix each file is fixed during the linting process rather than at the end.
  • Bugfixes to comment and templated section indentation.
  • Performance improvements to parsing.
  • Bugfix to macros triggering LT01.
  • Renaming layout.end-of-file to layout.end_of_file in line with other rules.
  • Dialect improvements to SparkSQL, BigQuery, Hive & Snowflake.

What’s Changed

[2.0.5] - 2023-04-14

Highlights

This is a relatively swift bugfix to refine some of the changes made to widow function indentation in 2.0.4. In addition there are two dialect refinements also made since that release.

What’s Changed

[2.0.4] - 2023-04-14

Highlights

This is primarily a bugfix and dialect release:

  • Several bugfixes related to templating and indentation, in particular some improvements to the indentation of aliases and window functions.
  • Performance improvements to the parser.
  • The --persist-timing option is now also available on sqlfluff fix.
  • A refresh to getting started and rule documentation.
  • Dialect improvements to PostgreSQL, Athena, SparkSQL, MySQL & Snowflake.

Thanks also to @james-johnston-thumbtack and @Thashin who made their first contributions in this release. In particular, @james-johnston-thumbtack made twenty one contributions in their first month! 🎉🎉🎉

What’s Changed

New Contributors

[2.0.3] - 2023-04-05

Highlights

This is primarily a bugfix and dialect release:

  • Several bugfixes related to templating and indentation.
  • Configurable indentation before THEN in CASE statements (see #4598).
  • Performance improvements to TypedParser, LT03 & LT04.
  • Rule timings now appear in the --persist-timing option for deeper performance understanding.
  • The introduction of a Greenplum dialect.
  • Dialect improvements to TSQL, Athena, Snowflake, MySQL, SparkSQL BigQuery, Databricks, Clickhouse & Postgres.

We also saw a huge number of first time contributors with 9 contributing in this release 🎉🏆🎉.

What’s Changed

New Contributors

[2.0.2] - 2023-03-23

Highlights

This is primarily a bugfix release. Most notably this solves some of the issues introduced in 2.0.1 around spacing within datatypes. Expressions like 1.0::double precision should now be spaced correctly.

Beyond that, this contains a selection of smaller bugfixes and dialect improvements. Even for a relatively small release we saw three new contributors (thanks @aurany, @JackWolverson & @mikaeltw 🎉).

The one new feature (as such) is being able to now configure LT05 (aka layout.long_lines) to optionally move trailing comments after the line they are found on, rather than the default behaviour of moving them up and before. Users can enable this with the trailing_comments configuration setting in the indentation section.

This release also contains some performance optimisations in the parser, especially on queries with heavily nested expressions. There will be more to come in this space, but we hope this leads to a better experience for many users. 🚀

What’s Changed

New Contributors

[2.0.1] - 2023-03-17

Highlights

This is mostly a bugfix release addressing some of the issues from the recent 2.0 release. Notable fixes are:

  • Spacing for (as applied by LT01) for datatypes, hyphenated identifiers and casting operators.
  • Several bugs in the indentation routines (LT02), in particular with implicit indents.
  • Fixing a conflict between LT09 and LT02, by only limiting LT09 to bringing targets onto a single line if there is only one select target and that it contains no newlines.
  • Supporting arrays, and the new rules configuration more effectively in pyproject.toml.
  • Configuring dialects on a file by file basis using inline comments now works.

This release also brings one small new feature in allowing additional flags to be passed to SQLFluff when called as a pre-commit hook.

Thanks especially to @JavierMonton and @LauraRichter who made their first contributions to the project as part of this release! 🎉🏆

What’s Changed

New Contributors

[2.0.0] - 2023-03-13

Highlights

Upgrading to 2.0 brings several important breaking changes:

  • All bundled rules have been recoded, both from generic L00X formats into groups within similar codes (e.g. an aliasing group with codes of the format AL0X), but also given names to allow much clearer referencing (e.g. aliasing.column).
  • Configuring rules now uses the rule name rather than the rule code to specify the section. Any unrecognised references in config files (whether they are references which do match existing rules by code or alias, or whether the match no rules at all) will raise warnings at runtime.
  • A complete re-write of layout and whitespace handling rules (see layout), and with that a change in how layout is configured (see configuring layout) and the combination of some rules that were previously separate. One example of this is that the legacy rules L001, L005, L006, L008, L023, L024, L039, L048 & L071 have been combined simply into LT01.
  • Dropping support for dbt versions before 1.1.

To help users upgrade to 2.0, we've put together a recommended process as part of our release notes.

Beyond the breaking changes, this release brings a load of additional changes:

  • Introduces the the sqlfluff format CLI command (a la sqlfmt or black) to auto-format sql files using a known set of fairly safe rules.
  • Databricks as a distinct new dialect (rather than as previously an alias for sparksql).
  • Performance improvements in our parsing engine.
  • Dialect improvements to almost all of them.

As a new major release, especially with significant rewrites of large portions of the codebase, we recommend using compatible release specifiers in your dependencies (i.e. sqlfluff~=2.0.0) so that you can automatically take advantage of any bugfix releases in the coming weeks. The alpha releases of 2.0.0 have been tested on a range of large projects, but we know that the range of use cases "in the wild" is very diverse. If you do experience issues, please post them on GitHub in the usual manner.

Finally thanks to everyone who has worked on this release, especially @konnectr, @ValentinCrr, @FabianScheidt, @dflem97, @timcosta, @AidanHarveyNelson, @joar, @jmpfar, @jared-rimmer, @vesatoivonen, @briankravec, @saintamh, @tdurieux, @baa-ableton, & @WillAyd who made their first contributions during the development of 2.0.0. Thanks for your contributions, and especially your patience in the slightly slower release of your efforts into the wild. 🙏🎉

What’s Changed

New Contributors

[2.0.0a6] - 2023-03-06

NOTE: This is effectively a release candidate for testing purposes. There are several new features here, and breaking changes to configuration. We welcome testing feedback from the community, and the intent is that following this release there will be no more major breaking changes in the before the 2.0.0 release.

Highlights

This is the sixth alpha release for 2.0.0, and effectively the first release candidate for 2.0.0. All the intended breaking changes for the upcoming release have now been made and only bugfixes and non breaking feature changes should happen between this release and the full release.

It contains:

  • A reorganisation of rules. All rules have been recoded, and can now be referred to by their name, code, alias or group. The legacy code for the rule is included as an alias for each rule to support some backward compatibility.
  • Configuration files (and inline configuration flags), should now use the name of the rule rather than the code. Any configuration files which reference using legacy rules (or reference unknown rules) should now display warnings.
  • Introduces the the sqlfluff format CLI command (a la sqlfmt or black) to auto-format sql files using a known set of fairly safe rules.
  • Databricks as a distinct new dialect (rather than as previously an alias for sparksql).

There are also numerous dialect improvements to ANSI, Athena, TSQL, Teradata, SQLite & MySQL.

What’s Changed

New Contributors

[2.0.0a5] - 2023-02-24

NOTE: This is an alpha release for testing purposes. There are several new features here, and breaking changes to configuration. We welcome testing feedback from the community, but know that this release may feel less polished than usual.

Highlights

This is the fifth alpha release for 2.0.0. It contains:

  • Significant rework to rule naming and categorisation.
  • Several performance improvements.
  • Many dialect improvements to several dialects.
  • Bugfixes to many of the issues raised in 2.0.0a4.

There will likely be more changes to rule classification before a full release of 2.0.0, so anticipate that configuration files may change slightly again in future alpha releases.

What’s Changed

New Contributors

[2.0.0a4] - 2023-01-26

Highlights

This is the fourth alpha release for 2.0.0. It contains a fix for the renamed dbt exceptions in dbt version 1.4.0, a fix for a major performance issue with the 2.0 dbt templater, and improvements to parse performance of large SQL files.

What’s Changed

[2.0.0a3] - 2023-01-16

NOTE: This is an alpha release for testing purposes. There are several new features here, and breaking changes to configuration. We welcome testing feedback from the community, but know that this release may feel less polished than usual.

Highlights

This is the third alpha release for 2.0.0. It contains primarily bugfixes from 2.0.0a2 to allow continued testing. In particular, some of the recent 2.0.0-related changes to the dbt templater have been reverted, primarily due to performance and other issues. If those issues can be resolved, the changes will be re-introduced. The long-term goal of this work is to ease maintenance of the templater by separating dbt integration concerns from SQLFluff concerns.

There will likely be more changes to rule classification before a full release of 2.0.0, so anticipate that configuration files may change slightly again in future alpha releases.

What’s Changed

New Contributors

[2.0.0a2] - 2023-01-07

Highlights

This is the second alpha release for 2.0.0. It contains primarily bugfixes from 2.0.0a1 to allow continued testing along with dialect improvements for Snowflake, Postgres and DB2.

What’s Changed

[2.0.0a1] - 2022-12-28

Highlights

This is the first alpha version for 2.0.0. It brings all of the changes to whitespace handing, including a total rewrite of indentation and long line logic (L003 & L016). That brings several breaking changes to the configuration of layout, see the layout docs for more details and familiarise yourself with the new default configuration.

In addition, for the dbt templater, this introduces a large re-write of the codebase, dropping support for dbt versions before 1.0.0. This leverages functionality from dbt-osmosis to reduce the amount of functionality supported directly by SQLFluff, and performance during testing of the new version has been reported as significantly faster.

What’s Changed

New Contributors

[1.4.5] - 2022-12-18

Highlights

This is a bugfix release, primarily for diff-quality. In addition, a new rules for spacing around parenthesis is also included.

This is also the final 1.x.x release. Following releases will be a series of alpha releases for 2.x.x. If you affected by any outstanding bugs or regressions from this release, consider either rolling backward to a previous release without those issues, or failing forward to on an alpha release for 2.x.x (or a full release if that's out). Note that 2.x.x will bring a selection of breaking changes to config file structure, rule categorisation and dbt support.

What’s Changed

New Contributors

[1.4.4] - 2022-12-14

Highlights

Bug fix for 1.4.3 which was incorrectly flagging L006 for concat operators (||) and other two-symbol binary operators.

What’s Changed

[1.4.3] - 2022-12-13

Highlights

  • Rewrote diff-quality plugin to run SQLFluff as a subprocess. More reliable, easier to switch between diff-quality and running sqlfluff lint directly.
  • New rule L067 enforces consistent syntax for type casts.
  • New rule L068 enforces a consistent number of columns in set queries (e.g. UNION).
  • Initial support for Materialize dialect.

What's Changed

New Contributors

[1.4.2] - 2022-11-13

Highlights

This release is less about internals and much more about some quality of life improvements and dialect changes. The most notable are:

  • The introduction of a sqlfluff render command to preview the results of templated sql.
  • Linting errors within templated loops should now only appear once in the linting output.
  • Indentation around jinja {% set %} statements should now be more consistent.
  • Linting errors around unparsable code are now more appropriately handled (with more to come soon on that front).
  • Error messages when specified files aren't found are now more specific.

We've also got dialect improvements for Redshift, SOQL & SparkSQL.

What’s Changed

New Contributors

[1.4.1] - 2022-10-31

Highlights

This is a fix to the configuration migration from 1.4.0. In that release, the configuration of leading/trailing operators would be migrated the wrong way around and precedence between new and old configuration values behaved unexpectedly.

What’s Changed

[1.4.0] - 2022-10-31

Highlights

This release brings several internal changes, and acts as a prelude to 2.0.0 which will be released fairly soon. In particular, the following config values have changed:

  • sqlfluff:rules:L007:operator_new_lines has been changed to sqlfluff:layout:type:binary_operator:line_position.
  • sqlfluff:rules:comma_style and sqlfluff:rules:L019:comma_style have both been consolidated into sqlfluff:layout:type:comma:line_position.

If any of these values have been set in your config, they will be automatically translated to the new values at runtime, and a warning will be shown. To silence the warning, update your config file to the new values. For more details on configuring layout (including some changes yet to come in future versions) see the docs.

These changes are driven by underlying centralisation in the routines which control layout. While for this release, no breaking changes are expected - you may find slight differences in how SQLFluff handles edge cases. We believe in the majority of cases these are more consistent, but if you find any which are problematic then do post an issue on GitHub as usual.

Other highlights from this release:

  • Better dbt supportfor graph nodes and avoiding dependency conflicts.
  • Numerous dialect improvements to T-SQL, MySQL, SparkSQL, SQLite, Athena Snowflake, Hive, Postgres & Databricks.

There have also been first time contributions from 10 new contributors! 🎉🎉🎉

What’s Changed

New Contributors

[1.3.2] - 2022-09-27

Highlights

This is primarily a release for dialect fixes and improvements with additions and changes to TSQL, Snowflake, MySQL & Redshift. The other changes of note are:

  1. Support for warnings when users set old removed config values. This supports future change work by allowing a mechanism to warn if they are used.
  2. Improvements to the fix routines for L014 and L042 to handle some trickier cases.

What’s Changed

New Contributors

[1.3.1] - 2022-09-09

Highlights

  • More refactoring of parse structures in preparation for upcoming refactor of formatting/whitespace rules.
  • Fixes some bugs in L003 (indentation).
  • New config flag large_file_skip_byte_limit which applies prior to loading the file.

What’s Changed

New Contributors

[1.3.0] - 2022-08-21

Highlights

This release brings several potentially breaking changes to the underlying parse tree. For users of the cli tool in a linting context you should notice no change. If however your application relies on the structure of the SQLFluff parse tree or the naming of certain elements within the yaml format, then this may not be a drop-in replacement. Specifically:

  • The addition of a new end_of_file meta segment at the end of the parse structure.
  • The addition of a template_loop meta segment to signify a jump backward in the source file within a loop structure (e.g. a jinja for loop).
  • Much more specific types on some raw segments, in particular identifier and literal type segments will now appear in the parse tree with their more specific type (which used to be called name) e.g. naked_identifier, quoted_identifier, numeric_literal etc...

If using the python api, the parent type (such as identifier) will still register if you call .is_type("identifier"), as this function checks all inherited types. However the eventual type returned by .get_type() will now be (in most cases) what used to be accessible at .name. The name attribute will be deprecated in a future release.

Other highlights:

  • New command-line option --show-lint-violations to show details on unfixable errors when running sqlfluff fix.
  • Improved consistency of process exit codes.
  • Short CLI options for many common options.
  • Jinja templater: When --ignore=templating is enabled, undefined Jinja variables now take on "reasonable" default values rather than blank string (""). This can streamline initial rollout of SQLFluff by reducing or eliminating the need to configure templater variables.

There are also a ton of other features and bug fixes in this release, including first-time contributions from 11 new contributors! 🎉

What’s Changed

New Contributors

[1.2.1] - 2022-07-15

Highlights

This is primarily a bugfix release to resolve an issue with the 1.2.0 release where the new version of sqlfluff-templater-dbt relied on functionality from the new version of sqlfluff but the package configuration had not been updated. Versions of the two packages are now pinned together.

What’s Changed

[1.2.0] - 2022-07-13

Highlights

Major changes include:

  • Adding AWS Athena as a dialect.
  • A fix routine for L046 (whitespace in jinja tags), and the mechanisms for more source-only fixes in future.
  • By default, large files (over 20k characters) are now skipped by sqlfluff. This limit is configurable and disable-able but exists as a sensible default to avoid the performance overhead of linting very large files.
  • For the dbt templater, fatal compilation errors now no longer stop linting, and these files are now skipped instead. This enables projects to continue linting beyond the offending file and much better logging information to enable better debugging.

What’s Changed

New Contributors

[1.1.0] - 2022-07-03

Highlights

Major changes include:

  • L066 - New rule to allow you to set min/max length requirements for aliases to ensure they are meaningful
  • L062 - addition of blocked_regex as well as blocked_words
  • L025 - fix several corner cases where aliases were removed inappropriately
  • L059 is now disabled by default for Postgres
  • Many more dialect improvements and bug fixes.

Highlights

What’s Changed

New Contributors

[1.0.0] - 2022-06-17

Highlights

This is the first stable release of SQLFluff 🎉🎉🎉.

  • Does this mean there are no more bugs? No.
  • Does this mean we're going to stop developing new features? No.
  • Does this mean that this is a tool that is now broadly usable for many teams? Yes.

We've intentionally chosen to release 1.0.0 at a time of relative stability within SQLFluff and not at a time when new big structural changes are being made. This means that there's a good chance that this release is broadly usable. This also recognises that through the hard work of a huge number of contributors that we've built out this from a fringe tool, to something which gets over 500k downloads a month and over 4k stars on Github.

There's still a lot to do, and some more exciting things on the horizon. If you want to be part of this and join the team of contributors, come and hang out in our slack community or on our twitter account where people can help you get started. If you're a long time user, keep submitting bug reports and inputting on issues on Github.

If you've never used SQLFluff before, or are hesitant about starting to use it in your day to day work, now might be a good time to try it. We have guides on how to get started with the tool, and how to get started with rolling out to a team in our docs.

What’s Changed

New Contributors

[0.13.2] - 2022-05-20

Highlights

Major changes include:

  • Fix bug causing L003 to report indentation errors for templated code - sorry we know that one's caused many of you some grief :-(
  • Initial support of SOQL (Salesforce Object Query Language).
  • Additional Placeholder templating options.
  • Start of BigQuery procedural language support (starting simple FOR statements and CREATE PROCEDURE statements).
  • New rule L065 to put set operators onto new lines.
  • Many more dialect improvements and bug fixes.

What’s Changed

New Contributors

[0.13.1] - 2022-05-06

Highlights

Major changes include:

  • Addition of "rule groups" (currently core and all) to allow ease of turning on and off groups of rules.
  • Addition of db2 dialect
  • PRS errors are now highlighted in red.
  • Many bugs fixes and dialect improvements

What’s Changed

New Contributors

[0.13.0] - 2022-04-22

Highlights

Major changes include:

  • New Rule (L064) for preferred quotes for quoted literals
  • Rule speed improvements and fixing performance regression from 0.12.0
  • Add configuration option to disallow hanging indents in L003
  • Add ignore_words_regex configuration option for rules
  • New GitHub Annotations option
  • Many bug fixes and dialect improvements

What’s Changed

New Contributors

[0.12.0] - 2022-04-07

Highlights

Major changes include:

  • Dialect is now mandatory, either in command line, or in config BREAKING CHANGE
  • Rename spark3 dialect to sparksql BREAKING CHANGE
  • L027 now checks tables references exist BREAKING CHANGE
  • New rule L063 to allow Datatypes to have a different capitalisation policy from L010. BREAKING CHANGE
  • Refactor and performance improvements of Delimited and L003
  • Many dialect improvements and fixes

What’s Changed

New Contributors

[0.11.2] - 2022-03-25

What’s Changed

[0.11.1] - 2022-03-17

Highlights

Major changes include:

  • A number of changes to fix code to make these more robust
  • Improvements to templating blocks
  • generate_parse_fixture_yml options to allow quicker, partial regeneration of YML files
  • Numerous rule fixes including adding auto fix to L042
  • Numerous grammar changes

What’s Changed

New Contributors

[0.11.0] - 2022-03-07

Highlights

Major changes include:

  • Changes rule L030 to use extended_capitalisation_policy to support PascalCase BREAKING CHANGE
  • Fixes dbt error on ephemeral models
  • Log warnings for fixes that seem to corrupt the parse SQL as may cause incorrect fixes in other rules.
  • Bug fix to rule L011 for implicit aliases
  • Bug fix to rule L019 for commas besides templated code
  • Rule L051 can now optionally be applied to LEFT/RIGHT/OUTER JOINs
  • Improvements to Test Suite
  • Many dialect improvements

What’s Changed

New Contributors

[0.10.1] - 2022-02-15

Highlights

Major changes include:

  • Improvements to rules L023, L045, L048, L052, L059 to make them more accurate.
  • If sqlfluff fix cannot find a stable fix after runaway_limit iterations (default 10) then no fixes will be applied.
  • Addition of --write-output config to command line so prevent errors corrupting output.
  • Various dialect improvements

What’s Changed

[0.10.0] - 2022-02-10

Highlights

Major changes include:

  • Dropping support of DBT < 0.20 BREAKING CHANGE
  • sqlfluff fix no will no longer fix SQL containing parsing or templating errors BREAKING CHANGE
  • New rule L062 to allow blocking of list of configurable words (e.g. syntax, or schemas, or tables you do not want people to use)
  • Lots and lots of docs improvements
  • Looser requirements for click python package

What’s Changed

New Contributors

[0.9.4] - 2022-01-30

Highlights

Major changes include:

  • dbt performance improvements
  • Fix click dependency error.
  • Better datepart versus identifier parsing.
  • Fix some Jinja errors.
  • Various grammar fixes and improvements

What’s Changed

  • Spark3: test cases for HAVING clause in SELECT statement #2518 @R7L208
  • Update click version requirement in setup.cfg to match that in requirements.txt #2518 @barrywhart
  • Postgres: Implement DO Statements + Refactored Language Clause #2511 @PLBMR
  • Spark3: Support for Grouping Sets, CUBE and ROLLUP in GROUP BY clause of SELECT statement #2505 @R7L208
  • Refactor date part functions #2510 @tunetheweb
  • Postgres: EXPLAIN ANALYSE allows British spelling #2507 @jpy-git
  • "noqa": Add support for ignoring template (TMP) and parse (PRS) errors #2509 @barrywhart
  • Freeze Black due to incompatibility between 22.1 and flake8-black #2513 @tunetheweb
  • Support NATURAL JOINS #2506 @tunetheweb
  • dbt Docker environment: Mount the test profiles.yml at ~/.dbt #2502 @barrywhart
  • Add dbt_artifacts package to in the wild docs #2504 @NiallRees
  • Spark3: Support DISTRIBUTE BY clause in SELECT statement #2503 @R7L208
  • dbt templater: For performance reasons, cache the database connection across models #2498 @barrywhart
  • Bug fix: Defining and using Jinja macro in the same file causes runtime error #2499 @barrywhart
  • Spark3: Support CLUSTER BY clause in SELECT statement #2491 @R7L208
  • Grammar: Adds support for COPY statement for Postgres dialect #2481 @derickl
  • Add raiserror for T-SQL #2490 @fdw
  • Enforce parentheses for function definitions in T-SQL #2489 @fdw
  • Add guards to prevent rule crashes #2488 @barrywhart

New Contributors

[0.9.3] - 2022-01-26

Highlights

Major changes include:

  • Add ignore_words option for rules L010, L014, L029, L030, L040
  • Fix some issues in 0.9.2 preventing some queries linting

What’s Changed

[0.9.2] - 2022-01-24

Highlights

We are pleased to include 110 improvements and fixes in this release, and welcome 7 new contributors to the code.

Major changes include:

  • Initial Oracle support (note: SQL, but not PL/SQL)
  • Fix more dbt 1.0.0 connection issues
  • Improved configuration documentation
  • New rule (L059) to flag unnecessary quoted identifiers
  • New rule (L060) to prefer COALESCE instead of IFNULL or NVL
  • New rule (L061) to prefer != over <>
  • Many rule fixes
  • Many dialect improvements

What’s Changed

New Contributors

[0.9.1] - 2022-01-08

Highlights

  • Fix dbt 1.0.0 connection issue
  • Fix some SQL corruption issues with templated code
  • New components to simplify creating rules
  • Remove support for Python 3.6

What’s Changed

New Contributors

[0.9.0] - 2021-12-13

What’s Changed

This release brings about several great new additions including:

  • dbt 1.0.0 compatibility.
  • CLI and Simple API parameters to provide custom paths to config files.
  • Refinement to Simple API to return parse output in JSON format rather than as an internal SQLFluff object (BREAKING CHANGE).
  • An Official SQLFluff Docker Image.
  • Grammar improvements across various dialects.
  • A new rule (L057) to check for non-alphanumeric values in identifiers.

There have also been many bug fixes and improvements to the CI and development processes.

🚀 Enhancements

🐛 Bug Fixes

[0.8.2] - 2021-11-22

What’s Changed

One of the biggest new features in this release is the support for SQLAlchemy and other "placeholder" templating within SQL queries. Check out the documentation on how to set it up.

This release also adds seven new rules. Get some help with your leading whitespace, semi-colon placement, inconsistent column references in GROUP BY/ORDER BY, and getting rid of RIGHT JOIN's among other useful lints with our new rules! See our rules documentation for more details.

On top of those, we have made loads of grammar improvements across many dialects, improvements to the dbt templater (including issues where sqlfluff fix would corrupt the code 😱), more fix routines, and lots more improvements.

🚀 Enhancements

🐛 Bug Fixes

[0.8.1] - 2021-11-07

What’s Changed

Fixes missing dependency issue with 0.8.0 for tqdm, plus add a test to ensure this does not happen again.

🐛 Bug Fixes

[0.8.0] - 2021-11-07

What’s Changed

This release brings an improvement to the performance of the parser, a rebuild of the Jinja Templater, and a progress bar for the CLI. Lots of dialect improvements have also been done. Full list of changes below:

🚀 Enhancements

🐛 Bug Fixes

[0.7.1] - 2021-10-22

What’s Changed

Highlights of this release contains a lot of T-SQL dialect improvements (shout out to @jpers36 for most of these!). We also added Spark3 as a new dialect thanks to @R7L208. The complete list of changes are shown below.

🚀 Enhancements

🐛 Bug Fixes

[0.7.0] - 2021-10-14

BREAKING CHANGE

This release extracts the dbt templater to a separately installable plugin sqlfluff-templater-dbt. For users who take advantage of the dbt templater see the updated docs on how to migrate. It also adds the redshift dialect and removes the exasol_fs dialect which has been merged into the exasol dialect.

What’s Changed

🚀 Enhancements

🐛 Bug Fixes

[0.6.9] - 2021-10-08

Another dbt bugfix from 0.6.7 and 0.6.8, plus a host of dialect and syntax improvements.

🚀 Enhancements

🐛 Bug Fixes

[0.6.8] - 2021-10-05

Fixed a DBT bug introduced in 0.6.7 - apologies!

What’s Changed

SQLFluff can't find dbt models #1513 @barrywhart T-SQL: Support for unicode literals #1511 @adam-tokarski

[0.6.7] - 2021-10-04

Lots of fixes to our rules (particularly when running sqlfluff fix, and particularly for Jinja and DBT templates). We also have good improvements to Exasol, Snowflake, and T-SQL dialects amongst others. Plus we added Hive and SQLite as supported dialects!

What’s Changed

[0.6.6] - 2021-09-20

Fixed some of our autofix rules where running fix sometimes made unintended changes. Added config to rules L011 and L012 to allow preferring implicit aliasing. Also further improved our Postgres support and documentation.

What’s Changed

[0.6.5] - 2021-09-10

What’s Changed

This release includes initial support of Transact-SQL (T-SQL), much better Postgres and Snowflake support, improvements to our documentation, 100% coverage for Python code (with a small number of accepted exceptions), along with numerous other bug fixes and improvements.

Many thanks to all the contributors helping to improve SQLFluff!

Complete list of changes

[0.6.4] - 2021-08-20

Added

Changed

[0.6.3] - 2021-08-16

Added

  • Support for primary index name, collect stats improvement, COMMENT statement for teradata dialect #1232
  • Support config for L007 to prefer end of line operators #1261
  • Support for DETERMINISTIC user defined functions in BigQuery dialect #1251
  • Support more identifiers in BigQuery dialect #1253
  • Support function member field references in BigQuery dialect #1255
  • Support alternative indentation for USING and ON clauses #1250
  • Support COUNT(0) preference over COUNT(*) or COUNT(1) #1260
  • Support for BigQuery "CREATE table OPTIONS ( description = 'desc' )" #1205
  • Support wildcard member field references in BigQuery dialect #1269
  • Support ARRAYS of STRUCTs in BigQuery dialect #1271
  • Support fields of field references in BigQuery dialect #1276
  • Support OFFSET and ORDINAL clauses of Array Functions in BigQuery dialect #1171
  • Added check for generated YML files #1277
  • Support QUALIFY to BigQuery dialect #1242

Changed

  • Fix comma removed by L019 #939
  • Update L019 (leading/trailng comma rule) so it doesn't run on unparsable code.
  • The --nocolor command-line option should suppress emoji output #1246
  • Added HTTP Archive to the In The Wild page

[0.6.2] - 2021-07-22

Added

  • Support for looping statements (loop, while, repeat) and supporting statements to mysql dialect #1180

Changed

  • Added dbt 0.20.* to the default test suite.
  • Updated manifest loading in dbt 0.20.* to use the new ManifestLoader #1220
  • Handle newlines in rule list configuration in .sqlfluff #1215
  • Fix looping interaction between L008 and L030 #1207

[0.6.1] - 2021-07-16

Added

  • Linting output now supports GitHub Actions #1190
  • Support for QUALIFY syntax specific to teradata dialect #1184
  • Support for TRUNCATE statement #1194
  • Support for prepared statement syntaxes specific to mysql dialect #1147
  • Support for GET DIAGNOSTICS statement syntax specific to mysql dialect #1148
  • Support for cursor syntax specific to mysql dialect #1145
  • Support sequential shorthand casts #1178
  • Support for select statement syntax specific to mysql dialect #1175
  • Support for the CALL statement for the mysql dialect #1144
  • Support for OVERLAPS predicate #1091
  • Support for the CREATE/DROP PROCEDURE statement for the mysql dialect #901
  • Specific allowed/required syntaxes for CREATE/DROP FUNCTION within the mysql dialect #901
  • Support for DECLARE statement for the mysql dialect #1140
  • Support for the IF-THEN-ELSEIF-ELSE syntax for the mysql dialect #1140
  • Support for the DEFINER syntax for the mysql dialect #1131
  • Preserve existing file encoding in the "fix" command. Partially addresses #654
  • Support for DECLARE and SET variable syntax for the BigQuery dialect #1127
  • Support for ALTER TASK statement on Snowflake #1211

Changed

  • Fix runtime error in diff-cover plugin caused by new diff-cover release 6.1.0 #1195
  • Resolved an issue with the snowflake dialect where backslash escaped single quoted strings led to fatal lexing errors #1200

Contributors

[0.6.0] - 2021-06-06

Added

  • Respect XDG base dirs on Mac OS (#889).
  • Added support for additional delimiters by creating a new DelimiterSegment in the ANSI dialect which defaults to the semicolon, but allows it to be more intuitive when overridden in a specific child dialect (mysql) #901)
  • Added support for the DELIMITER statement in the mysql dialect #901)
  • Added support for additional delimiters by creating a new DelimiterSegment in the ANSI dialect which defaults to the semicolon, but allows it to be more intuitive when overridden in a specific child dialect (mysql) #901)
  • Added support for function as a default column value #849.
  • Add an --include-metas option for parse output to show the meta segments in the parse tree.
  • Allow CREATE FUNCTION syntax without arguments @bolajiwahab #1063.
  • Added support for the CREATE/DROP PROCEDURE statement for the mysql dialect #901)
  • Added specific allowed/required syntaxes for CREATE/DROP FUNCTION within the mysql dialect #901)
  • Now possible to run sqlfluff commands outside the project root when using the dbt templater.

Changed

  • Renamed --parallel CLI argument to --processes to be more accurate.
  • L034 now ignores select statements which contain macros.
  • L034 now ignores select statements part of a set expression, most commonly a union.
  • Fix bug #1082, adding support for BigQuery select as struct '1' as bb, 2 as aa syntax.
  • Rationalisation of the placement of meta segments within templated queries to support more reliable indentation. This includes prioritising longer invariant sections first and then dropping any shorter ones which then are relatively out of place.
  • Additional logging within the lexer and templater engines.
  • Allow meta segments to parse within Delimited grammars which otherwise don't allow gaps. This is facilitated through an optional argument to trim_non_code_segments.
  • Fix bug #1079, addressing issues with L025 and L026 with BigQuery column references involving STRUCT.
  • #1080 Add SET SCHEMA and DROP SCHEMA support to ANSI dialect.

Contributors

[0.6.0a2] - 2021-05-27

Changed

  • Better exception handling for the simple parsing API (sqlfluff.parse) which now raises an exception which holds all potential parsing issues and prints nicely with more than one issue.
  • Fix bug #1037, in which fix logging had been sent to stdout when reading data from stdin.
  • Add a little bit of fun on CLI exit 🎉!
  • Disabled models in the dbt templater are now skipped entirely rather than returning an untemplated file.
  • Add a changelog check to SQLFluff continuous integration.
  • Fix bug #1083, adding support for BigQuery named function arguments, used with functions such as ST_GEOGFROMGEOJSON()
  • Update documentation links to sqlfluff-online.

[0.6.0a1] - 2021-05-15

Added

  • Lint and fix parallelism using --parallel CLI argument
  • Fix 1051, adding support for bitwise operators &, |, ^, <<, >>

[0.5.6] - 2021-05-14

  • Bugfix release for an issue in L016 introduced in 0.5.4.
  • Fix for L016 issue where DISTINCT keywords were mangled during fixing #1024.

[0.5.5] - 2021-05-13

  • Bugfix release for an off-by-one error introduced in L016 as part of 0.5.4.

[0.5.4] - 2021-05-12

Added

  • Parsing of Postgres dollar quoted literals.
  • Parsing of Postgres filter grammar.
  • Parsing of "ALTER DEFAULT PRIVILEGES" Postgres statement.
  • Parsing of Postgres non-explicit role granting and function execution.
  • Early failing on fatal dbt templater fails.

Changed

  • Big rewrite of the lexer, segments and position markers for simplicity and to support future parallelism work.
  • Fix to L036 which previously mangled whitespace.

[0.5.3] - 2021-05-04

Added

  • L009 can now be enforced when templater = dbt.
  • Parsing of EXPLAIN, USE statements.
  • Parsing of ALTER TABLE x RENAME TO y syntax.
  • Parsing of ALTER SESSION in snowflake.
  • Parsing of numeric literals with exponents.
  • Added rule codes to diff_cover output.

Changed

  • Fix templater = dbt L009 bug #861 where:
    • sqlfluff lint would incorrectly always return L009 | Files must end with a trailing newline.
    • sqlfluff fix would remove trailing newlines when exclude_rules = L009.
  • Fix bug with BigQuery comparison operators.
  • Fix recursion bug with L045.
  • Fix tuple index bug with L016.
  • Fix mange coalecse bug with L043.
  • Fix Jinja templating error with UnboundLocalError.
  • Improve array parsing.
  • Simplify bracket parsing.
  • Speed up L010 with caching capitalisation policy.
  • Output of sqlfluff dialects is now sorted.
  • Handle disabled dbt models.

[0.5.2] - 2021-04-11

Changed

  • Fix false positive in L045 when CTE used in WHERE clause (#944)
  • Logging and readout now includes more detail and a notification of dbt compilation.
  • Fix bug in L048 which flagged adjoining commas as failures.
  • Fix bug in L019 with inline comments.
  • Fix bug in L036 with multiple newlines.
  • Skip disabled dbt models. (#931).
  • Support "USE" statement in ANSI (#902).
  • Parse explain statement (#893).

[0.5.1] - 2021-04-09

Changed

  • Parsing improvements around optional brackets.
  • Better parsing of set operators (like UNION) and how they interact with ORDER BY clauses.
  • Support for comparison operators like ~.
  • Fix parsing of snowflake SAMPLE syntax.
  • Fix recursion issues in L044.
  • SPACE keyword now has no special meaning in the postgres dialect.

[0.5.0] - 2021-04-05

Added

  • pascal (PascalCase) capitalisation_policy option for L014 (unquoted identifiers)
  • only_aliases configuration option for L014 (unquoted identifiers)
  • Dialects now have more advanced dependency options to allow less repetition between related dialects. The methods get_segment and get_grammar can be used on unexpanded grammars to access elements of the parent grammars. The copy method on grammars can be used to copy with alterations.
  • Rule L046 to line whitespace within jinja tags.
  • Enable and Disable syntax for ignoring violations from ranges of lines.

Changed

  • Renamed the BaseCrawler class to BaseRule. This is the base class for all rules. This is a breaking change for any custom rules that have been added via plugins or by forking the SQLFluff repo.
  • Renamed sqlfluff.rules() to sqlfluff.list_rules() and sqlfluff.dialects() to sqlfluff.list_dialects() due to naming conflicts with the now separate sqlfluff.dialects module.
  • Extracted dialect definitions from the sqlfluff.core module so that each dialect is better isolated from each other. This also allows more focused testing and the potential for dialect plugins in future. Dialects are now only imported as needed at runtime. All dialects should now be accessed using the selector methods in sqlfluff.core.dialects rather than importing from sqlfluff.dialects directly.
  • Add support for ALTER USER commands in Snowflake dialect.
  • Added describe statement to ANSI dialect
  • Renamed capitalisation_policy to extended_capitalisation_policy for L014 to reflect the fact that it now accepts more options (pascal) than regular capitalisation_policy still used by L010 and others.
  • Replaced only_aliases config with unquoted_identifiers_policy and added it to rule L014 in addition to L029.
  • Parse structure of FROM clauses to better represent nested joins and table functions.
  • Parse structure of expressions to avoid unnecessary nesting and overly recursive method calls.

[0.4.1] - 2021-02-25

Added

  • Initial architecture for rule plugins to allow custom rules. This initial release should be considered beta until the release of 0.5.0.
  • Add tests for dbt 0.19.0.
  • General increased parsing coverage.
  • Added some missing Postgres syntax elements.
  • Added some basic introspection API elements to output what dialects and rules are available for use within the API.

Changed

  • Fix several Snowflake parsing bugs.
  • Refactor from clause to handle flattens after joins.
  • Fix .get_table_references() in Snowflake dialect.
  • Macros defined within the .sqlfluff config will take precedence over the macros defined in the path that is defined with config value sqlfluff:templater:jinja:load_macros_from_path.
  • Fix Snowflake indent parsing.
  • Fixed incorrect parsing of syntax-like elements in comments.
  • Altered parsing of NULL keywords, so parse as Literals where appropriate.
  • Fixed bug in expression parsing leading to recursion errors.

[0.4.0] - 2021-02-14

Added

  • Public API to enable people to import sqlfluff as a python module and call parse, lint and fix within their own projects. See the docs for more information. (#501)
  • The ability to use dbt as a templating engine directly allowing richer and more accurate linting around dbt macros (and packages related to dbt). For more info see the docs. (#508)
  • Support for modulo (%) operator. (#447)
  • A limit in the internal fix routines to catch any infinite loops. (#494)
  • Added the .is_type() method on segments to more intelligently deal with type matching in rules when inheritance is at play.
  • Added the ability for the user to add their own rules when interacting with the Linter directly using user_rules.
  • Added L034 'Fields should be stated before aggregates / window functions' per dbt coding convenventions (#495)
  • Templating tags, such as {{ variables }}, {# comments #} and {% loops %} (in jinja) now have placeholders in the parsed structure. Rule L003 (indentation), also now respects these placeholders so that their indentation is linted accordingly. For loop or block tags, they also generate an Indent and Dedent tag accordingly (which can be enabled or disabled) with a configuration value so that indentation around these functions can be linted accordingly. (#541)
  • MyPy type linting into a large proportion of the core library. (#526, #580)
  • Config values specific to a file can now be defined using a comment line starting with -- sqlfluff:. (#541)
  • Added documentation for --noqa: use in rules. (#552)
  • Added pre-commit hooks for lint and fix. (#576)
  • Added a fix routine for Rule L019 (comma placement). (#575)
  • Added Rule L031 to enforce "avoid using alias in the FROM/JOIN clauses" from the dbt coding conventions. (#473, #479)
  • Added Rule L032 to enforce "do not use USING" from the dbt coding conventions. (#487)
  • Added Rule L033 to enforce "prefer UNION ALL to UNION *" from the dbt coding conventions. (#489)
  • Added Rule L034 to enforce "fields should be stated before aggregate/window functions" from the dbt coding conventions. (#495)
  • Added Rule L038 to forbid (or require) trailing commas in select clauses. (#362)
  • Added Rule L039 to lint unnecessary whitespace between elements. (#502)
  • Added a fix routine for L015. (#732)
  • Added a fix routine for L025. (#404)
  • Adopted the black coding style. (#485)
  • Added validation and documentation for rule configuration options. (#462)
  • Added documentation for which rules are fixable. (#594)
  • Added EPOCH keyword for postgres dialect. (#522)
  • Added column index identifier in snowflake dialect. (#458)
  • Added USE statement to the snowflake dialect. (#537)
  • Added CODE_OF_CONDUCT to the project. (#471)
  • Added ISNULL and NOTNULL keywords to ansi dialect. (#441)
  • Added support for python 3.9. (#482)
  • Added requirements_dev.txt for local testing/linting. (#500)
  • Added CLI option --disregard-sqlfluffignores to allow direct linting of files in the .sqlfluffignore. (#486)
  • Added dbt incremental macro. (#363)
  • Added links to cockroachlabs expression grammars in ansi dialect. (#592)
  • Added favicon to the docs website. (#589)
  • Added CREATE FUNCTION syntax for postgres and for bigquery. (#325)
  • Added CREATE INDEX and DROP INDEX for mysql. (#740)
  • Added IGNORE NULLS, RESPECT NULLS, GENERATE_DATE_ARRAY and GENERATE_TIMESTAMP_ARRAY for bigquery. ( #667, #527)
  • Added CREATE and CREATE ... CLONE for snowflake. (#539)
  • Added support for EXASOL. (#684)

Changed

  • Fixed parsing of semi-structured objects in the snowflake of dialects with whitespace gaps. #634
  • Handle internal errors elegantly, reporting the stacktrace and the error-surfacing file. #632
  • Improve message for when an automatic fix is not available for L004. #633
  • Linting errors raised on templated sections are now ignored by default and added a configuration value to show them. (#713)
  • Big refactor of logging internally. Linter is now decoupled from logging so that it can be imported directly by subprojects without needing to worry about weird output or without the log handing getting in the way of your project. (#460)
  • Linting errors in the final file are now reported with their position in the source file rather than in the templated file. This means when using sqlfluff as a plugabble library within an IDE, the references match the file which is being edited. (#541)
  • Created new Github Organisation (https://github.com/sqlfluff) and migrated from https://github.com/alanmcruickshank/sqlfluff to https://github.com/sqlfluff/sqlfluff. (#444)
  • Changed the handling of * and a.b.* expressions to have their own expressions. Any dependencies on this structure downstream will be broken. This also fixes the linting of both kinds of expressions with regard to L013 and L025. (#454)
  • Refactor of L022 to handle poorly formatted CTEs better. (#494)
  • Restriction of L017 to only fix when it would delete whitespace or newlines. (#598)
  • Added a configuration value to L016 to optionally ignore lines containing only comments. (#299)
  • Internally added an EphemeralSegment to aid with parsing efficiency without altering the end structure of the query. (#491)
  • Split ObjectReference into ColumnReference and TableReference for more useful API access to the underlying structure. (#504)
  • KeywordSegment and the new SymbolSegment both now inherit from _ProtoKeywordSegment which allows symbols to match in a very similar way to keywords without later appearing with the type of keyword. (#504)
  • Introduced the Parser class to parse a lexed query rather than relying on users to instantiate a FileSegment directly. As a result the FileSegment has been moved from the core parser directly into the dialects. Users can refer to it via the get_root_segment() method of a dialect. (#510)
  • Several performance improvements through removing unused functionality, sensible caching and optimising loops within functions. (#526)
  • Split up rule tests into separate yml files. (#553)
  • Allow escaped quotes in strings. (#557)
  • Fixed ESCAPE parsing in LIKE clause. (#566)
  • Fixed parsing of complex BETWEEN statements. (#498)
  • Fixed BigQuery EXCEPT clause parsing. (#472)
  • Fixed Rule L022 to respect leading comma configuration. (#455)
  • Improved instructions on adding a virtual environment in the README. (#457)
  • Improved documentation for passing CLI defaults in .sqlfluff. (#452)
  • Fix bug with templated blocks + capitalisation_policy = lower. (#477)
  • Fix array accessors in snowflake dialect. (#442)
  • Color logging warnings red. (#497)
  • Allow whitespace before a shorthand cast. (#544)
  • Silenced warnings when fixing from stdin. (#522)
  • Allow an underscore as the first char in a semi structured element key. (#596)
  • Fix PostFunctionGrammar in the Snowflake dialect which was causing strange behaviour in L012. (#619)
  • Bracketed segment now obtains its brackets directly from the dialect using a set named bracket_pairs. This now enables better configuration of brackets between dialects. (#325)

Removed

  • Dropped support for python 3.5. (#482)
  • From the CLI, the --no-safety option has been removed, the default is now that all enabled rules will be fixed. (#583)
  • Removed BaseSegment.grammar, BaseSegment._match_grammar() and BaseSegment._parse_grammar() instead preferring references directly to BaseSegment.match_grammar and BaseSegment.parse_grammar. (#509)
  • Removed EmptySegmentGrammar and replaced with better non-code handling in the FileSegment itself. (#509)
  • Remove the ContainsOnly grammar as it remained only as an anti-pattern. (#509)
  • Removed the expected_string() functionality from grammars and segments (#509) as it was poorly supported.
  • Removed BaseSegment.as_optional() as now this functionality happens mostly in grammars (including Ref). (#509)
  • Removed ColumnExpressionSegment in favour of ColumnReference. (#512)
  • Removed the LambdaSegment feature, instead replacing with an internal to the grammar module called NonCodeMatcher. (#512)
  • Case sensitivity as a feature for segment matching has been removed as not required for existing dialects. (#517)
  • Dependency on difflib or cdifflib, by relying on source mapping instead to apply fixes. (#541)

[0.3.6] - 2020-09-24

Added

  • sqlfluff dialects command to get a readout of available dialects [+ associated docs].
  • More helpful error messages when trying to run in Python2.
  • Window functions now parse with IGNORE/RESPECT NULLS.
  • Parsing of current_timestamp and similar functions. Thanks @dmateusp.
  • Snowflake QUALIFY clause.

Changed

  • Respect user config directories. Thanks @sethwoodworth.
  • Fix incorrect reporting of L013 with *. Thanks @dmateusp.
  • Fix incorrect reporting of L027 with column aliases. Thanks @pwildenhain.
  • Simplification of application of fixes and correction of a case where fixes could be depleted. Thanks @NiallRees.
  • Fix functions with a similar structure to SUBSTRING.
  • Refactor BigQuery REPLACE and EXCEPT clauses.
  • Bigquery date parts corrected.
  • Snowflake array accessors.
  • Psotgres NOTNULL and ISNULL.
  • Bugfix in snowflake for keywords used in semistructured queries.
  • Nested WITH statements now parse.
  • Performance improvements in the fix command.
  • Numeric literals starting with a decimal now parse.
  • Refactor the jinja templater.

[0.3.5] - 2020-08-03

Added

  • Patterns and Anti-patterns in documentation. Thanks @flpezet.
  • Functions in GROUP BY. Thanks @flpezet.

Changed

  • Deep bugfixes in the parser to handle simple matching better for a few edge cases. Also added some logging deeper in the parser.
  • Added in the SelectableGrammar and some related segments to make it easier to refer to select-like things in other grammars.
  • Fixes to CASE statement parsing. Thanks @azhard.
  • Fix to snowflake SAMPLE implementation. Thanks @rkm3.
  • Numerous docs fixes. Thanks @SimonStJG, @flpezet, @s-pace, @nolanbconaway.

[0.3.4] - 2020-05-13

Changed

  • Implementation of the bigquery CREATE MODEL syntax. Thanks @barrywhart.
  • Bugfixes for:
    • Edge cases for L006
    • False alarms on L025
    • ORDER BY x NULLS FIRST|LAST
    • FOR keyword in bigquery SYSTEM_TIME syntax.

[0.3.3] - 2020-05-11

Added

  • Added the --nofail option to parse and lint commands to assist rollout.
  • Added the --version option to complement the version option already available on the cli.
  • Parsing for ALTER TABLE.
  • Warning for unset dialects when getting parsing errors.
  • Configurable line lengths for output.

[0.3.2] - 2020-05-08

Added

  • Support for the Teradata dialect. Thanks @Katzmann1983!
  • A much more detailed getting started guide in the docs.
  • For the parse command, added the --profiler and --bench options to help debugging performance issues.
  • Support for the do command in the jinja templater.
  • Proper parsing of the concatenate operator (||).
  • Proper indent handling of closing brackets.
  • Logging and benchmarking of parse performance as part of the CI pipeline.
  • Parsing of object references with defaults like my_db..my_table.
  • Support for the INTERVAL '4 days' style interval expression.
  • Configurable trailing or leading comma linting.
  • Configurable indentation for JOIN clauses.
  • Rules now have their own logging interface to improve debugging ability.
  • Snowflake and Postgres dialects.
  • Support for a .sqlfluffignore file to ignore certain paths.
  • More generic interfaces for managing keywords in dialects, including set interfaces for managing and creating keywords and the Ref.keyword() method to refer to them, and the ability to refer directly to keyword names in most grammars using strings directly. Includes SegmentGenerator objects to bind dialect objects at runtime from sets. Thanks @Katzmann1983!
  • Rule L029 for using unreserved keywords as variable names.
  • The jinja templater now allows macros loaded from files, and the hydration of variables ending in _path in the config files.
  • JSON operators and the DISTINCT ON () syntax for the postgres dialect.

Changed

  • Refactor of whitespace and non-code handling so that segments are less greedy and default to not holding whitespace on ends. This allows more consistent linting rule application.
  • Change config file reading to case-sensitive to support case sensitivity in jinja templating.
  • Non-string values (including lists) now function in the python and jinja templating libraries.
  • Validation of the match results of grammars has been reduced. In production cases the validation will still be done, but only on parse and not on match.
  • At low verbosities, python level logging is also reduced.
  • Some matcher rules in the parser can now be classified as simple which allows them to shortcut some of the matching routines.
  • Yaml output now double quotes values with newlines or tab characters.
  • Better handling on hanging and closing indents when linting rule L003.
  • More capable handline of multi-line comments so that indentation and line length parsing works. This involves some deep changes to the lexer.
  • Getting violations from the linter now automatically takes into account of ignore rules and filters.
  • Several bugfixes, including catching potential infinite regress during fixing of files, if one fix would re-introduce a problem with another.
  • Behaviour of the Bracketed grammar has been changed to treat its content as a Sequence rather than a OneOf.
  • Move to SandboxedEnvironment rather than Environment for jinja templating for security.
  • Improve reporting of templating issues, especially for the jinja templater so that missing variables are rendered as blanks, but still reported as templating violations.

[0.3.1] - 2020-02-17

Added

  • Support for a.b.* on top of a.* in select target expressions.

[0.3.0] - 2020-02-15

Changed

  • Deprecated python 2.7 and python 3.4 which are now both past their maintenance horizon. The 0.2.x branch will remain available for continued development for these versions.
  • Rule L003 is now significantly smarter in linting indentation with support for hanging indents and comparison to the most recent line which doesn't have an error. The old (more simple) functionality of directly checking whether an indent was a multiple of a preset value has been removed.
  • Fixed the "inconsistent" bug in L010. Thanks @nolanbconaway.
  • Updated logging of parsing and lexing errors to have more useful error codes.
  • Changed parsing of expressions to favour functions over identifiers to fix the expression bug.
  • Fixed the "inconsistent" bug in L010. Thanks @nolanbconaway.
  • Moved where the SELECT keyword is parsed within a select statement, so that it belongs as part of the newly renamed select_clause (renamed from previously select_target_group).
  • Clarified handling of the type and name properties of the BaseSegment class and its children. name should be specific to a particular kind of segment, and type should express a wider group. Handling of the newline, whitespace and comma segments has been updated so that we use the type property for most use cases rather than name.

Added

  • Meta segments for indicating where things can be present in the parsed tree. This is mostly illustrated using the Indent and Dedent segments used for indicating the position of theoretical indents in the structure. Several helper functions have been added across the codebase to handle this increase in the kinds of segments which might be encountered by various grammars.
  • Rule L016 has been added to lint long lines. In the fix phase of this rule, there is enough logic to try and reconstruct a sensible place for line breaks as re-flow the query. This will likely need further work and may still encounter places where it doesn't fix all errors but should be able to deal with the majority of simple cases.
  • BigQuery dialect, initially just for appropriate quoting.
  • Added parsing of DDL statements such as COMMIT, DROP, GRANT, REVOKE and ROLLBACK. Thanks @barrywhart.
  • --format option to the parse command that allows a yaml output. This is mostly to make test writing easier in the development process but might also be useful for other things.
  • Parsing of set operations like UNION.
  • Support for the diff-cover tool. Thanks @barrywhart.
  • Enabled the fix command while using stdin. Thanks @nolanbconaway.
  • Rule to detect incorrect use of DISTINCT. Thanks @barrywhart.
  • Security fixes from DeepCover. Thanks @sanketsaurav.
  • Automatic fix testing, to help support the newer more complicated rules.
  • Interval literals
  • Support for the source macro from dbt. Thanks @Dandandan
  • Support for functions with spaces between the function name and the brackets and a linting rule L017 to catch this.
  • Efficiency cache for faster pruning of the parse tree.
  • Parsing of array notation as using in BigQuery and Postgres.
  • Enable the ignore parameter on linting and fixing commands to ignore particular kinds of violations.

[0.2.4] - 2019-12-06

Added

  • A --code-only option to the parse command to spit out a more simplified output with only the code elements.
  • Rules can now optionally override the description of the violation and pass that back via the LintingResult.

Changed

  • Bugfix, correct missing files in setup.py install_requires section.
  • Better parsing of the not equal operator.
  • Added more exclusions to identifier reserved words to fix cross joins.
  • At verbosity levels 2 or above, the root config is printed and then any diffs to that for specific files are also printed.
  • Linting and parsing of directories now reports files in alphabetical order. Thanks @barrywhart.
  • Better python 2.7 stability. Thanks @barrywhart.
  • Fixing parsing of IN/NOT IN and IS/IS NOT.

[0.2.3] - 2019-12-02

Changed

  • Bugfix, default config not included.

[0.2.2] - 2019-12-02

Changed

  • Tweak rule L005 to report more sensibly with newlines.
  • Rework testing of rules to be more modular.
  • Fix a config file bug if no root config file was present for some values. Thanks @barrywhart.
  • Lexing rules are now part of the dialect rather than a global so that they can be overridden by other dialects when we get to that stage.

[0.2.0] - 2019-12-01

Added

  • Templating support (jinja2, python or raw).
    • Variables + Macros.
    • The fix command is also sensitive to fixing over templates and will skip certain fixes if it feels that it's conflicted.
  • Config file support, including specifying context for the templater.
  • Documentation via Sphinx and readthedocs.
    • Including a guide on the role of SQL in the real world. Assisted by @barrywhart.
  • Documentation LINTING (given we're a linting project) introduced in CI.
  • Reimplemented L006 & L007 which lint whitespace around operators.
  • Ability to configure rule behaviour directly from the config file.
  • Implemented L010 to lint capitalisation of keywords.
  • Allow casting in the parser using the :: operator.
  • Implemented GROUP BYand LIMIT.
  • Added ORDER BY using indexes and expressions.
  • Added parsing of CASE statements.
  • Support for window/aggregate functions.
  • Added linting and parsing of alias expressions.

Changed

  • Fixed a bug which could cause potential infinite recursion in configuration
  • Changed how negative literals are handled, so that they're now a compound segment rather than being identified at the lexing stage. This is to allow the parser to resolve the potential ambiguity.
  • Restructure of rule definitions to be more streamlined and also enable autodocumentation. This includes a more complete RuleSet class which now holds the filtering code.
  • Corrected logging in fix mode not to duplicate the reporting of errors.
  • Now allows insert statements with a nested with clause.
  • Fixed verbose logging during parsing.
  • Allow the Bracketed grammar to optionally match empty brackets using the optional keyword.

[0.1.5] - 2019-11-11

Added

  • Python 3.8 Support!

Changed

  • Moved some of the responsibility for formatted logging into the linter to mean that we can log progressively in large directories.
  • Fixed a bug in the grammar where one of the return values was messed up.

[0.1.4] - 2019-11-10

Added

  • Added a --exclude-rules argument to most of the commands to allow rule users to exclude specific subset of rules, by @sumitkumar1209
  • Added lexing for !=, ~ and ::.
  • Added a new common segment: LambdaSegment which allows matching based on arbitrary functions which can be applied to segments.
  • Recursive Expressions for both arithmetic and functions, based heavily off the grammar provided by the guys at CockroachDB.
  • An Anything grammar, useful in matching rather than in parsing to match anything.

Changed

  • Complete rewrite of the bracket counting functions, using some centralised class methods on the BaseGrammar class to support common matching features across multiple grammars. In particular this affects the Delimited grammar which is now much simpler but does also require slightly more liberal use of terminators to match effectively.
  • Rather than passing around multiple variables during parsing and matching, there is now a ParseContext object which contains things like the dialect and various depths. This simplifies the parsing and matching code significantly.
  • Bracket referencing is now done from the dialect directly, rather than in individual Grammars (except the Bracketed grammar, which still implements it directly). This takes out some originally duplicated code.
  • Corrected the parsing of ordering keywords in and ORDER BY clause.

Removed

  • Removed the bracket_sensitive_forward_match method from the BaseGrammar. It was ugly and not flexible enough. It's been replaced by a suite of methods as described above.

[0.1.3] - 2019-10-30

Changed

  • Tweak to the L001 rule so that it doesn't crash the whole thing.

[0.1.2] - 2019-10-30

Changed

  • Fixed the errors raised by the lexer.

[0.1.1] - 2019-10-30

Changed

  • Fixed which modules from sqlfluff are installed in the setup.py. This affects the version command.

[0.1.0] - 2019-10-29

Changed

  • Big Rewrite - some loss in functionality might be apparent compared to pre-0.1.0. Please submit any major problems as issues on github
  • Changed unicode handling for better escape codes in python 2. Thanks @mrshu
  • BIG rewrite of the parser, completely new architecture. This introduces breaking changes and some loss of functionality while we catch up.
    • In particular, matches now return partial matches to speed up parsing.
    • The Delimited matcher has had a significant re-write with a major speedup and broken the dependency on Sequence.
    • Rewrite of StartsWith and Sequence to use partial matches properly.
    • Different treatment of numeric literals.
    • Both Bracketed and Delimited respect bracket counting.
    • MASSIVE rewrite of Bracketed.
  • Grammars now have timers.
  • Joins properly parsing,
  • Rewrite of logging to selectively output commands at different levels of verbosity. This uses the verbosity_logger method.
  • Added a command line sqlfluff parse option which runs just the parsing step of the process to better understand how a file is being parsed. This also has options to configure how deep we recurse.
  • Complete Re-write of the rules section, implementing new crawlers which implement the linting rules. Now with inbuilt fixers in them.
  • Old rules removed and re implemented so we now have parity with the old rule sets.
  • Moved to using Ref mostly within the core grammar so that we can have recursion.
  • Used recursion to do a first implementation of arithmetic parsing. Including a test for it.
  • Moved the main grammar into a separate dialect and renamed source and test files accordingly.
  • Moved to file-based tests for the ansi dialect to make it easier to test using the tool directly.
  • As part of file tests - expected outcomes are now encoded in yaml to make it easier to write new tests.
  • Vastly improved readability and debugging potential of the _match logging.
  • Added support for windows line endings in the lexer.

[0.0.7] - 2018-11-19

Added

  • Added a sqlfluff fix as a command to implement auto-fixing of linting errors. For now only L001 is implemented as a rule that can fix things.
  • Added a rules command to introspect the available rules.
  • Updated the cli table function to use the testwrap library and also deal a lot better with longer values.
  • Added a --rules argument to most of the commands to allow rule users to focus their search on a specific subset of rules.

Changed

  • Refactor the cli tests to use the click CliRunner. Much faster

[0.0.6] - 2018-11-15

Added

  • Number matching

Changed

  • Fixed operator parsing and linting (including allowing the exception of (*))

[0.0.5] - 2018-11-15

Added

  • Much better documentation including the DOCS.md

Changed

  • Fixed comma parsing and linting

[0.0.4] - 2018-11-14

Added

  • Added operator regexes
  • Added a priority for matchers to resolve some ambiguity
  • Added tests for operator regexes
  • Added ability to initialise the memory in rules

[0.0.3] - 2018-11-14

Added

  • Refactor of rules to allow rules with memory
  • Adding comma linting rules (correcting the single character matchers)
  • Adding mixed indentation linting rules
  • Integration with CircleCI, CodeCov and lots of badges

Changed

  • Changed import of version information to fix bug with importing config.ini
  • Added basic violations/file reporting for some verbosities
  • Refactor of rules to simplify definition
  • Refactor of color cli output to make it more reusable

[0.0.2] - 2018-11-09

Added

  • Longer project description
  • Proper exit codes
  • colorama for colored output

Changed

  • Significant CLI changes
  • Much improved output from CLI

[0.0.1] - 2018-11-07

Added

  • Initial Commit! - VERY ALPHA
  • Restructure into package layout
  • Adding Tox and Pytest so that they work