Create a "Guides and Howtos" section of the docs. (#6301)

CONTRIBUTING.md

@@ -25,8 +25,7 @@ for the wiki.
 
 :star2: **Fifth** - if you are so inclined - pull requests on the core codebase
 are always welcome. Dialect additions are often a good entry point for new
-contributors, and we have
-[a wiki page](https://github.com/sqlfluff/sqlfluff/wiki/Contributing-Dialect-Changes)
+contributors, and we have [a set of guides](https://docs.sqlfluff.com/en/stable/perma/guides.html)
 to help you through your first contribution. Bear in mind that all the tests
 should pass, and test coverage should not decrease unduly as part of the
 changes which you make. You may find it useful to familiarise yourself with the

docs/source/conf.py

@@ -133,18 +133,22 @@ redirects = {
     "perma/cli": "../reference/cli.html",
     "perma/rules": "../reference/rules.html",
     "perma/dialects": "../reference/dialects.html",
-    "perma/architecture": "../development/architecture.html",
+    "perma/architecture": "../guides/contributing/architecture.html",
     "perma/rule_disabling": (
         "../configuration/rule_configuration.html#enabling-and-disabling-rules"
     ),
-    "perma/internals": "../development/index.html",
+    "perma/internals": "../reference/internals/index.html",
     "perma/modularity": "../why_sqlfluff.html#modularity",
     "perma/indentation": "../configuration/layout.html#configuring-indent-locations",
     "perma/releasenotes": "../reference/releasenotes.html",
     "perma/why": "../why_sqlfluff.html",
-    "perma/plugin_dev": "../development/plugins.html",
+    "perma/plugin_dev": "../guides/contributing/plugins.html",
     "perma/variables": "../configuration/templating/index.html",
     "perma/python_templating": "../configuration/templating/python.html",
+    "perma/guides": "../guides/index.html",
+    "perma/contribute_dialect_keywords": (
+        "../guides/contributing/dialect.html#dialect_keywords"
+    ),
     # Add permalinks for rule codes
     **{
         f"perma/rule/{code}": (

docs/source/configuration/default_configuration.rst

@@ -4,7 +4,7 @@ Default Configuration
 ---------------------
 
 The default configuration is as follows, note the :ref:`builtin_jinja_blocks`
-in section *[sqlfluff:templater:jinja:macros]* as referred to above.
+in section :code:`[sqlfluff:templater:jinja:macros]` as referred to above.
 
 .. note::
 

docs/source/configuration/setting_configuration.rst

@@ -186,3 +186,5 @@ We recommend only using this configuration approach for configuration that
 applies to one file in isolation. For configuration changes for areas of
 a project or for whole projects we recommend :ref:`nesting` of configuration
 files.
+
+This syntax is very similar to the method for :ref:`inline_ignoring_errors`.

docs/source/development/dialects.rst

@@ -1,7 +0,0 @@
-Contributing Dialect Changes
-============================
-
-There's an excellent `guide on our wiki`_ on how to help the project by
-contributing dialect changes.
-
-.. _`guide on our wiki`: https://github.com/sqlfluff/sqlfluff/wiki/Contributing-Dialect-Changes

docs/source/development/index.rst

@@ -1,21 +0,0 @@
-.. _development:
-
-SQLFluff Development
-====================
-
-It is recommended that the following is read in conjunction with exploring
-the codebase. `dialect_ansi.py` in particular is helpful to understand the
-recursive structure of segments and grammars. Some more detail is also given
-on our Wiki_ including a `Contributing Dialect Changes`_ guide.
-
-.. _Wiki: https://github.com/sqlfluff/sqlfluff/wiki/
-.. _`Contributing Dialect Changes`: https://github.com/sqlfluff/sqlfluff/wiki/Contributing-Dialect-Changes
-
-.. toctree::
-   :maxdepth: 2
-
-   architecture
-   dialects
-   rules
-   reflow
-   plugins

docs/source/gettingstarted.rst

@@ -45,6 +45,8 @@ the best instructions for what to do next are `on the python website`_.
 .. _`instructions for all platforms here`: https://wiki.python.org/moin/BeginnersGuide/Download
 .. _`on the python website`: https://pip.pypa.io/en/stable/installation/
 
+.. _installingsqlfluff:
+
 Installing SQLFluff
 -------------------
 

docs/source/guides/contributing/architecture.rst

@@ -1,3 +1,5 @@
+.. _architecture:
+
 Architecture
 ------------
 

docs/source/guides/contributing/docs.rst

@@ -0,0 +1,121 @@
+Documentation Contributions
+===========================
+
+Contributing to the docs is one of the easiest and most helpful ways
+to help the project. Documentation changes require relatively little
+specialist knowledge apart from being familiar with how to use SQLFluff
+and the docs are read by a very wide range of people.
+
+Documentation takes two forms:
+
+1. Embedded documentation found in function and module `docstrings`_.
+
+2. The free-standing documentation which you're reading now, and hosted
+   at `docs.sqlfluff.com`_ (built using `sphinx`_ and `ReadtheDocs`_).
+
+The two are somewhat blurred by the use of `autodoc`_ (and some other custom
+integrations), where documentation is generated directly off `docstrings`_
+within the codebase, for example the :ref:`ruleref`, :ref:`cliref` and
+:ref:`dialectref`. To understand more about how the custom integrations
+we use to generate these docs, see the `generate-auto-docs.py`_ file.
+
+.. _`docstrings`:  https://en.wikipedia.org/wiki/Docstring
+.. _`docs.sqlfluff.com`:  https://docs.sqlfluff.com
+.. _`autodoc`: https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html
+.. _`generate-auto-docs.py`: https://github.com/sqlfluff/sqlfluff/blob/main/docs/generate-auto-docs.py
+
+.. _`ReadtheDocs`: https://about.readthedocs.com/
+
+Docstrings
+----------
+
+Embedded documentation of functions, classes and modules is most useful
+for *developer-focussed* documentation as it's most accessible in the places
+which those developers are working: *directly in the codebase*. We enforce
+that docstrings are present and correctly formatted using the
+`pydocstyle rules for ruff`_, which we have configured to enforce the
+`google style of docstrings`_.
+
+.. _`pydocstyle rules for ruff`:  https://docs.astral.sh/ruff/rules/#pydocstyle-d
+.. _`google style of docstrings`:  https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html
+
+Sphinx Docs
+-----------
+
+The main documentation (which you're reading now), is build using `sphinx`_,
+and written using `reStructuredText`_ (files ending with :code:`.rst`). The
+`sphinx`_ project offers a `reStructuredText primer`_ for people who are new
+to the syntax (and the SQLFluff project uses `doc8`_ in the CI process to try
+and catch any issues early).
+
+On top of those docs, there are a few areas worth highlighting for new (or
+returning) users, which are either specific to the SQLFluff project, or not
+particularly clear in the sphinx docs:
+
+* `reStructuredText`_ is very similar to, but differs from (the somewhat more
+  well known) `Markdown`_ syntax. Importantly:
+
+  * :code:`*text with single asterisks*` renders as *italics*. Use
+    :code:`**double asterisks**` for **bold text**.
+
+  * :code:`code snippets` are created using the |codesnippet|
+    directive, rather than just lone backticks (|backquotes|) as found in
+    most `Markdown`_.
+
+* To create links to other parts of the documentation (i.e.
+  `Cross-referencing`_), use either the :code:`:ref:` syntax.
+
+  * Docs for all the SQL dialects are auto generated with associated anchors
+    to use for referencing. For example to link to the
+    :ref:`postgres_dialect_ref` dialect docs, you can use the |postgresref|.
+    Replace the :code:`postgres` portion with the :code:`name` of the
+    dialect you want to link to.
+
+  * Docs for all the bundled rules and handled using a customer `sphinx`_
+    plugin, which means you can refer to them using their name or code:
+    |LT01ref| resolves to :sqlfluff:ref:`LT01` and |layoutspacingref|
+    resolves to :sqlfluff:ref:`layout.spacing`.
+
+  * Docs for any of the python classes and modules handled using `autodoc`_
+    can be referenced as per their docs, so the
+    :py:class:`sqlfluff.core.rules.base.BaseRule` class can be referenced
+    with |baseruleref|. You can also use the :code:`~` prefix (i.e.
+    |shortbaseruleref|) so that it just renders as
+    :py:class:`~sqlfluff.core.rules.base.BaseRule`. See the docs for
+    `Cross-referencing`_ for more details.
+
+.. _`sphinx`: https://www.sphinx-doc.org/en/master/
+.. _`reStructuredText`: https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html
+.. _`reStructuredText primer`: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
+.. _`doc8`: https://github.com/PyCQA/doc8
+.. _`Markdown`: https://www.markdownguide.org/
+.. _`Cross-referencing`: https://www.sphinx-doc.org/en/master/usage/referencing.html
+.. _`autodoc`: https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html
+
+.. |codesnippet| raw:: html
+
+    <code class="code docutils literal notranslate">:code:`...`</code>
+
+.. |backquotes| raw:: html
+
+    <code class="code docutils literal notranslate">`...`</code>
+
+.. |postgresref| raw:: html
+
+    <code class="code docutils literal notranslate">:ref:`postgres_dialect_ref`</code>
+
+.. |LT01ref| raw:: html
+
+    <code class="code docutils literal notranslate">:sqlfluff:ref:`LT01`</code>
+
+.. |layoutspacingref| raw:: html
+
+    <code class="code docutils literal notranslate">:sqlfluff:ref:`layout.spacing`</code>
+
+.. |baseruleref| raw:: html
+
+    <code class="code docutils literal notranslate">:py:class:`sqlfluff.core.rules.base.BaseRule`</code>
+
+.. |shortbaseruleref| raw:: html
+
+    <code class="code docutils literal notranslate">:py:class:`~sqlfluff.core.rules.base.BaseRule`</code>

docs/source/guides/contributing/plugins.rst

@@ -15,7 +15,7 @@ to be included in the core library.
     must be able to fully import before any rule implementations are imported.
     More specifically, SQLFluff must be able to both *import* **and**
     *run* any implementations of ``get_configs_info()`` before any plugin
-    rules (i.e. any derivatives of :py:class:`BaseRule <sqlfluff.core.rules.base.BaseRule>`)
+    rules (i.e. any derivatives of :py:class:`~sqlfluff.core.rules.base.BaseRule`)
     are *imported*. Because of this, we recommend that rules are defined in a
     separate module to the root of the plugin and then only imported within
     the ``get_rules()`` method.

docs/source/guides/contributing/rules.rst

@@ -98,45 +98,3 @@ times.
 NOTE: ``post`` rules also run on the *first* pass of the ``main`` phase so that
 any issues they find will be presented in the list of issues output by
 ``sqlfluff fix`` and ``sqlfluff lint``.
-
-Base Rules
-----------
-
-`base_rules` Module
-^^^^^^^^^^^^^^^^^^^
-
-.. automodule:: sqlfluff.core.rules.base
-   :members:
-
-Functional API
---------------
-These newer modules provide a higher-level API for rules working with segments
-and slices. Rules that need to navigate or search the parse tree may benefit
-from using these. Eventually, the plan is for **all** rules to use these
-modules. As of December 30, 2021, 17+ rules use these modules.
-
-The modules listed below are submodules of `sqlfluff.utils.functional`.
-
-`segments` Module
-^^^^^^^^^^^^^^^^^
-
-.. automodule:: sqlfluff.utils.functional.segments
-   :members:
-
-`segment_predicates` Module
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. automodule:: sqlfluff.utils.functional.segment_predicates
-   :members:
-
-`raw_file_slices` Module
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. automodule:: sqlfluff.utils.functional.raw_file_slices
-   :members:
-
-`raw_file_slice_predicates` Module
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. automodule:: sqlfluff.utils.functional.raw_file_slice_predicates
-   :members:

docs/source/guides/index.rst

@@ -0,0 +1,45 @@
+.. _guides:
+
+Guides & How-tos
+================
+
+This section is of short guides and articles is meant to be read alongside
+the rest of the documentation which is more reference-oriented.
+
+Setting up SQLFluff
+-------------------
+
+.. toctree::
+   :maxdepth: 1
+
+   setup/teamrollout
+
+..
+    NOTE: Eventually we should add a troubleshooting section here too,
+    but that doesn't have any content yet.
+
+    Troubleshooting SQLFluff
+    ------------------------
+
+.. _development:
+
+Contributing to SQLFluff
+------------------------
+
+It is recommended that the following is read in conjunction with exploring
+the codebase. `dialect_ansi.py`_ in particular is helpful to understand the
+recursive structure of segments and grammars.
+
+You may also need to reference the :ref:`internal_api_docs`.
+
+.. _`dialect_ansi.py`: https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py
+
+.. toctree::
+   :maxdepth: 1
+
+   contributing/architecture
+   contributing/git
+   contributing/dialect
+   contributing/rules
+   contributing/plugins
+   contributing/docs

docs/source/index.rst

@@ -57,14 +57,14 @@ Contents
 ^^^^^^^^
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 3
    :caption: Documentation for SQLFluff:
 
    gettingstarted
    why_sqlfluff
+   guides/index
    configuration/index
    production/index
-   development/index
    reference/index
    inthewild
    jointhecommunity

docs/source/jointhecommunity.rst

@@ -1,7 +1,7 @@
-.. _jointhecommunity:
+.. _sqlfluff_slack:
 
 SQLFluff Slack
-====================
+==============
 
 We have a fast-growing `community on Slack <https://join.slack.com/t/sqlfluff/shared_invite/zt-2qtu36kdt-OS4iONPbQ3aCz2DIbYJdWg>`_, come and join us!
 

docs/source/production/cli_use.rst

@@ -1,6 +1,10 @@
 Using SQLFluff directly as a CLI application
 --------------------------------------------
 
+The :ref:`SQLFluff CLI application <cliref>` is a python application which
+means if depends on your host python environment
+(see :ref:`installingsqlfluff`).
+
 The `exit code`_ provided by SQLFluff when run as a command line utility is
 designed to assist usefulness in deployment pipelines. If no violations
 are found then the `exit code`_ will be 0. If violations are found then
@@ -15,4 +19,7 @@ more.
   not be completed. For example a configuration issue or an internal error
   within SQLFluff.
 
+For details of what commands and options are available in the CLI see the
+:ref:`cliref`.
+
 .. _`exit code`: https://shapeshed.com/unix-exit-codes/

docs/source/production/index.rst

@@ -12,7 +12,6 @@ be part of `CI/CD`_ pipelines.
    :maxdepth: 2
 
    security
-   teamrollout
    cli_use
    diff_quality
    pre_commit

docs/source/reference/api.rst

@@ -1,7 +1,7 @@
 .. _apiref:
 
-Python API Reference
-====================
+Python API
+==========
 
 SQLFluff exposes a public api for other python applications to use.
 A basic example of this usage is given here, with the documentation

docs/source/reference/index.rst

@@ -10,4 +10,5 @@ Reference
    rules
    cli
    api
+   internals/index
    releasenotes

docs/source/reference/internals/functional.rst

@@ -0,0 +1,33 @@
+:code:`sqlfluff.utils.functional`: Functional Traversal API
+-----------------------------------------------------------
+
+These newer modules provide a higher-level API for rules working with segments
+and slices. Rules that need to navigate or search the parse tree may benefit
+from using these. Eventually, the plan is for **all** rules to use these
+modules. As of December 30, 2021, 17+ rules use these modules.
+
+The modules listed below are submodules of `sqlfluff.utils.functional`.
+
+:code:`segments` Module
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: sqlfluff.utils.functional.segments
+   :members:
+
+:code:`segment_predicates` Module
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: sqlfluff.utils.functional.segment_predicates
+   :members:
+
+:code:`raw_file_slices` Module
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: sqlfluff.utils.functional.raw_file_slices
+   :members:
+
+:code:`raw_file_slice_predicates` Module
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: sqlfluff.utils.functional.raw_file_slice_predicates
+   :members:

docs/source/reference/internals/index.rst

@@ -0,0 +1,23 @@
+.. _internal_api_docs:
+
+Internal API
+============
+
+Anything within this section should only be necessary for people who are
+developing plugins or rules to interact with SQLFluff on a deeper level or
+people who've decided to help the project by
+:ref:`contributing to SQLFluff <development>`.
+
+As these docs are some of the least commonly used, you may find that not
+all modules are documented directly here, and so you may find it instructive
+to read this together with docstrings and comments directly within the
+SQLFluff codebase on `GitHub`_.
+
+.. _`GitHub`: https://github.com/sqlfluff/sqlfluff
+
+.. toctree::
+   :maxdepth: 2
+
+   rules
+   functional
+   reflow

docs/source/reference/internals/reflow.rst

@@ -1,7 +1,7 @@
 .. _reflowinternals:
 
-Reflow Internals
-----------------
+:code:`sqlfluff.utils.reflow`: Whitespace Reflow API
+----------------------------------------------------
 
 Many rules supported by SQLFluff involve the spacing and layout of different
 elements, either to enforce a particular layout or just to add or remove

docs/source/reference/internals/rules.rst

@@ -0,0 +1,5 @@
+:code:`sqlfluff.core.rules.base`: Base Rule Classes
+---------------------------------------------------
+
+.. automodule:: sqlfluff.core.rules.base
+   :members:

src/sqlfluff/cli/commands.py

@@ -464,11 +464,16 @@ def get_linter_and_formatter(
 
 @click.group(
     context_settings={"help_option_names": ["-h", "--help"]},
-    epilog="""\b\bExamples:\n
-  sqlfluff lint --dialect postgres .\n
-  sqlfluff lint --dialect postgres --rules ST05 .\n
-  sqlfluff fix --dialect sqlite --rules LT10,ST05 src/queries\n
-  sqlfluff parse --dialect sqlite --templater jinja src/queries/common.sql
+    # NOTE: The code-block directive here looks a little odd in the CLI
+    # but is a good balance between what appears in the CLI and what appears
+    # in the auto generated docs for the CLI by sphinx.
+    epilog="""Examples:\n
+.. code-block:: sh
+
+   sqlfluff lint --dialect postgres .\n
+   sqlfluff lint --dialect mysql --rules ST05 my_query.sql\n
+   sqlfluff fix --dialect sqlite --rules LT10,ST05 src/queries\n
+   sqlfluff parse --dialect duckdb --templater jinja path/my_query.sql\n\n
 """,
 )
 @click.version_option()

src/sqlfluff/core/dialects/base.py

@@ -311,7 +311,7 @@ class Dialect:
                 " raise an issue on GitHub.\n\n"
                 "Or, even better, see this guide on how to help contribute"
                 " keyword and/or dialect updates:\n"
-                "https://github.com/sqlfluff/sqlfluff/wiki/Contributing-Dialect-Changes#keywords"  # noqa E501
+                "https://docs.sqlfluff.com/en/stable/perma/contribute_dialect_keywords.html"  # noqa E501
             )
             # Keyword errors are common so avoid printing the whole, scary,
             # traceback as not that useful and confusing to people.