documentation: Use Sphinx for building HTML documentation (#87)

* documentation: Use Sphinx for building HTML documentation

Backwards compatible; the only (minor) alteration to `boardwalk` -- which is the only reason this is getting a
version increment -- is invocations of `boardwalk` now only emit the current loglevel -- that is,
`Log level is {loglevel}`, at verbosity 1 or higher (e.g., `boardwalk -v ...`).

Implements Sphinx to have some HTML-based documentation, so referencing documentation that is
not directly in Python source code files is possible. While is not intended to fully document all available
functions with this change, this _does_ provide the necessary framework which
allows future documentation of the code -- within the code itself by way of docstrings -- to
auto-generate updated documentation.

This can be accomplished by running `make docs` from the root of the repository, or in the alternative,
running `make html` from the `docs/` directory (if in a `poetry shell` session). This directs Sphinx to
build the HTML documentation, which when complete is available at `docs/build/html/index.html`.

.github/dependabot.yml

@@ -0,0 +1,19 @@
+# Please see the documentation for all configuration options:
+# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+
+version: 2
+updates:
+  # To monitor pyproject.toml / Poetry.lock files
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "daily"
+  # For keeping workflows up-to-date.
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "daily"
+    labels:
+      - "CI/CD"
+    commit-message:
+      prefix: ci

.github/workflows/make-test.yml

@@ -22,7 +22,7 @@ jobs:
 
       - name: Set up python
         id: setup-python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v5.3.0
         with:
           python-version: '3.11'
 
@@ -31,7 +31,7 @@ jobs:
       #----------------------------------------------
       - name: Load cached Poetry installation
         id: cached-poetry
-        uses: actions/cache@v3
+        uses: actions/cache@v4.1.2
         with:
           path: ~/.local  # the path depends on the OS
           key: poetry-0  # increment to reset cache
@@ -49,7 +49,7 @@ jobs:
       #----------------------------------------------
       - name: Load cached venv
         id: cached-poetry-dependencies
-        uses: actions/cache@v3
+        uses: actions/cache@v4.1.2
         with:
           path: .venv
           key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}

.gitignore

@@ -9,3 +9,5 @@ __pycache__/
 *.egg-info/
 build/
 dist/
+# Contains files which are automatically generated by the documentation build process.
+docs/source/_autogenerated

Makefile

@@ -11,8 +11,9 @@ container:
 clean:
 	set -o pipefail
 
-	@echo '[.] Removing built packages ...'
+	@echo '[.] Removing built packages and documentation ...'
 	rm -rf dist/
+	make --directory=./docs/ clean
 
 	@echo '[.] Cleaning __pycache__ directories ...'
 	find . -type d -name '__pycache__' | xargs rm -rf
@@ -50,6 +51,12 @@ endif
 dist: clean
 	poetry build
 
+# Builds the Sphinx HTML documentation -- Shortcut for `cd docs && make html`
+.PHONY: docs
+docs: develop
+	poetry install --with=docs --sync
+	poetry run make --directory=./docs/ html
+
 # Applies fixable errors, and formats code
 .PHONY: format
 format:

README.md

@@ -1,6 +1,8 @@
 Boardwalk
 =========
+<div id="boardwalk-readme-icon">
 <img src="src/boardwalkd/static/boardwalk_icon.jpg" style="width: 25%;" align="right" alt="A picture of a boardwalk through a forest."/>
+</div>
 
 Boardwalk is an open-source linear [Ansible](https://www.ansible.com/) workflow engine. It's
 purpose-built to help systems engineers automate low-and-slow background jobs
@@ -17,9 +19,11 @@ stopping and resuming long-running Ansible workflows easy and efficient.
 💥 __Boardwalk is alpha software. Interfaces and behaviors may change between
 updates.__ 💥
 
-<!-- toc -->
+
+<!-- README.md-Table-of-Contents_Before -->
 # Table of Contents
 - [Motivation & Goals](#motivation--goals)
+- [Installation](#installation)
 - [Open-Source License](#open-source-license)
 - [Contributing](#contributing)
 - [Concepts](#concepts)
@@ -29,11 +33,12 @@ updates.__ 💥
     - [Local State](#local-state)
     - [Remote State](#remote-state)
 - [Usage](#usage)
-    - [Installation](#installation)
     - [The `Boardwalkfile.py`](#the-boardwalkfilepy)
 - [Command-line Interface](#environment-variables)
 - [`boardwalkd` Server](#boardwalkd-server)
 
+<!-- README.md-Table-of-Contents_After -->
+
 # Motivation & Goals
 
 Ansible is already a very capable agentless, modular, remote execution engine.
@@ -70,13 +75,71 @@ interface used by [AWX](https://github.com/ansible/awx).
   own secret management system, because Ansible already has solutions to those
   areas.
 
+# Installation
+<!-- README.md_Installation_Section_Before -->
+
+We recommend installing `boardwalk` in a [`pipx`](https://pipx.pypa.io/stable/)
+environment, to ensure that your system or user pip packages are not affected by
+Boardwalk's dependencies. Refer to `pipx`'s documentation for how to install it,
+and then execute the following commands:
+```sh
+# Boardwalk depends on `ansible` in order to run.
+pipx install ansible>=6.5.0
+# Inject Boardwalk into the environment so it can use Ansible. Optionally,
+# appending an @<git reference> allows for installing specific commits,
+# branches, or tags.
+pipx inject ansible git+ssh://[email protected]/Backblaze/boardwalk.git
+```
+
+__Note__:
+Boardwalk should be designed to be compatible with the current stable Python3
+release, minus 1 minor version. So, if the current stable version is `3.22.x`,
+Boardwalk should work with `3.21.x`. Consider using
+[`pyenv`](https://github.com/pyenv/pyenv) to maintain fresh python environment.
+
+## Shell Completion
+
+To enable shell completion for `boardwalk` and `boardwalkd`, the following set
+of commands will generate the completion script and add them to your shell (a
+shell restart will be needed):
+
+### Bash
+```bash
+_BOARDWALK_COMPLETE=bash_source boardwalk > ~/.boardwalk-complete.bash
+_BOARDWALKD_COMPLETE=bash_source boardwalkd > ~/.boardwalkd-complete.bash
+echo '. ~/.boardwalk-complete.bash' >> ~/.bashrc
+echo '. ~/.boardwalkd-complete.bash' >> ~/.bashrc
+```
+
+### Zsh
+```zsh
+_BOARDWALK_COMPLETE=zsh_source boardwalk > ~/.boardwalk-complete.zsh
+_BOARDWALKD_COMPLETE=zsh_source boardwalkd > ~/.boardwalkd-complete.zsh
+echo '. ~/.boardwalk-complete.zsh' >> ~/.zshrc
+echo '. ~/.boardwalkd-complete.zsh' >> ~/.zshrc
+```
+
+### Fish
+```sh
+_BOARDWALK_COMPLETE=fish_source boardwalk > ~/.config/fish/completions/boardwalk.fish
+_BOARDWALKD_COMPLETE=fish_source boardwalkd > ~/.config/fish/completions/boardwalkd.fish
+```
+
+## Container Install
+
+Boardwalk may be built as a container image by running `make container`.
+
+The entrypoint is simply `python -m` so either `boardwalk` or `boardwalkd`
+must be specified as the command when running.
+<!-- README.md_Installation_Section_After -->
+
 # Open-Source License
 
-Boardwalk is open source, licensed under the terms of the [MIT license](LICENSE).
+Boardwalk is open source, licensed under the terms of the [MIT license](https://github.com/Backblaze/boardwalk/blob/main/LICENSE).
 
 # Contributing
 
-See [CONTRIBUTING.md](./CONTRIBUTING.md).
+See [CONTRIBUTING.md](https://github.com/Backblaze/boardwalk/blob/main/CONTRIBUTING.md).
 
 # Concepts
 
@@ -88,7 +151,7 @@ Workspaces define isolated configurations and state for working on projects with
 Boardwalk. They define the Ansible host pattern Boardwalk should target, the
 Workflow Boardwalk will use, and some essential configuration options.
 
-Workspaces are defined in the [Boardwalkfile.py](#the-boardwalkfile-py). The
+Workspaces are defined in the [Boardwalkfile.py](#the-boardwalkfilepy). The
 active Workspace is selected with `boardwalk workspace use <workspace name>`.
 
 ## Workflow
@@ -100,7 +163,7 @@ purpose of a Workflow is to mutate a host from one state to another. Typically
 Workflows depend upon some Ansible fact(s) having changed after the Workflow has
 completed all of its Jobs.
 
-Workflows are defined in the [Boardwalkfile.py](#the-boardwalkfile-py).
+Workflows are defined in the [Boardwalkfile.py](#the-boardwalkfilepy).
 Workflows can be dry-run with `boardwalk check`, which runs Ansible in `--check`
 mode. Workflows are run with `boardwalk run`.
 
@@ -111,7 +174,7 @@ that are run against hosts. They accept options that can be passed into them and
 used in tasks. They define preconditions that a host must meet before a Workflow
 will run against it.
 
-Jobs are defined in the [Boardwalkfile.py](#the-boardwalkfile-py).
+Jobs are defined in the [Boardwalkfile.py](#the-boardwalkfilepy).
 
 ### Job Preconditions
 
@@ -160,62 +223,6 @@ Boardwalk.
 
 # Usage
 
-## Installation
-
-We recommend installing `boardwalk` in a [`pipx`](https://pipx.pypa.io/stable/)
-environment, to ensure that your system or user pip packages are not affected by
-Boardwalk's dependencies. Refer to `pipx`'s documentation for how to install it,
-and then execute the following commands:
-```sh
-# Boardwalk depends on `ansible` in order to run.
-pipx install ansible>=6.5.0
-# Inject Boardwalk into the environment so it can use Ansible. Optionally,
-# appending an @<git reference> allows for installing specific commits,
-# branches, or tags.
-pipx inject ansible git+ssh://[email protected]/Backblaze/boardwalk.git
-```
-
-__Note__:
-Boardwalk should be designed to be compatible with the current stable Python3
-release, minus 1 minor version. So, if the current stable version is `3.22.x`,
-Boardwalk should work with `3.21.x`. Consider using
-[`pyenv`](https://github.com/pyenv/pyenv) to maintain fresh python environment.
-
-### Shell Completion
-
-To enable shell completion for `boardwalk` and `boardwalkd`, the following set
-of commands will generate the completion script and add them to your shell (a
-shell restart will be needed):
-
-#### Bash
-```bash
-_BOARDWALK_COMPLETE=bash_source boardwalk > ~/.boardwalk-complete.bash
-_BOARDWALKD_COMPLETE=bash_source boardwalkd > ~/.boardwalkd-complete.bash
-echo '. ~/.boardwalk-complete.bash' >> ~/.bashrc
-echo '. ~/.boardwalkd-complete.bash' >> ~/.bashrc
-```
-
-#### Zsh
-```zsh
-_BOARDWALK_COMPLETE=zsh_source boardwalk > ~/.boardwalk-complete.zsh
-_BOARDWALKD_COMPLETE=zsh_source boardwalkd > ~/.boardwalkd-complete.zsh
-echo '. ~/.boardwalk-complete.zsh' >> ~/.zshrc
-echo '. ~/.boardwalkd-complete.zsh' >> ~/.zshrc
-```
-
-#### Fish
-```sh
-_BOARDWALK_COMPLETE=fish_source boardwalk > ~/.config/fish/completions/boardwalk.fish
-_BOARDWALKD_COMPLETE=fish_source boardwalkd > ~/.config/fish/completions/boardwalkd.fish
-```
-
-### Container Install
-
-Boardwalk may be built as a container image by running `make container`.
-
-The entrypoint is simply `python -m` so either `boardwalk` or `boardwalkd`
-must be specified as the command when running.
-
 ## The `Boardwalkfile.py`
 
 Boardwalk is both a python library and command-line tool. The `boardwalk`

docs/Boardwalkfile.py

@@ -0,0 +1,10 @@
+# Dummy file to let help be generated...
+# TODO: Figure out how to let things like the below work without erroring; it should "just work"
+#
+# rich-click --output html boardwalk workspace --help
+# 2024-10-31 16:33:18.521 | INFO     | boardwalk.cli:cli:77 - Log level is INFO
+
+# ╭─ Error ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+# │ No Boardwalkfile.py found                                                                                                                                                                                                                       │
+# ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+# (boardwalk-py3.12) asullivan@MBP-NT9RPG2XV7 boardwalk %

docs/Makefile

@@ -0,0 +1,36 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+.PHONY: help Makefile
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+# Define some build targets manually, since we also have the auto-generated documentation to handle
+.PHONY: clean
+clean:
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	-rm -r source/_autogenerated
+
+.PHONY: generate_cli_help_page_sources
+generate_cli_help_page_sources:
+	./build_cli_help_pages.sh
+
+.PHONY: html
+html: generate_cli_help_page_sources
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+livehtml: generate_cli_help_page_sources
+	sphinx-autobuild "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/build_cli_help_pages.sh

@@ -0,0 +1,69 @@
+#!/bin/bash
+
+# TODO / Possibility: Integrate rich-click into boardwalk/boardwalkd directly,
+# so this workaround isn't needed to generate a pretty help text.
+
+commands=(
+    "boardwalk"
+    "boardwalk catch"
+    "boardwalk check"
+    "boardwalk init"
+    "boardwalk login"
+    "boardwalk release"
+    "boardwalk run"
+    "boardwalk version"
+    "boardwalk workspace"
+    "boardwalk workspace dump"
+    "boardwalk workspace list"
+    "boardwalk workspace reset"
+    "boardwalk workspace show"
+    "boardwalk workspace use"
+
+    "boardwalkd"
+    "boardwalkd serve"
+    "boardwalkd version"
+)
+
+OUTPUT_WIDTH_COLUMMNS=110
+
+GET_PAGE_NAME() {
+    echo '`'"$1"'`'
+}
+
+AUTO_GEN_DIR=source/_autogenerated/cli_helpdocs
+mkdir -p "$AUTO_GEN_DIR"
+mkdir -p "$AUTO_GEN_DIR/boardwalk"
+mkdir -p "$AUTO_GEN_DIR/boardwalkd"
+
+for cmd in "${commands[@]}"; do
+    # Define regular expression patterns for 'boardwalk' and 'boardwalkd', so we
+    # can sort them into the correct subdirectories. Note that we need to use
+    # POSIX-compliant EREs, here, since this is portable between MacOS/Linux.
+    # See: https://stackoverflow.com/a/12696899
+    # This is, effectively, '\bboardwalk\b' and '\bboardwalkd\b'
+    BOARDWALK_REGEX="^[[:<:]]boardwalk[[:>:]]"
+    BOARDWALKD_REGEX="^[[:<:]]boardwalkd[[:>:]]"
+    if [[ $cmd =~ $BOARDWALK_REGEX ]]; then
+        SUBDIR=boardwalk
+    elif [[ $cmd =~ $BOARDWALKD_REGEX ]]; then
+        SUBDIR=boardwalkd
+    else
+        # Don't assume if we cannot correctly parse the command being generated
+        echo "[!] Skipping generation for $cmd; is this a boardwalk or boardwalkd command?"
+        continue
+    fi
+    FILENAME=$AUTO_GEN_DIR/$SUBDIR/$(echo "$cmd" | tr ' ' _).md
+    echo "[+] Generating doc page for $cmd"
+    touch "$FILENAME"
+    {
+        echo "# $(GET_PAGE_NAME "$cmd")"
+        echo ""
+        echo '<div class="full-width" id="cmd-help-text">'
+        echo '<pre>'
+        # shellcheck disable=SC2086  # We actually want word splitting, here
+        COLUMNS=$OUTPUT_WIDTH_COLUMMNS rich-click --output html $cmd --help
+        echo '</pre>'
+        echo '</div>'
+        echo ''
+    }  > "$FILENAME"
+done

docs/source/_img/diagrams

@@ -0,0 +1 @@
+../../../diagrams