doc: Add doctests (#1574)

* chore: enable doctests, edit one doc file to use doctests

* test: make advanced_usage.rst into doctests

* test: make compiler.rst into doctests

* test: make exercises.rst into doctests

* test: make remaining rst files into doctests

* test: make pytest run doctests on api docs

* doc: apply suggestions from self code review

* doc: fix docs formatting

* ci: add doctest step to CI

* chore: mention 'make doctest' in CONTRIBUTING.md

* chore: add link to issue

* chore: update dependencies

* doc: metadata are now properties, not dictionary keys

* fix: fix type hint

* test: disable test requiring QCS API access

* ci: install pandoc for doctests

* test: disable test requiring QCS API access

* test: disable tests requiring QCS API access

* fix: fall back to defaults when creating a QCSClient fails

* uncomment native_quil_metadata print

* remove redundant client load in _wavefunction_simulator

* set protoquil flag

* feedback

---------

Co-authored-by: marquessv <[email protected]>

.github/workflows/test.yml

@@ -106,6 +106,31 @@ jobs:
           . scripts/ci_install_deps
           poetry run make check-types
 
+  test-doctest:
+    name: Run Doctests
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - uses: actions/cache@v2
+        with:
+          path: .venv
+          key: poetry-${{ hashFiles('poetry.lock') }}
+      - name: Test Unit (Python ${{ matrix.python-version }})
+        run: |
+          sudo apt update
+          . scripts/ci_install_deps
+          sudo apt install pandoc
+          docker run --rm -itd -p 5555:5555 rigetti/quilc -R
+          docker run --rm -itd -p 5000:5000 rigetti/qvm -S
+          poetry run make doctest
+
   test-unit:
     name: Test Unit
     runs-on: ubuntu-latest

CONTRIBUTING.md

@@ -166,6 +166,12 @@ To skip [slow tests](#slow-tests), you may run:
 make test-fast
 ```
 
+You can run documentation tests with:
+
+```bash
+make doctest
+```
+
 You can run end-to-end tests with:
 
 ```bash
@@ -182,6 +188,51 @@ make test-all TEST_QUANTUM_PROCESSOR=<quantum processor ID>
 > `get_qc()`. End-to-end tests are most useful against a real QPU, but they can also be run
 > against a QVM (e.g. `make e2e TEST_QUANTUM_PROCESSOR=2q-qvm`).
 
+#### Documentation Tests
+
+We utilize doctests to validate the examples in both our docstrings and Sphinx documentation. This
+ensures that they are correct and remain up to date.
+
+When contributing, consider adding an example to illustrate to users how pyQuil should be used.
+
+To add an example to a docstring, we use Python's [doctest module](https://docs.python.org/3/library/doctest.html#module-doctest).
+As a quick primer, you can add a doctest to a docstring by pre-pending your example code with `>>> `
+and following it with the expected output. For example:
+
+```py
+def hello_world():
+    """Prints Hello World!
+    >>> hello_world()
+    Hello World!
+    """
+    print("Hello World!")
+```
+
+To customize how output is validated, take a look at [the available option flags](https://docs.python.org/3/library/doctest.html#option-flags).
+
+If you want to add an example to Sphinx, here's a quick guide. Your example will be split between a hidden `testsetup` block 
+and a visible `testcode` block. Your expected output will go in a `testoutput` block. Each block shares a test name to tell 
+Sphinx that they are related. Building off the previous hello world example:
+
+```rst
+.. testsetup:: hello_world
+   # Code in the `testsetup` block doesn't appear in the documentation.
+   # Put any code that your example might need, but would unnecessarily
+   # clutter the documentation here.
+   from foo import hello_world
+
+.. testcode:: hello_world
+   # Code in the `testcode` block will appear in the documentation.
+   # Include code needed to illustrate your example here
+   hello_world()
+
+.. testoutput:: hello_world
+   Hello World!
+```
+
+In many cases, this simple structure will suffice, but consider reading the [Sphinx doctest documentation](https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html)
+for more details on how to use it for more complex examples.
+
 #### Slow Tests
 
 Some tests (particularly those related to operator estimation and readout symmetrization)

Makefile

@@ -43,6 +43,12 @@ docs:
 	poetry install --extras docs --extras latex
 	make -C docs clean html
 
+.PHONY: doctest
+doctest:
+	poetry install --extras docs --extras latex
+	pytest -v --cov=pyquil --doctest-modules pyquil
+	make -C docs clean doctest
+
 .PHONY: docker
 docker: Dockerfile
 	docker build -t $(DOCKER_TAG) .
@@ -75,7 +81,7 @@ e2e:
 	pytest -n 1 -v --cov=pyquil test/e2e
 
 .PHONY: test-all
-test-all: test e2e
+test-all: doctest test e2e
 
 docs/quil/grammars/Quil.g4:
 	git submodule init