feat: setup.py redesign and helpers (#2433)

* feat: setup.py redesign and helpers

* refactor: simpler design with two outputs

* refactor: helper file update and Windows support

* fix: review points from @YannickJadoul

* refactor: fixes to naming and more docs

* feat: more customization points

* feat: add entry point pybind11-config

* refactor: Try Extension-focused method

* refactor: rename alt/inplace to global

* fix: allow usage with git modules, better docs

* feat: global as an extra (@YannickJadoul's suggestion)

* feat: single version location

* fix: remove the requirement that setuptools must be imported first

* fix: some review points from @wjacob

* fix: use .in, add procedure to docs

* refactor: avoid monkeypatch copy

* docs: minor typos corrected

* fix: minor points from @YannickJadoul

* fix: typo on Windows C++ mode

* fix: MSVC 15 update 3+ have c++14 flag

See <https://docs.microsoft.com/en-us/cpp/build/reference/std-specify-language-standard-version?view=vs-2019>

* docs: discuss making SDists by hand

* ci: use pep517.build instead of manual setup.py

* refactor: more comments from @YannickJadoul

* docs: updates from @ktbarrett

* fix: change to newly recommended tool instead of pep517.build

This was intended as a proof of concept; build seems to be the correct replacement.

See pypa/pyproject-hooks#83

* docs: updates from @wjakob

* refactor: dual version locations

* docs: typo spotted by @wjakob

.github/CONTRIBUTING.md

@@ -210,6 +210,108 @@ cmake -S pybind11/ -B build
 cmake --build build
 ```
 
+### Explanation of the SDist/wheel building design
+
+> These details below are _only_ for packaging the Python sources from git. The
+> SDists and wheels created do not have any extra requirements at all and are
+> completely normal.
+
+The main objective of the packaging system is to create SDists (Python's source
+distribution packages) and wheels (Python's binary distribution packages) that
+include everything that is needed to work with pybind11, and which can be
+installed without any additional dependencies. This is more complex than it
+appears: in order to support CMake as a first class language even when using
+the PyPI package, they must include the _generated_ CMake files (so as not to
+require CMake when installing the `pybind11` package itself). They should also
+provide the option to install to the "standard" location
+(`<ENVROOT>/include/pybind11` and `<ENVROOT>/share/cmake/pybind11`) so they are
+easy to find with CMake, but this can cause problems if you are not an
+environment or using ``pyproject.toml`` requirements. This was solved by having
+two packages; the "nice" pybind11 package that stores the includes and CMake
+files inside the package, that you get access to via functions in the package,
+and a `pybind11-global` package that can be included via `pybind11[global]` if
+you want the more invasive but discoverable file locations.
+
+If you want to install or package the GitHub source, it is best to have Pip 10
+or newer on Windows, macOS, or Linux (manylinux1 compatible, includes most
+distributions).  You can then build the SDists, or run any procedure that makes
+SDists internally, like making wheels or installing.
+
+
+```bash
+# Editable development install example
+python3 -m pip install -e .
+```
+
+Since Pip itself does not have an `sdist` command (it does have `wheel` and
+`install`), you may want to use the upcoming `build` package:
+
+```bash
+python3 -m pip install build
+
+# Normal package
+python3 -m build -s .
+
+# Global extra
+PYBIND11_GLOBAL_SDIST=1 python3 -m build -s .
+```
+
+If you want to use the classic "direct" usage of `python setup.py`, you will
+need CMake 3.15+ and either `make` or `ninja` preinstalled (possibly via `pip
+install cmake ninja`), since directly running Python on `setup.py` cannot pick
+up and install `pyproject.toml` requirements. As long as you have those two
+things, though, everything works the way you would expect:
+
+```bash
+# Normal package
+python3 setup.py sdist
+
+# Global extra
+PYBIND11_GLOBAL_SDIST=1 python3 setup.py sdist
+```
+
+A detailed explanation of the build procedure design for developers wanting to
+work on or maintain the packaging system is as follows:
+
+#### 1. Building from the source directory
+
+When you invoke any `setup.py` command from the source directory, including
+`pip wheel .` and `pip install .`, you will activate a full source build. This
+is made of the following steps:
+
+1. If the tool is PEP 518 compliant, like Pip 10+, it will create a temporary
+   virtual environment and install the build requirements (mostly CMake) into
+   it. (if you are not on Windows, macOS, or a manylinux compliant system, you
+   can disable this with `--no-build-isolation` as long as you have CMake 3.15+
+   installed)
+2. The environment variable `PYBIND11_GLOBAL_SDIST` is checked - if it is set
+   and truthy, this will be make the accessory `pybind11-global` package,
+   instead of the normal `pybind11` package. This package is used for
+   installing the files directly to your environment root directory, using
+   `pybind11[global]`.
+2. `setup.py` reads the version from `pybind11/_version.py` and verifies it
+   matches `includes/pybind11/detail/common.h`.
+3. CMake is run with `-DCMAKE_INSTALL_PREIFX=pybind11`. Since the CMake install
+   procedure uses only relative paths and is identical on all platforms, these
+   files are valid as long as they stay in the correct relative position to the
+   includes. `pybind11/share/cmake/pybind11` has the CMake files, and
+   `pybind11/include` has the includes. The build directory is discarded.
+4. Simpler files are placed in the SDist: `tools/setup_*.py.in`,
+   `tools/pyproject.toml` (`main` or `global`)
+5. The package is created by running the setup function in the
+   `tools/setup_*.py`.  `setup_main.py` fills in Python packages, and
+   `setup_global.py` fills in only the data/header slots.
+6. A context manager cleans up the temporary CMake install directory (even if
+   an error is thrown).
+
+### 2. Building from SDist
+
+Since the SDist has the rendered template files in `tools` along with the
+includes and CMake files in the correct locations, the builds are completely
+trivial and simple. No extra requirements are required. You can even use Pip 9
+if you really want to.
+
+
 [pre-commit]: https://pre-commit.com
 [pybind11.readthedocs.org]: http://pybind11.readthedocs.org/en/latest
 [issue tracker]: https://github.com/pybind/pybind11/issues

.github/workflows/ci.yml

@@ -10,6 +10,8 @@ on:
       - v*
 
 jobs:
+  # This is the "main" test suite, which tests a large number of different
+  # versions of default compilers and Python versions in GitHub Actions.
   standard:
     strategy:
       fail-fast: false
@@ -23,6 +25,12 @@ jobs:
         - pypy2
         - pypy3
 
+        # Items in here will either be added to the build matrix (if not
+        # present), or add new keys to an existing matrix element if all the
+        # existing keys match.
+        #
+        # We support three optional keys: args (both build), args1 (first
+        # build), and args2 (second build).
         include:
           - runs-on: ubuntu-latest
             python: 3.6
@@ -52,6 +60,7 @@ jobs:
             args: >
               -DPYBIND11_FINDPYTHON=ON
 
+        # These items will be removed from the build matrix, keys must match.
         exclude:
             # Currently 32bit only, and we build 64bit
           - runs-on: windows-latest
@@ -102,10 +111,11 @@ jobs:
     - name: Prepare env
       run: python -m pip install -r tests/requirements.txt --prefer-binary
 
-    - name: Setup annotations
+    - name: Setup annotations on Linux
       if: runner.os == 'Linux'
       run: python -m pip install pytest-github-actions-annotate-failures
 
+    # First build - C++11 mode and inplace
     - name: Configure C++11 ${{ matrix.args }}
       run: >
         cmake -S . -B .
@@ -130,6 +140,7 @@ jobs:
     - name: Clean directory
       run: git clean -fdx
 
+    # Second build - C++17 mode and in a build directory
     - name: Configure ${{ matrix.args2 }}
       run: >
         cmake -S . -B build2
@@ -152,6 +163,28 @@ jobs:
     - name: Interface test
       run: cmake --build build2 --target test_cmake_build
 
+    # Eventually Microsoft might have an action for setting up
+    # MSVC, but for now, this action works:
+    - name: Prepare compiler environment for Windows 🐍 2.7
+      if: matrix.python == 2.7 && runner.os == 'Windows'
+      uses: ilammy/msvc-dev-cmd@v1
+      with:
+        arch: x64
+
+    # This makes two environment variables available in the following step(s)
+    - name: Set Windows 🐍 2.7 environment variables
+      if: matrix.python == 2.7 && runner.os == 'Windows'
+      run: |
+        echo "::set-env name=DISTUTILS_USE_SDK::1"
+        echo "::set-env name=MSSdk::1"
+
+    # This makes sure the setup_helpers module can build packages using
+    # setuptools
+    - name: Setuptools helpers test
+      run: pytest tests/extra_setuptools
+
+
+  # Testing on clang using the excellent silkeh clang docker images
   clang:
     runs-on: ubuntu-latest
     strategy:
@@ -196,6 +229,7 @@ jobs:
       run: cmake --build build --target test_cmake_build
 
 
+  # Testing NVCC; forces sources to behave like .cu files
   cuda:
     runs-on: ubuntu-latest
     name: "🐍 3.8 • CUDA 11 • Ubuntu 20.04"
@@ -218,6 +252,7 @@ jobs:
       run: cmake --build build --target pytest
 
 
+  # Testing CentOS 8 + PGI compilers
   centos-nvhpc8:
     runs-on: ubuntu-latest
     name: "🐍 3 • CentOS8 / PGI 20.7 • x64"
@@ -256,6 +291,8 @@ jobs:
     - name: Interface test
       run: cmake --build build --target test_cmake_build
 
+
+  # Testing on CentOS 7 + PGI compilers, which seems to require more workarounds
   centos-nvhpc7:
     runs-on: ubuntu-latest
     name: "🐍 3 • CentOS7 / PGI 20.7 • x64"
@@ -303,6 +340,7 @@ jobs:
     - name: Interface test
       run: cmake3 --build build --target test_cmake_build
 
+  # Testing on GCC using the GCC docker images (only recent images supported)
   gcc:
     runs-on: ubuntu-latest
     strategy:
@@ -351,6 +389,8 @@ jobs:
       run: cmake --build build --target test_cmake_build
 
 
+  # Testing on CentOS (manylinux uses a centos base, and this is an easy way
+  # to get GCC 4.8, which is the manylinux1 compiler).
   centos:
     runs-on: ubuntu-latest
     strategy:
@@ -398,6 +438,7 @@ jobs:
       run: cmake --build build --target test_cmake_build
 
 
+  # This tests an "install" with the CMake tools
   install-classic:
     name: "🐍 3.5 • Debian • x86 •  Install"
     runs-on: ubuntu-latest
@@ -440,31 +481,39 @@ jobs:
       working-directory: /build-tests
 
 
+  # This verifies that the documentation is not horribly broken, and does a
+  # basic sanity check on the SDist.
   doxygen:
     name: "Documentation build test"
     runs-on: ubuntu-latest
-    container: alpine:3.12
 
     steps:
     - uses: actions/checkout@v2
 
-    - name: Install system requirements
-      run: apk add doxygen python3-dev
+    - uses: actions/setup-python@v2
 
-    - name: Ensure pip
-      run: python3 -m ensurepip
+    - name: Install Doxygen
+      run: sudo apt install -y doxygen
 
     - name: Install docs & setup requirements
-      run: python3 -m pip install -r docs/requirements.txt pytest setuptools
+      run: python3 -m pip install -r docs/requirements.txt
 
     - name: Build docs
       run: python3 -m sphinx -W -b html docs docs/.build
 
     - name: Make SDist
       run: python3 setup.py sdist
 
+    - run: git status --ignored
+
+    - name: Check local include dir
+      run: >
+        ls pybind11;
+        python3 -c "import pybind11, pathlib; assert (a := pybind11.get_include()) == (b := str(pathlib.Path('include').resolve())), f'{a} != {b}'"
+
     - name: Compare Dists (headers only)
+      working-directory: include
       run: |
-        python3 -m pip install --user -U ./dist/*
-        installed=$(python3 -c "import pybind11; print(pybind11.get_include(True) + '/pybind11')")
-        diff -rq $installed ./include/pybind11
+        python3 -m pip install --user -U ../dist/*
+        installed=$(python3 -c "import pybind11; print(pybind11.get_include() + '/pybind11')")
+        diff -rq $installed ./pybind11

.github/workflows/configure.yml

@@ -10,6 +10,8 @@ on:
       - v*
 
 jobs:
+  # This tests various versions of CMake in various combinations, to make sure
+  # the configure step passes.
   cmake:
     strategy:
       fail-fast: false
@@ -50,11 +52,14 @@ jobs:
     - name: Prepare env
       run: python -m pip install -r tests/requirements.txt
 
+    # An action for adding a specific version of CMake:
+    #   https://github.com/jwlawson/actions-setup-cmake
     - name: Setup CMake ${{ matrix.cmake }}
       uses: jwlawson/[email protected]
       with:
         cmake-version: ${{ matrix.cmake }}
 
+    # These steps use a directory with a space in it intentionally
     - name: Make build directories
       run: mkdir "build dir"
 
@@ -67,6 +72,7 @@ jobs:
         -DDOWNLOAD_CATCH=ON
         -DPYTHON_EXECUTABLE=$(python -c "import sys; print(sys.executable)")
 
+    # Only build and test if this was manually triggered in the GitHub UI
     - name: Build
       working-directory: build dir
       if: github.event_name == 'workflow_dispatch'
@@ -76,3 +82,57 @@ jobs:
       working-directory: build dir
       if: github.event_name == 'workflow_dispatch'
       run: cmake --build . --config Release --target check
+
+  # This builds the sdists and wheels and makes sure the files are exactly as
+  # expected. Using Windows and Python 2.7, since that is often the most
+  # challenging matrix element.
+  test-packaging:
+    name: 🐍 2.7 • 📦 tests • windows-latest
+    runs-on: windows-latest
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Setup 🐍 2.7
+      uses: actions/setup-python@v2
+      with:
+        python-version: 2.7
+
+    - name: Prepare env
+      run: python -m pip install -r tests/requirements.txt --prefer-binary
+
+    - name: Python Packaging tests
+      run: pytest tests/extra_python_package/
+
+
+  # This runs the packaging tests and also builds and saves the packages as
+  # artifacts.
+  packaging:
+    name: 🐍 3.8 • 📦 & 📦 tests • ubuntu-latest
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Setup 🐍 3.8
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+
+    - name: Prepare env
+      run: python -m pip install -r tests/requirements.txt build twine --prefer-binary
+
+    - name: Python Packaging tests
+      run: pytest tests/extra_python_package/
+
+    - name: Build SDist and wheels
+      run: |
+        python -m build -s -w .
+        PYBIND11_GLOBAL_SDIST=1 python -m build -s -w .
+
+    - name: Check metadata
+      run: twine check dist/*
+
+    - uses: actions/upload-artifact@v2
+      with:
+        path: dist/*

.github/workflows/format.yml

@@ -1,3 +1,6 @@
+# This is a format job. Pre-commit has a first-party GitHub action, so we use
+# that: https://github.com/pre-commit/action
+
 name: Format
 
 on:
@@ -17,6 +20,9 @@ jobs:
     - uses: actions/checkout@v2
     - uses: actions/setup-python@v2
     - uses: pre-commit/[email protected]
+      with:
+        # Slow hooks are marked with manual - slow is okay here, run them too
+        extra_args: --hook-stage manual
 
   clang-tidy:
     name: Clang-Tidy