Add projects' interactive documentation

[trallard]

📝 Summary

Expand the availability of interactive documentation examples and tutorials to at least five projects.

🚀 Tasks / Deliverables

See the "Status" section below

📅 Estimated completion

24 months milestone

📋 Additional information

Status

Interactive documentation websites for projects: insights

This table is best complemented by the one in #18 which contains further linked PRs about out-of-tree Pyodide/WASM CI builds (first column here) and around PRs linked towards the availability of nightly wheels (third column here). The nightly wheels for projects have been hosted on the Anaconda.org PyPI index at https://anaconda.org/scientific-python-nightly-wheels. For the lack of CORS headers around the installation of nightly wheels, please refer to pyodide/pyodide#4898. For discussions predating this choice where thoughts pertaining to an ideal location for nightly wheels were being exchanged, please refer to pyodide/pyodide#3049.

Package name	🧱 Builds for WASM (either out-of-tree CI builds, in-tree within the Pyodide distribution, or both)	🚀 Interactive documentation deployments	🌃 Use of nightly WASM wheels in deployments
NumPy	✅ Both	🚀 Docstring-based API examples (where applicable) merged at numpy/numpy#26745 and available on the "latest" documentation at https://numpy.org/devdocs/reference/. They will be available in the stable release with NumPy 2.3.0: https://numpy.org/devdocs/release/2.3.0-notes.html#interactive-examples-in-the-numpy-documentation; NumPy 2.3 maintenance branch cut hasn't happened yet. The "User Guide" section is being converted and awaits design decisions. The NumPy tutorials website is not ready yet.	Nightly WASM wheels are available but have not yet been utilised.
PyWavelets	✅ Both	Examples under "API Reference" were enabled for interactivity via PyWavelets/pywt#728. PyWavelets/pywt#741 aims to convert the "Usage examples" section with long-form notebook-based content. There are some discussions around switching to GitHub Pages (to be finalised) at PyWavelets/pywt#777.	Planned, and nightly wheels available but not used yet
pandas	✅ Both	Planned after pandas-dev/pandas#60758 is merged	Planned, nightly wheels available but not used yet
awkward and awkward-cpp	CI builds for WASM in out-of-tree are enabled with a docs-based workflow, though not fully tested via the test suite. In-tree builds available with Pyodide releases	An interactive shell is available at https://awkward-array.org/doc/main/_static/try-it.html which preloads Awkward Array for importing – does not use jupyterlite-sphinx yet	Extra wheels left in with Pyodide distribution using JupyterLite. Nightly wheels are not hosted on Anaconda.org or used
scikit-learn	✅ Both	Yes, deployed using Sphinx-Gallery's JupyterLite integration	Not yet – version 1.4.2 is being used for the notebooks. WASM nightly wheels are not being uploaded yet.
scikit-image	✅ Both	scikit-image/scikit-image#7644 attempts to add interactive examples	In progress in scikit-image/scikit-image#7440, waiting on Pyodide's 0.27 release and a released version of cibuildwheel to support it via pypa/cibuildwheel#2117
statsmodels	✅ Both	Design decision is needed; MkDocs is used instead of Sphinx – so either an alternative extension to enable JupyterLite would be needed, or a JupyterLite distribution has to be hosted separately that opens https://www.statsmodels.org/stable/examples/index.html in a gallery	Planned, nightly wheels available but not used yet
Zarr	WASM builds in progress. Updated versions of all dependencies are now available in Pyodide in-tree with Pyodide 0.27	Planned	Planned
numcodecs	WASM builds completed via zarr-developers/numcodecs#529, awaiting pyodide/pyodide#5432 to be released	❌ Not planned, since numcodecs is not particularly user-facing and is not a major dependency for Zarr's codecs	❌ Not planned
SciPy	In-tree builds up-to-date in Pyodide 0.27 alpha releases. Out-of-tree CI job not implemented yet because of magnitude of FORTRAN 77 patches. Possibility to use flang via LLVM 19 via pyodide/pyodide#5030 to cross-compile F77/F90 code to WASM directly instead of f2c, needs experimentation	Docstring-based API examples deployed scipy/scipy#20019, later disabled and need to be re-enabled – tracked in scipy/scipy#22241 and being looked into. The scipy.stats notebooks were deployed for a start in in scipy/scipy#20303 and scipy/scipy#21042 in a similar fashion to PyWavelets's above – via Jupytext for documentation build-time conversion from reST to MyST Markdown notebooks	Planned
SymPy	✅ Both	Documentation is built with Furo theme instead of PST, some aesthetic changes to jupyterlite-sphinx's style applied via sympy/sympy#27419, awaiting merge. Additional context: interactive doctest-based examples used to exist with a server-side Google App Engine deployment and a Sphinx extension, but have been long deprecated.	Nightly wheels are available, and are planned for use in SymPy's live shell first before proceeding to SymPy tutorials. Update as of 03/01/2025: via sympy/live#22, https://live.sympy.org is now at the latest available stable version at the time of writing. Update as of 21/01/2025: nightly wheels will be used post merge of SymPy docs interactivity PR noted above.
Matplotlib	Out-of-tree builds tracked in matplotlib/matplotlib#27870, being tracked in matplotlib/matplotlib#29093. Builds available in-tree up to version 3.8.4 via pyodide/pyodide#4510, and patched WebAgg planned for further use (see pyodide/pyodide#5398 and matplotlib/matplotlib#29568)	In progress at matplotlib/matplotlib#29506 which adds a shell and enables the gallery, Sphinx-Gallery's JupyterLite integration planned next	Planned for addition in matplotlib/matplotlib#29093
h5py and libhdf5	h5py/h5py#2397	Planned, decision needed	Planned
PyTables	Planned, decision needed	Planned, decision needed	Planned

Miscellaneous items of note

Installation of nightly wheels for projects' "dev"/"latest" docs websites

• CORS headers now available for Anaconda.org via Web proxy to work around the lack of CORS headers for nightly wheels uploaded to the Anaconda.org index pyodide/pyodide#4898 as of 06/02/2025, eliminating use of proxy

• an --index-urls CLI command for piplite is in progress at Add a --index-urls CLI flag for pip-install magics jupyterlite/pyodide-kernel#166
• Extra indices for lookup in addition to Pyodide CDN and PyPI (and local) is in progress at Allow to install extra packages from extra indexes from jupyter-lite.json jupyterlite/pyodide-kernel#158

Communication and adoption

See #17 for interactive docs SPEC – first draft in progress by @agriyakhetarpal as of 21/02/2025

Notebook styling improvements for docs

See #142 for the major round-up around design decisions for the docs.

• Gist: DOC: Add interactive notebooks to pages in the "Usage Examples" section PyWavelets/pywt#741 added better notebook input/output styling, with upstreaming and standardisation to take place in the PyData Sphinx Theme and notebook collaborators.

• Super compact error for docs tracebacks ipython/ipython#14591 addressed via Create a new xmode "Docs" ipython/ipython#14752, brings as small a traceback as possible for docs; awaits Python 9.0 release for usage.

[rgommers]

Year 1 goal of the first interactive docs for 1 project (SciPy) almost met, see gh-16.

@Carreau is now joining in to work on this larger year 2 goal.

[rgommers]

As a quick update:

• a lot of work was done on https://github.com/jupyterlite/jupyterlite-sphinx to make the user experience for interactive docs polished
• https://pywavelets.readthedocs.io/en/latest/ is deployed and contains buttons with each example in the reference guide to start JupyterLite and try things interactively
• the SciPy infra is all there, the switch just needs to be flipped now that SciPy 0.12.0 is becoming available in Pyodide 0.26.0 (not yet up on PyPI, but expected very soon).

[rgommers]

Also related: a fair amount of work is going into making binary wheels smaller (e.g. the machinery to strip off tests and data with a single build flag), reducing the number of build variants in compiled code, etc. Small binaries will be quite important to make the user experience good, because downloading many MBs of packages yields high latency and can be costly when a user doesn't have a connection with unlimited bandwidth.

[rgommers]

@agriyakhetarpal it may be useful to have a Status table in the issue description here (just like in gh-18), linking to deployed interactive docs and/or open PRs. Would you be able to create one?

[agriyakhetarpal]

Yes, I'll do this in a moment, @rgommers. I expect that my interactive docs work will increase in magnitude in the coming months.

[Carreau]

With respect to installing nightly wheels when running jupyter-lite:

• My own opinion is that one should install package when starting the kernel, cf https://github.com/jupyterlite/pyodide-kernel/pull/158 (backticks for not crosslinking). I see installing those packages not as arbitrary code execution, as IMHO jupyter lite goal is to have a environment ready for end user to install.
• There are some concern about load on Pypi and other indexes (despite CDN), and breaking pyodide itself.
• As Agriya also pointed out there is --piplite-wheel, which allows embedding wheel into jupyter-lite, it raises the questions of dependencies (usually this is used for building a PR, but deps are pulled from indexes). My understanding is that even if we had --piplite-wheel-and-all-deps, they would still not allow install at kernel startup.
  • There is a current technical limitation that piplite/micropip cannot install only from the --piplite-wheel, but that's just a technical limitation.

Alternatives to a patch in pyodide-kernel are:

• A fork.
• An Jupyterlite Extension
  • A full extension – not sure there are the right hooks ?
  • A shallow extension pyodide-kernel-autoinstall a shallow extension that just overrwite the startup method to install packages – but not sure how hard.
• Auto inject a await piplite.install([..packages..], index_urls=['proxy?']) (potentially from a "file") to avoid a big cell.
• Patch in IPython if in emscripten to install things.

See also https://github.com/jupyterlite/jupyterlite/issues/508, https://github.com/jupyterlite/pyodide-kernel/issues/60

[juntyr]

I've been working on a similar problem and am now using the following solution (not quite for micropip installs since I need locked environments):

• use micropip to create a custom lockfile that can be hosted elsewhere
• use my very bare-bones https://github.com/juntyr/a-jupyterlite-query-config extension to allow overriding the lockfile URL and preload package list via URL query parameters: https://github.com/climet-eu/lab/blob/8d33d03de19caf1a22f8611f4e634e69a6a1b931/jupyter-lite.json#L17
• now other projects can load their custom lock files and preload their packages without needing to deploy their own JupyterLite + Pyodide env