Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added configuration for using specific SQLite versions. #50

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
+97 −7
Draft

Added configuration for using specific SQLite versions. #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3bd0e30
Bump default Python version to 3.12
laymonage Jan 27, 2025
d584e91
Added sqlite-lib service to test against specific SQLite versions.
laymonage Jan 28, 2025
975b9d3
Consolidated sqlite-lib service into sqlite.
laymonage Jan 28, 2025
9b59095
Fixed libsqlite3.so output path on SQLite 3.48.0+.
laymonage Jan 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
PYTHON_IMPLEMENTATION=python
PYTHON_VERSION=3.10
PYTHON_VERSION=3.12
MARIADB_VERSION=10.5
MYSQL_VERSION=8.0
ORACLE_VERSION=23.5.0.0
POSTGRESQL_VERSION=14
POSTGIS_VERSION=3.1
SQLITE_VERSION=
SQLITE_CFLAGS="-DSQLITE_ENABLE_DESERIALIZE \
-DSQLITE_ENABLE_JSON1 \
-DSQLITE_MAX_VARIABLE_NUMBER=32766"
Comment on lines +9 to +11
Copy link
Author

@laymonage laymonage Jan 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained this before in more detail in django/django#18899 (comment), but here's a breakdown of each flag:

SQLITE_ENABLE_DESERIALIZE

This flag was not enabled by default until SQLite 3.36. Without this flag, you'll encounter an error like pyenv/pyenv#2625.

  • This is because Python checks for the availability of the deserialize functions at compile time.
  • If we don't want to add this flag, we can work around this by using an older version of Python that didn't have support for serialize() (< 3.11), or by using bullseye (which ships with 3.34.1 and thus Python 3.12 wasn't built to have SQLite deserialize functions). Or, of course, compiling Python from source ourselves.

SQLITE_ENABLE_JSON1

This flag was not enabled by default until SQLite 3.38.

We have the following test that fails if you run it with a database backend that does not support JSONField, because it's missing the @skipUnlessDBFeature decorator:

https://github.com/django/django/blob/8a6b4175d790424312965ec77e4e9b072fba188b/tests/schema/tests.py#L2425-L2443

You can verify this by setting SQLITE_VERSION to a version < 3.38 and removing -DSQLITE_ENABLE_JSON1 from SQLITE_CFLAGS, or by using a version >= 3.38 and adding -DSQLITE_OMIT_JSON.

Happy to raise a ticket and PR for that.

Edit: I've raised a ticket https://code.djangoproject.com/ticket/36156#ticket

SQLITE_MAX_VARIABLE_NUMBER

There are a few tests that would fail when it does ContentType.objects.all().delete() due to a row count query exceeding the variable limit. This only happens when the whole test suite is run, but not when the failing tests are run in isolation (likely because other tests would create more ContentType instances).

This flag defaulted to 999 in < 3.32 (and 32766 in >= 3.32.0).

This is something that Debian/Ubuntu has customized, even in 3.31.

Related:

2 changes: 1 addition & 1 deletion Containerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# syntax=docker/dockerfile:1.12

ARG PYTHON_IMPLEMENTATION=python
ARG PYTHON_VERSION=3.10
ARG PYTHON_VERSION=3.12
FROM ${PYTHON_IMPLEMENTATION}:${PYTHON_VERSION}-slim-bookworm

LABEL org.opencontainers.image.authors="Django Software Foundation"
Expand Down
57 changes: 53 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Tooling and test execution support for [Django][0] :unicorn:
3. Build the image:

```console
$ docker compose build sqlite
$ docker compose build base
```

4. Run the tests:
Expand Down Expand Up @@ -193,7 +193,7 @@ environment variables:
| ----------------------- | ------------- | ---------------------------------------------------- |
| `DJANGO_PATH` | `../django` | Path to the Django repostory on your local machine |
| `PYTHON_IMPLEMENTATION` | `python` | Implementation of Python to use — `python` or `pypy` |
| `PYTHON_VERSION` | `3.10` | Version of Python container image to use |
| `PYTHON_VERSION` | `3.12` | Version of Python container image to use |

The versions of various backend services can be switched by setting these environment variables:

Expand All @@ -204,15 +204,21 @@ The versions of various backend services can be switched by setting these enviro
| `ORACLE_VERSION` | `23.5.0.0` | Version of Oracle container image to use |
| `POSTGRESQL_VERSION` | `14` | Version of PostgreSQL container image to use |
| `POSTGIS_VERSION` | `3.1` | Version of PostGIS extension to use |
| `SQLITE_VERSION` | | Version of SQLite to compile and use |

> [!NOTE]
>
> If left unspecified, the SQLite version provided by Debian will be used.
> Using a specific SQLite version requires compiling it from source. For more
> details, see [SQLite Versions](#SQLite-Versions).

### Python Versions

The `PYTHON_VERSION` environment variable controls which version of Python you
are running the tests against, e.g.

```console
$ PYTHON_VERSION=3.10 docker compose run --rm sqlite
$ PYTHON_VERSION=3.12 docker compose run --rm sqlite
```

In addition, it's possible to select a different implementation of Python, i.e.
Expand All @@ -229,7 +235,8 @@ restrictions with respect to the range of versions available.
### Database Versions

Most database container images are pulled from [Docker Hub][2]. Oracle database
is pulled from the [Oracle Container Registry][3].
is pulled from the [Oracle Container Registry][3]. Specific versions of SQLite
are compiled directly from the tags in the [official Git mirror][11].

You can switch the version of the database you test against by changing the
appropriate environment variable. Available options and their defaults can be
Expand Down Expand Up @@ -273,6 +280,46 @@ To determine what database versions can be used you can check the release notes
for the branch of Django that you have checked out, or alternatively there is
the [supported database versions][4] page on Django's Trac Wiki.

#### SQLite Versions

SQLite is normally bundled in the Python installation using the version
available on the system where Python is compiled. We use the Python Docker image
based on Debian `bookworm`, which has SQLite 3.40.1.
ngnpope marked this conversation as resolved.
Show resolved Hide resolved

To use a different version, we compile SQLite from source and load the library
dynamically using `LD_PRELOAD`. There are a few caveats as a result:

- Some SQLite features are only available if certain flags are set during
compilation. SQLite is known to change these flags in newer releases, such as
to enable features by default that were previously opt-in. When Python is
compiled, it inspects the system's SQLite to determine features that are
included in the `sqlite` module. A mismatch in the module and the dynamically
loaded library may result in Python failing to load, which may happen if we
use an SQLite version that is older than the system version.
- Debian and Ubuntu use a custom `CFLAGS` variable to compile their distributed
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQLite. Historically, Django's CI has only been configured with SQLite
versions that come with the operating system. If SQLite is compiled with
different flags, some tests may fail.

We currently work around the above caveats by setting the simplest `CFLAGS`
value that allows all the tests to pass. To customize the `CFLAGS` used for the
compilation, you can set the `SQLITE_CFLAGS` environment variable. See the
[`.env`][10] file for its default value.

```
SQLITE_VERSION=3.48.0 SQLITE_CFLAGS="-DSQLITE_OMIT_JSON -DSQLITE_MAX_VARIABLE_NUMBER=999" docker compose run --build --rm sqlite
```

> [!NOTE]
>
> The `--build` argument is necessary if you've changed `SQLITE_CFLAGS` since
> the last run, as it's not part of the image tag. You can also rebuild the
> image separately by running `docker compose build sqlite`, optionally with
> `--no-cache` to ignore the cached build.

In the future, the Django codebase may be more robust when tested against
different SQLite configurations and the `CFLAGS` workaround may no longer be
necessary.

### Other Versions

Expand Down Expand Up @@ -309,3 +356,5 @@ with no promises that they'll be delivered:
[7]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-unit-tests
[8]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-selenium-tests
[9]: https://docs.djangoproject.com/en/stable/ref/contrib/gis/testing/#geodjango-tests
[10]: .env
[11]: https://github.com/sqlite/sqlite
38 changes: 37 additions & 1 deletion compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ x-base: &base
args:
- PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION}
- PYTHON_VERSION=${PYTHON_VERSION}
additional_contexts:
additional_contexts: &additional-contexts
src: ${DJANGO_PATH:-../django}
volumes:
- ${DJANGO_PATH:-../django}:/django/source:rw
Expand Down Expand Up @@ -139,6 +139,10 @@ volumes:

services:

# Base service to allow building the image with `docker compose build base`.
base:
<<: *base

# Services: Databases

mariadb-db:
Expand Down Expand Up @@ -285,6 +289,38 @@ services:

sqlite:
<<: *base
image: django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}-sqlite${SQLITE_VERSION}
pull_policy: never
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't push the image to Docker Hub. This ensures doing FROM django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION} won't make Docker try to pull it from Docker Hub if the image isn't available locally.

Could also try build instead, but if I recall correctly it doesn't work in this context unless you already built the base image. Using build is even worse as it will run the build process even if the image has already been built. It does use the cache, but it also adds a few seconds to the run, compared to never that would just skip it.

Copy link
Member

@charettes charettes Feb 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there might be something off here. Here's what I get trying to build the image against Docker 4.38

$ PYTHON_VERSION=3.12 docker compose build sqlite
[+] Building 0.5s (3/3) FINISHED                                                                                                                                docker:desktop-linux
 => [sqlite internal] load build definition from Dockerfile                                                                                                                     0.0s
 => => transferring dockerfile: 734B                                                                                                                                            0.0s
 => ERROR [sqlite internal] load metadata for docker.io/library/django-docker-box:python-3.12                                                                                   0.4s
 => [sqlite auth] library/django-docker-box:pull token for registry-1.docker.io                                                                                                 0.0s
------
 > [sqlite internal] load metadata for docker.io/library/django-docker-box:python-3.12:
------
failed to solve: django-docker-box:python-3.12: failed to resolve source metadata for docker.io/library/django-docker-box:python-3.12: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh I see. I need to build the base first with PYTHON_VERSION=3.12 docker compose build base.

Might be worth documenting.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I investigated how we could define x-service image dependencies and I stumbled upon this page (which was deleted not too long along to encourage users to move to bake).

Unfortunately the solution they propose doesn't work when the image that you depend upon is parametrized

diff --git a/compose.yml b/compose.yml
index 301fb3e..e5d99f6 100644
--- a/compose.yml
+++ b/compose.yml
@@ -320,7 +320,9 @@ services:
         - PYTHON_VERSION=${PYTHON_VERSION}
         - SQLITE_VERSION=${SQLITE_VERSION}
         - SQLITE_CFLAGS=${SQLITE_CFLAGS}
-      additional_contexts: *additional-contexts
+      additional_contexts:
+        <<: *additional-contexts
+        "django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}": "service:base"
     depends_on:
       <<: *depends-on-caches
     environment:

The only solution I could find was to use depends_on to ensure the image was successfully

diff --git a/compose.yml b/compose.yml
index 301fb3e..10b4a03 100644
--- a/compose.yml
+++ b/compose.yml
@@ -142,6 +142,7 @@ services:
   # Base service to allow building the image with `docker compose build base`.
   base:
     <<: *base
+    command: --help

   # Services: Databases

@@ -323,6 +324,8 @@ services:
       additional_contexts: *additional-contexts
     depends_on:
       <<: *depends-on-caches
+      base:
+        condition: service_completed_successfully
     environment:
       - DATABASE_ENGINE=django.db.backends.sqlite3

Copy link
Member

@charettes charettes Feb 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is also mentioned in these docs

additional_contexts can also refer to an image built by another service. This allows a service image to be built using another service image as a base image, and to share layers between service images.

But I can't get it work either with

diff --git a/compose.yml b/compose.yml
index 301fb3e..9f761fb 100644
--- a/compose.yml
+++ b/compose.yml
@@ -294,7 +294,7 @@ services:
     build:
       context: .
       dockerfile_inline: |
-        FROM django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}
+        FROM base
         SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-o", "xtrace", "-c"]
         # Only compile SQLite and set LD_PRELOAD if a version is specified.
         RUN <<EOF
@@ -320,7 +320,9 @@ services:
         - PYTHON_VERSION=${PYTHON_VERSION}
         - SQLITE_VERSION=${SQLITE_VERSION}
         - SQLITE_CFLAGS=${SQLITE_CFLAGS}
-      additional_contexts: *additional-contexts
+      additional_contexts:
+        <<: *additional-contexts
+        base: service:base
     depends_on:
       <<: *depends-on-caches
     environment:

It results in

[+] Building 0.0s (0/0)                                                                                                                                         docker:desktop-linux
failed to get build context base: stat /path/to/django-docker-box/service:base: no such file or directory

Which is unsurprising as even their minimal example fails to run (when fixing their duplicate base service name 🤦 )

services:
 base:
  build:
    context: .
    dockerfile_inline: |
      FROM alpine
      RUN ...
 other: -- This was base wrongly set to base in their example
  build:
    context: .
    dockerfile_inline: |
      FROM base # image built for service base
      RUN ...
    additional_contexts:
      base: service:base
$ docker compose build service
[+] Building 0.0s (0/0)                                                                                                                                         docker:desktop-linux
failed to get build context base: stat /tmp/service:base: no such file or directory

build:
context: .
dockerfile_inline: |
FROM django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}
SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-o", "xtrace", "-c"]
# Only compile SQLite and set LD_PRELOAD if a version is specified.
RUN <<EOF
if [[ "${SQLITE_VERSION}" ]]; then
export CFLAGS="${SQLITE_CFLAGS}"
git clone --depth 1 --branch version-${SQLITE_VERSION} \
https://github.com/sqlite/sqlite.git /tmp/sqlite
cd /tmp/sqlite
./configure
make
if [ -f libsqlite3.so ]; then
cp libsqlite3.so /tmp/
else
cp .libs/libsqlite3.so /tmp/
fi
rm -rf /tmp/sqlite
Comment on lines +308 to +313
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "configure" script underwent major refactoring in 3.48.0: https://www.sqlite.org/releaselog/3_48_0.html

As a result, the compiled library is placed directly in the source root. In older versions, it's placed inside a .libs directory.

The rm -rf is just a cleanup step to keep the image size small.

fi
EOF
SHELL ["/bin/bash", "-c"]
ENV LD_PRELOAD=${SQLITE_VERSION:+/tmp/libsqlite3.so}
args:
- PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION}
- PYTHON_VERSION=${PYTHON_VERSION}
- SQLITE_VERSION=${SQLITE_VERSION}
- SQLITE_CFLAGS=${SQLITE_CFLAGS}
additional_contexts: *additional-contexts
depends_on:
<<: *depends-on-caches
environment:
Expand Down
1 change: 1 addition & 0 deletions packages.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,5 @@ libpq-dev
libproj-dev
libsqlite3-mod-spatialite
pkg-config
tcl-dev
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires re-building base first otherwise you'll run into /bin/sh: 1: tclsh: not found.

unzip