docker-container build driver doesn't use cache when building already built image without internet (i/o timeout)

[stasadev]

When I have an image already built, I expect it to be built again (using cache), regardless of whether I have internet or not.

This works with the docker build driver, but does not work with the docker-container driver, which tries to pull the image, resulting in an i/o timeout error:

ERROR: failed to solve: busybox: failed to resolve source metadata for docker.io/library/busybox:latest: failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/latest": dial tcp: lookup registry-1.docker.io on 8.8.4.4:53: read udp 172.17.0.2:52305->8.8.4.4:53: i/o timeout

I'm using Manjaro Linux:

$ docker info
Client:
 Version:    27.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.16.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.29.1
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 50
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.103-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.19GiB
 Name: aspire
 ID: GG2L:B3IQ:5Y6L:42AW:GAF4:5CCH:E5P6:CWVU:XY6C:Q6HU:YOD3:4UBV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

To reproduce:

# start with "docker" DRIVER
$ docker buildx use default

$ docker buildx inspect
Name:          default
Driver:        docker
Last Activity: 2024-08-05 10:27:58 +0000 UTC

Nodes:
Name:             default
Endpoint:         default
Status:           running
BuildKit version: v0.15.0
Platforms:        linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1

# I will build "FROM busybox" with and without internet
$ docker pull busybox

# with internet
$ echo "FROM busybox" | docker buildx build -
[+] Building 0.1s (5/5) FINISHED                                 docker:default
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 50B                                        0.0s
 => [internal] load metadata for docker.io/library/busybox:latest          0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => CACHED [1/1] FROM docker.io/library/busybox:latest                     0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:138ec1bf8127895542cfd7d054b7fffaf9a5681e58b8b  0.0s

# without internet
$ echo "FROM busybox" | docker buildx build -
[+] Building 0.0s (5/5) FINISHED                                 docker:default
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 50B                                        0.0s
 => [internal] load metadata for docker.io/library/busybox:latest          0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => CACHED [1/1] FROM docker.io/library/busybox:latest                     0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:138ec1bf8127895542cfd7d054b7fffaf9a5681e58b8b  0.0s

# switch to "docker-container" DRIVER
$ docker buildx create --name test --use
test

# with internet
$ echo "FROM busybox" | docker buildx build -
[+] Building 2.3s (4/4) FINISHED                          docker-container:test
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 50B                                        0.0s
 => [internal] load metadata for docker.io/library/busybox:latest          2.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [1/1] FROM docker.io/library/busybox:latest@sha256:9ae97d36d26566ff84  0.0s
 => => resolve docker.io/library/busybox:latest@sha256:9ae97d36d26566ff84  0.0s
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

# without internet
$ echo "FROM busybox" | docker buildx build -
[+] Building 20.1s (2/2) FINISHED                         docker-container:test
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 50B                                        0.0s
 => ERROR [internal] load metadata for docker.io/library/busybox:latest   20.0s
------
 > [internal] load metadata for docker.io/library/busybox:latest:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:1
--------------------
   1 | >>> FROM busybox
   2 |     
--------------------
ERROR: failed to solve: busybox: failed to resolve source metadata for docker.io/library/busybox:latest: failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/latest": dial tcp: lookup registry-1.docker.io on 8.8.4.4:53: read udp 172.17.0.2:52305->8.8.4.4:53: i/o timeout

$ docker buildx inspect
Name:          test
Driver:        docker-container
Last Activity: 2024-08-05 10:35:30 +0000 UTC

Nodes:
Name:                  test0
Endpoint:              unix:///var/run/docker.sock
Status:                running
BuildKit daemon flags: --allow-insecure-entitlement=network.host
BuildKit version:      v0.15.1
Platforms:             linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.executor:         oci
 org.mobyproject.buildkit.worker.hostname:         9388b4e05ef4
 org.mobyproject.buildkit.worker.network:          host
 org.mobyproject.buildkit.worker.oci.process-mode: sandbox
 org.mobyproject.buildkit.worker.selinux.enabled:  false
 org.mobyproject.buildkit.worker.snapshotter:      overlayfs
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
 Keep Bytes:    488.3MiB
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    14.9GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 14.9GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 14.9GiB

# switch back
$ docker buildx use default
$ docker buildx rm test
test removed