Multiplatform builds do not always use buildx

[jgulotta]

TLDR: publishTask first triggers publishLocalTask which runs docker build even for multiplatform builds that require docker buildx build; docker buildx build is only triggered in the publishTask later. I'd guess that dockerBuildCommand should check dockerBuildxPlatforms to select between build and buildx build such that publishLocal issues a single build, and publish just pushes the tags from the local build instead of triggering a new build.

Multiplatform builds require using buildx with the docker-container driver, which I am doing

docker buildx create --name multiarch --driver docker-container --use

I have modified stage0 to include a layer that builds a binary (not the easiest thing since #1417 is open and #1437 was closed without comment). This layer is now a slow process and so I want the layer cached. Because this is only in stage0 the layer is not included in the final docker image publish. The way to solve this is to add --cache-to and --cache-from such that even intermediate layers get cached and reused

dockerBuildOptions ++= {
  val registry = s"${dockerRepo.value}/${name.value}-build-cache"
  Seq(
    "--cache-to", s"type=registry,ref=$registry,mode=max",
    "--cache-from", s"type=registry,ref=$registry",
  )
}

Running this setup results in an error

[error] ERROR: cache export feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
[info] Removing intermediate image(s) (labeled "snp-multi-stage-id=739842cd-6d16-4c12-8195-67def738cf32") 
[info] Total reclaimed space: 0B
[error] java.lang.RuntimeException: Nonzero exit value: 1
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.publishLocalDocker(DockerPlugin.scala:691)
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.$anonfun$projectSettings$42(DockerPlugin.scala:267)
[error] 	at com.typesafe.sbt.packager.docker.DockerPlugin$.$anonfun$projectSettings$42$adapted(DockerPlugin.scala:259)

This is unexpected and means publishLocalDocker is using the docker driver and not using buildx with the docker-container driver I set.

Indeed, publishLocalDocker is called in publishLocalTask with

        publishLocalDocker(
          stage.value,
          dockerBuildCommand.value,
          dockerExecCommand.value,
          dockerPermissionStrategy.value,
          dockerAutoremoveMultiStageIntermediateImages.value,
          log
        )

And publishLocalTask is invoked as the first step in publishTask with val _ = publishLocal.value. At this point, dockerBuildCommand is docker build ... and dockerExecCommand is docker. Only later in publishTask does it check for a multiplatform build and alter the exec command to be docker buildx build ... --push.

This results in a docker build for the host architecture using the docker driver, followed by a docker buildx build for any configured platforms. In the presence of --cache-from and --cache-to, the initial docker build fails. If the host architecture matches a configured cross-platform, the docker buildx build will reuse those layers, and if not then the docker build is wasted work.

This can be worked around by resetting dockerBuildCommand:

dockerBuildCommand := dockerExecCommand.value ++ (if (dockerBuildxPlatforms.value.isEmpty) { Seq("build") } else { Seq("buildx", "build",  s"--platform=${dockerBuildxPlatforms.value.mkString(",")) }) ++ dockerBuildOptions.value ++ Seq(".")

It is only a workaround since buildx build gets invoked twice, but at least the second time is already cached locally.

Expected behaviour

Multiplatform builds consistently use docker buildx build and only build once

Actual behaviour

Multiplatform builds start with a docker build and later issue a docker buildx build

Information

• What sbt-native-packager are you using: 1.9.16
• What sbt version: 1.8.3
• What is your build system: Debian amd64
• What package are you building: docker
• What version has your build tool: Docker 24.0.2
• What is your target system: Debian amd64, Debian arm64