exec format error from linux/arm64 image

[thirdgen88]

What steps did you take and what happened:
I observed an exec /manager: exec format error while setting up my cluster-api infrastructure provider (Proxmox as it happens).

I inspected the manifest (using crane) and it shows a multi-arch image index:

$ crane manifest registry.k8s.io/capi-ipam-ic/cluster-api-ipam-in-cluster-controller:v1.0.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3023,
         "digest": "sha256:c21c92fecbd090737eace56b2df0f8e332599929e85d81574b223eb2f5c05d34",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3023,
         "digest": "sha256:6681b5131e4520d3edbe18b76a7dfa7329a6b13789dba00bf1a44a52a16691c0",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      }
   ]
}

However, upon inspecting each of those, the layer for manager executable (invoked as the entrypoint) is the same for both amd64 and arm64:

# AMD64
$ crane manifest registry.k8s.io/capi-ipam-ic/cluster-api-ipam-in-cluster-controller:v1.0.0@sha256:c21c92fecbd090737eace56b2df0f8e332599929e85d81574b223eb2f5c05d34
{
...
   "layers": [
...
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 25408896,
         "digest": "sha256:4a844c334ba3a146da89c4e3ebd301dd88e1290cb4c361c715c45e069f11d976"
      },
...
   ]
}
# ARM64
$ crane manifest registry.k8s.io/capi-ipam-ic/cluster-api-ipam-in-cluster-controller:v1.0.0@sha256:6681b5131e4520d3edbe18b76a7dfa7329a6b13789dba00bf1a44a52a16691c0
{
...
   "layers": [
...
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 25408896,
         "digest": "sha256:4a844c334ba3a146da89c4e3ebd301dd88e1290cb4c361c715c45e069f11d976"
      },
...
   ]
}

Observe the same digest there. Since I'm running this on arm64 (and getting the error), I'm assuming it is likely amd64 (especially considering the GH runner that is doing the build).

What did you expect to happen:
Other than not encountering the exec format error in general, I expected the executable to naturally be different between the AMD64/ARM64 images.

Anything else you would like to add:
I believe I've seen this behavior in a multi-arch build if the supporting QEMU emulation for the builder is not setup. The painful part is that I believe it will just silently proceed building AMD64 even though you specify the platform as linux/arm64 (again, because the backing emulation isn't available). Originally I thought GH Actions was being used to conduct the build (in which case there are some additional actions to make sure that buildkit and qemu are configured on the runner), but that doesn't appear to be the case. I'm unsure of whether additional provisions might be needed in cloudbuild.yaml, I've not worked with that before.

Environment:

• Cluster-api-ipam-provider-in-cluster version: v1.0.0
• Cluster-api version: v1.9.4
• Kubernetes version: (use kubectl version): v1.31.2+k3s1
• OS (e.g. from /etc/os-release): Ubuntu 24.04.1 LTS

/kind bug

[thirdgen88]

#285 is the only other issue that I could find to link here.