Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Copying tar file in container using deployment+service.rootless.yaml fails. #5244

Open
anthony-zawacki opened this issue Aug 12, 2024 · 2 comments
Open

Copying tar file in container using deployment+service.rootless.yaml fails. #5244

anthony-zawacki opened this issue Aug 12, 2024 · 2 comments

Comments

@anthony-zawacki
Copy link

anthony-zawacki commented Aug 12, 2024 •
edited
Loading

I'm running buildkit in a container in EKS based upon https://github.com/moby/buildkit/blob/master/examples/kubernetes/deployment%2Bservice.rootless.yaml using copacetic in a Jenkins build pipeline.

Running copa, the logs show:

+ copa patch -r trivy.json --debug --timeout 20m -i 948564350653.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 -t 25.0.1 -a tcp://buildkitd:1234
time="2024-08-12T16:31:54Z" level=debug msg="updates to apply: &{{{redhat 9.4} {amd64}} [{cups-libs 1:2.3.3op2-24.el9 1:2.3.3op2-27.el9_4 CVE-2024-35235} {java-21-openjdk 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21147} {java-21-openjdk 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21131} {java-21-openjdk 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21138} {java-21-openjdk 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21140} {java-21-openjdk 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21145} {java-21-openjdk-devel 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21147} {java-21-openjdk-devel 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21131} {java-21-openjdk-devel 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21138} {java-21-openjdk-devel 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21140} {java-21-openjdk-devel 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21145} {java-21-openjdk-headless 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21147} {java-21-openjdk-headless 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21131} {java-21-openjdk-headless 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21138} {java-21-openjdk-headless 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21140} {java-21-openjdk-headless 1:21.0.3.0.9-1.el9 1:21.0.4.0.7-1.el9 CVE-2024-21145} {python-unversioned-command 3.9.18-3.el9_4.1 3.9.18-3.el9_4.3 CVE-2024-4032} {python3 3.9.18-3.el9_4.1 3.9.18-3.el9_4.3 CVE-2024-4032} {python3-libs 3.9.18-3.el9_4.1 3.9.18-3.el9_4.3 CVE-2024-4032}]}"
time="2024-08-12T16:31:54Z" level=warning msg="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
time="2024-08-12T16:31:54Z" level=debug msg="serving grpc connection"
#1 resolve image config for docker-image://948564350653.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
#1 DONE 0.6s
time="2024-08-12T16:31:55Z" level=debug msg="latest unique RPMs: [{java-21-openjdk-headless  1:21.0.4.0.7-1.el9 } {python-unversioned-command  3.9.18-3.el9_4.3 } {python3  3.9.18-3.el9_4.3 } {python3-libs  3.9.18-3.el9_4.3 } {cups-libs  1:2.3.3op2-27.el9_4 } {java-21-openjdk  1:21.0.4.0.7-1.el9 } {java-21-openjdk-devel  1:21.0.4.0.7-1.el9 }]"
...
time="2024-08-12T16:37:29Z" level=debug msg="Required updates: [{cups-libs  1:2.3.3op2-27.el9_4 } {java-21-openjdk  1:21.0.4.0.7-1.el9 } {java-21-openjdk-devel  1:21.0.4.0.7-1.el9 } {java-21-openjdk-headless  1:21.0.4.0.7-1.el9 } {python-unversioned-command  3.9.18-3.el9_4.3 } {python3  3.9.18-3.el9_4.3 } {python3-libs  3.9.18-3.el9_4.3 }]"
time="2024-08-12T16:37:29Z" level=debug msg="Resulting updates: [cups-libs\t2.3.3op2-27.el9_4\tx86_64 java-21-openjdk\t21.0.4.0.7-1.el9\tx86_64 java-21-openjdk-devel\t21.0.4.0.7-1.el9\tx86_64 java-21-openjdk-headless\t21.0.4.0.7-1.el9\tx86_64 python-unversioned-command\t3.9.18-3.el9_4.3\tnoarch python3\t3.9.18-3.el9_4.3\tx86_64 python3-libs\t3.9.18-3.el9_4.3\tx86_64]"
time="2024-08-12T16:37:29Z" level=info msg="Validated package cups-libs version 2.3.3op2-27.el9_4 meets requested version 1:2.3.3op2-27.el9_4"
time="2024-08-12T16:37:29Z" level=info msg="Validated package java-21-openjdk version 21.0.4.0.7-1.el9 meets requested version 1:21.0.4.0.7-1.el9"
time="2024-08-12T16:37:29Z" level=info msg="Validated package java-21-openjdk-devel version 21.0.4.0.7-1.el9 meets requested version 1:21.0.4.0.7-1.el9"
time="2024-08-12T16:37:29Z" level=info msg="Validated package java-21-openjdk-headless version 21.0.4.0.7-1.el9 meets requested version 1:21.0.4.0.7-1.el9"
time="2024-08-12T16:37:29Z" level=info msg="Validated package python-unversioned-command version 3.9.18-3.el9_4.3 meets requested version 3.9.18-3.el9_4.3"
time="2024-08-12T16:37:29Z" level=info msg="Validated package python3 version 3.9.18-3.el9_4.3 meets requested version 3.9.18-3.el9_4.3"
time="2024-08-12T16:37:29Z" level=info msg="Validated package python3-libs version 3.9.18-3.el9_4.3 meets requested version 3.9.18-3.el9_4.3"

#14 diff (docker-image://948564350653.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0) -> (sh -c /usr/bin/microdnf update java-21-openjdk-headless python-unversioned-command python3 python3-libs cups-libs java-21-openjdk java-21-openjdk-devel -y && /usr/bin/microdnf clean all)
#14 DONE 0.0s

#15 merge (docker-image://948564350653.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0, diff (docker-image://948564350653.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0) -> (sh -c /usr/bin/microdnf update java-21-openjdk-headless python-unversioned-command python3 python3-libs cups-libs java-21-openjdk java-21-openjdk-devel -y && /usr/bin/microdnf clean all))
#15 DONE 0.0s

#16 exporting to docker image format
#16 exporting layers
#16 exporting layers 47.2s done
time="2024-08-12T16:38:16Z" level=debug msg="stopping session"
#16 exporting manifest sha256:8e58e58176d71d0116bee7bddcfe0259e9bbeb9ed2af1ff0b4405c703d7cc48e 0.0s done
#16 exporting config sha256:e2bcb88f064cdd96bb31efbd95a5a1fba63b5550b67e5131155f2d77eb445d77 done
#16 sending tarball
time="2024-08-12T16:51:55Z" level=error msg="patch exceeded timeout 20m0s"
Error: patch exceeded timeout 20m0s

Running buildkit with --debug --trace shows:

...
time="2024-08-12T16:38:16Z" level=trace msg="acquired cache ref" id=68y8atwokjljz3egurqs3u5vl mutable=false newRefCount=5 refID=0xc0008af680 stack="goroutine 2229 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/moby/buildkit/util/bklog.TraceLevelOnlyStack(...)\n\t/s
rc/util/bklog/log.go:72\ngithub.com/moby/buildkit/cache.(*immutableRef).traceLogFields(0xc0008af680)\n\t/src/cache/refs.go:492 +0x210\ngithub.com/moby/buildkit/cache.(*cacheRecord).ref(0xc0008bcb40, 0x0, 0x0, {0x0, 0x0})\n\t/src/cache/refs.go:113 +0xf2\ngithub.com/moby/buildkit/cache.(*cacheManager).get(0xc000aa6dc0
?, {0x1e5de40?, 0xc000838690?}, {0xc0008336db, 0x19}, {0x0, 0x0}, {0xc00090ccc0, 0x1, 0x1})\n\t/src/cache/manager.go:381 +0x385\ngithub.com/moby/buildkit/cache.(*cacheManager).Get(0xc000901f20?, {0x1e5de40?, 0xc000838690?}, {0xc0008336db?, 0xc000aa6e80?}, {0x0?, 0x0?}, {0xc00090ccc0, 0x1, 0x1})\n\t/src/cache/manager
.go:350 +0x165\ngithub.com/moby/buildkit/worker/base.(*Worker).LoadRef(0xc0000ddce0, {0x1e5de40, 0xc000838690}, {0xc0008336db, 0x19}, 0xf8?)\n\t/src/worker/base/worker.go:288 +0x168\ngithub.com/moby/buildkit/worker.(*cacheResultStorage).LoadRemotes(0x19?, {0x1e5de40, 0xc000838690}, {{0x1c24bbae, 0xede4c3133, 0x0}, {
0xc0008336c0, 0x34}}, 0x0, {0x0, ...})\n\t/src/worker/cacheresult.go:75 +0xe2\ngithub.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0010b83c0, {0x1e5de40, 0xc000838690}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/exporter.go:120 +0x55e\ngithub.com/moby/buildkit/solver.(*merg
edExporter).ExportTo(0xede4c3149?, {0x1e5de40, 0xc000838690}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/exporter.go:246 +0x13c\ngithub.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000534ac0, {0x1e5de40, 0xc000838690}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0},
 0x0, ...})\n\t/src/solver/exporter.go:173 +0xdf7\ngithub.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xede4c3149?, {0x1e5de40, 0xc000838690}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/exporter.go:246 +0x13c\ngithub.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000e
51340, {0x1e5de40, 0xc0008385d0}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/exporter.go:173 +0xdf7\ngithub.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xc0004d5cb8?, {0x1e5de40, 0xc0008385d0}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/s
olver/exporter.go:246 +0x13c\ngithub.com/moby/buildkit/solver/llbsolver.NewProvenanceCreator.func1()\n\t/src/solver/llbsolver/provenance.go:413 +0x192\ngithub.com/moby/buildkit/solver/llbsolver.(*ProvenanceCreator).Predicate(0xc000901300)\n\t/src/solver/llbsolver/provenance.go:463 +0xa2\ngithub.com/moby/buildkit/sol
ver/llbsolver.(*Solver).recordBuildHistory.func1.1({0x1b8fb4b?, 0x1?}, {0x1e61640, 0xc0008c5800}, 0xc00049a6c0)\n\t/src/solver/llbsolver/solver.go:237 +0x1d3\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1.2()\n\t/src/solver/llbsolver/solver.go:275 +0x5d\ngolang.org/x/sync/errgroup.(*Gr
oup).Go.func1()\n\t/src/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 130\n\t/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96\n"
time="2024-08-12T16:38:16Z" level=trace msg="released cache ref" id=68y8atwokjljz3egurqs3u5vl mutable=false newRefCount=4 refID=0xc0008af680 stack="goroutine 2229 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/moby/buildkit/util/bklog.TraceLevelOnlyStack(...)\n\t/s
rc/util/bklog/log.go:72\ngithub.com/moby/buildkit/cache.(*immutableRef).traceLogFields(0xc0008af680)\n\t/src/cache/refs.go:492 +0x210\ngithub.com/moby/buildkit/cache.(*immutableRef).release.func1()\n\t/src/cache/refs.go:1376 +0x55\ngithub.com/moby/buildkit/cache.(*immutableRef).release(0xc0008af680, {0x1e5dbf0, 0x2b
95ba0})\n\t/src/cache/refs.go:1402 +0x365\ngithub.com/moby/buildkit/cache.(*immutableRef).Release(0xc0008af680, {0x1e5dbf0, 0x2b95ba0})\n\t/src/cache/refs.go:1355 +0x10c\ngithub.com/moby/buildkit/worker.(*cacheResultStorage).LoadRemotes(0x19?, {0x1e5de40, 0xc000838690}, {{0x1c24bbae, 0xede4c3133, 0x0}, {0xc0008336c0
, 0x34}}, 0x0, {0x0, ...})\n\t/src/worker/cacheresult.go:96 +0x335\ngithub.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0010b83c0, {0x1e5de40, 0xc000838690}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/exporter.go:120 +0x55e\ngithub.com/moby/buildkit/solver.(*mergedExporter)
.ExportTo(0xede4c3149?, {0x1e5de40, 0xc000838690}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/exporter.go:246 +0x13c\ngithub.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000534ac0, {0x1e5de40, 0xc000838690}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})
\n\t/src/solver/exporter.go:173 +0xdf7\ngithub.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xede4c3149?, {0x1e5de40, 0xc000838690}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/exporter.go:246 +0x13c\ngithub.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000e51340, {0x1
e5de40, 0xc0008385d0}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/exporter.go:173 +0xdf7\ngithub.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xc0004d5cb8?, {0x1e5de40, 0xc0008385d0}, {0x1e5ad20, 0xc00087b0d0}, {0x1c2c788, 0x2, {0x0, 0x0}, 0x0, ...})\n\t/src/solver/expor
ter.go:246 +0x13c\ngithub.com/moby/buildkit/solver/llbsolver.NewProvenanceCreator.func1()\n\t/src/solver/llbsolver/provenance.go:413 +0x192\ngithub.com/moby/buildkit/solver/llbsolver.(*ProvenanceCreator).Predicate(0xc000901300)\n\t/src/solver/llbsolver/provenance.go:463 +0xa2\ngithub.com/moby/buildkit/solver/llbsolv
er.(*Solver).recordBuildHistory.func1.1({0x1b8fb4b?, 0x1?}, {0x1e61640, 0xc0008c5800}, 0xc00049a6c0)\n\t/src/solver/llbsolver/solver.go:237 +0x1d3\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1.2()\n\t/src/solver/llbsolver/solver.go:275 +0x5d\ngolang.org/x/sync/errgroup.(*Group).Go.fun
c1()\n\t/src/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 130\n\t/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96\n"
time="2024-08-12T16:38:16Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = failed to copy to tar: rpc error: code = Canceled desc = context canceled"
time="2024-08-12T16:38:16Z" level=trace msg="released cache ref" id=68xlrgwj3fh6ipi6g9rebas00 mutable=false newRefCount=1 refID=0xc0010ffc20 stack="goroutine 2347 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/moby/buildkit/util/bklog.TraceLevelOnlyStack(...)\n\t/src/util/bklog/log.go:72\ngithub.com/moby/buildkit/cache.(*immutableRef).traceLogFields(0xc0010ffc20)\n\t/src/cache/refs.go:492 +0x210\ngithub.com/moby/buildkit/cache.(*immutableRef).release.func1()\n\t/src/cache/refs.go:1376 +0x55\ngithub.com/moby/buildkit/cache.(*immutableRef).release(0xc0010ffc20, {0x1e5dbf0, 0x2b95ba0})\n\t/src/cache/refs.go:1402 +0x365\ngithub.com/moby/buildkit/cache.(*immutableRef).Release(0xc0010ffc20, {0x1e5dbf0, 0x2b95ba0})\n\t/src/cache/refs.go:1355 +0x10c\ngithub.com/moby/buildkit/worker.(*workerRefResult).Release(0x0?, {0x1e5dbf0?, 0x2b95ba0?})\n\t/src/worker/result.go:59 +0x32\ngithub.com/moby/buildkit/solver.(*splitResult).Release(0x2aa3220?, {0x1e5dbf0?, 0x2b95ba0?})\n\t/src/solver/result.go:56 +0x55\ngithub.com/moby/buildkit/solver.(*SharedResult).Release(0x16dae2a?, {0x1e5dbf0?, 0x2b95ba0?})\n\t/src/solver/result.go:35 +0x9f\ncreated by github.com/moby/buildkit/solver.(*sharedOp).release in goroutine 130\n\t/src/solver/jobs.go:1152 +0x55\n"
Canceled: failed to copy to tar: rpc error: code = Canceled desc = context canceled
30 add7328 buildkitd --debug --trace --addr unix:///run/user/1000/buildkit/buildkitd.sock --addr tcp://0.0.0.0:1234 --oci-worker-no-process-sandbox
github.com/moby/buildkit/session/filesync.(*streamWriterCloser).Write
        /src/session/filesync/diffcopy.go:65
bufio.(*Writer).Write
        /usr/local/go/src/bufio/bufio.go:679
archive/tar.(*regFileWriter).Write
        /usr/local/go/src/archive/tar/writer.go:539
archive/tar.(*Writer).Write
        /usr/local/go/src/archive/tar/writer.go:477
io.(*multiWriter).Write
        /usr/local/go/src/io/multi.go:85
io.copyBuffer
        /usr/local/go/src/io/io.go:431
io.Copy
        /usr/local/go/src/io/io.go:388
github.com/containerd/containerd/images/archive.blobRecord.func1
        /src/vendor/github.com/containerd/containerd/images/archive/exporter.go:419
github.com/containerd/containerd/images/archive.writeTar
        /src/vendor/github.com/containerd/containerd/images/archive/exporter.go:586
github.com/containerd/containerd/images/archive.Export
        /src/vendor/github.com/containerd/containerd/images/archive/exporter.go:354
github.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export
        /src/exporter/oci/export.go:267
github.com/moby/buildkit/solver/llbsolver.(*Solver).runExporters.func1.1
        /src/solver/llbsolver/solver.go:784
github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
        /src/solver/llbsolver/solver.go:1080
github.com/moby/buildkit/solver.(*Job).InContext
        /src/solver/jobs.go:808
github.com/moby/buildkit/solver/llbsolver.inBuilderContext
        /src/solver/llbsolver/solver.go:1076
github.com/moby/buildkit/solver/llbsolver.(*Solver).runExporters.func1
        /src/solver/llbsolver/solver.go:767
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:78
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695
30 add7328 buildkitd --debug --trace --addr unix:///run/user/1000/buildkit/buildkitd.sock --addr tcp://0.0.0.0:1234 --oci-worker-no-process-sandbox
main.unaryInterceptor
        /src/cmd/buildkitd/main.go:713
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
        /src/vendor/google.golang.org/grpc/server.go:1183
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
        /src/api/services/control/control.pb.go:2493
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1372
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1783
google.golang.org/grpc.(*Server).serveStreams.func2.1
        /src/vendor/google.golang.org/grpc/server.go:1016
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695

time="2024-08-12T16:38:16Z" level=trace msg="released cache ref" id=7babis8voeyqleb55a3cbxubh mutable=false newRefCount=2 refID=0xc001a28d50 stack="goroutine 2349 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/moby/buildkit/util/bklog.TraceLevelOnlyStack(...)\n\t/src/util/bklog/log.go:72\ngithub.com/moby/buildkit/cache.(*immutableRef).traceLogFields(0xc001a28d50)\n\t/src/cache/refs.go:492 +0x210\ngithub.com/moby/buildkit/cache.(*immutableRef).release.func1()\n\t/src/cache/refs.go:1376 +0x55\ngithub.com/moby/buildkit/cache.(*immutableRef).release(0xc001a28d50, {0x1e5dbf0, 0x2b95ba0})\n\t/src/cache/refs.go:1402 +0x365\ngithub.com/moby/buildkit/cache.(*immutableRef).Release(0xc001a28d50, {0x1e5dbf0, 0x2b95ba0})\n\t/src/cache/refs.go:1355 +0x10c\ngithub.com/moby/buildkit/worker.(*workerRefResult).Release(0x1e41580?, {0x1e5dbf0?, 0x2b95ba0?})\n\t/src/worker/result.go:59 +0x32\ngithub.com/moby/buildkit/solver.(*splitResult).Release(0x485042?, {0x1e5dbf0?, 0x2b95ba0?})\n\t/src/solver/result.go:56 +0x55\ngithub.com/moby/buildkit/solver.(*SharedResult).Release(0x16dae2a?, {0x1e5dbf0?, 0x2b95ba0?})\n\t/src/solver/result.go:35 +0x9f\ncreated by github.com/moby/buildkit/solver.(*sharedOp).release in goroutine 130\n\t/src/solver/jobs.go:1152 +0x55\n"
...

I have this block added to the deployment in case there is an issue in EKS similar to the issue described in #879 (comment) but the behavior didn't change. Additionally, I added workingDir: /home/user to see if that would help or not.

    volumeMounts:
    - mountPath: /home/user/.local/share/buildkit
      mountPropagation: None
      name: buildkitd
  volumes:
  - emptyDir: {}
    name: buildkitd

I'm not sure where buildkitd is attempting to write the tarball that is being sent to it, I'm guessing there is a permissions issue that can probably be fixed with another emptyDir, but I'm not sure where it needs to be mounted.
@tonistiigi
Copy link
Member

I guess report this to copacetic ?

@anthony-zawacki
Copy link
Author

I guess report this to copacetic ?

I worked with the copacetic people, the recommended that I talk to buildkitd people because it looked like a buildkitd error. They are sending the tar file, but buildkitd is erroring writing the file. As far as I can tell, it's not a copacetic issue, it's an issue when running buildkitd as deployment+service-rootless.

The error messages are what are logged in buildkitd. It looks like tar/writer.write is throwing an error. Again, I'm assuming it's a permissions issue, I just don't know where to look.

@anthony-zawacki anthony-zawacki mentioned this issue Sep 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tonistiigi @anthony-zawacki