Podman invocations using different --roots cannot share an --imagestore
#2257
Comments
|
I am not sure if imagestores are supposed to be shared between different podman instances, my understanding was that one uses cc @giuseppe |
|
that is also my understanding ... i think https://www.redhat.com/en/blog/image-stores-podman might be what you are after? |
|
I was taking a bit of a logical leap based on how
I am doubtful |
|
Another issue with Ex: podman --root /tmp/root1 pull mirror.gcr.io/library/bash
# pull again to the same root, its cached as expected
podman --root /tmp/root1 pull mirror.gcr.io/library/bash
# use a different root, with an additionalimagestore to the first root
# we download the image even though its present in the RO additionalimagestore :(
podman --root /tmp/root2 --storage-opt additionalimagestore=/tmp/root1 pull mirror.gcr.io/library/bashNot sure if that expected behavior. It seems quirky, but on the other hand if an image's existence in |
|
I've tried with Podman 5.3.2 and it works well for me, as the issue you've seen was fixed with defaae6 Can you please try with a newer version? |
|
I used the original reproducer on main yesterday and it failed. |
|
yeah sorry, I've tested only the last sequence of commands in #2257 The issue is that we write |
|
I think we need something like the following diff: diff --git a/layers.go b/layers.go
index 69a4887bf..ed10340fa 100644
--- a/layers.go
+++ b/layers.go
@@ -51,6 +51,7 @@ type layerLocations uint8
// unclean shutdown
const (
stableLayerLocation layerLocations = 1 << iota
+ imageStoreLayerLocation
volatileLayerLocation
numLayerLocationIndex = iota
@@ -167,6 +168,9 @@ type Layer struct {
// volatileStore is true if the container is from the volatile json file
volatileStore bool `json:"-"`
+ // imageStore is true if the layer is from the image store json file
+ imageStore bool `json:"-"`
+
// BigDataNames is a list of names of data items that we keep for the
// convenience of the caller. They can be large, and are only in
// memory when being read from or written to disk.
@@ -435,6 +439,9 @@ func layerLocation(l *Layer) layerLocations {
if l.volatileStore {
return volatileLayerLocation
}
+ if l.imageStore {
+ return imageStoreLayerLocation
+ }
return stableLayerLocation
}
@@ -456,6 +463,7 @@ func copyLayer(l *Layer) *Layer {
CompressionType: l.CompressionType,
ReadOnly: l.ReadOnly,
volatileStore: l.volatileStore,
+ imageStore: l.imageStore,
BigDataNames: copySlicePreferringNil(l.BigDataNames),
Flags: copyMapPreferringNil(l.Flags),
UIDMap: copySlicePreferringNil(l.UIDMap),
@@ -823,6 +831,9 @@ func (r *layerStore) load(lockedForWriting bool) (bool, error) {
if location == volatileLayerLocation {
layer.volatileStore = true
}
+ if location == imageStoreLayerLocation {
+ layer.imageStore = true
+ }
layers = append(layers, layer)
ids[layer.ID] = layer
}
@@ -1144,6 +1155,7 @@ func (s *store) newLayerStore(rundir, layerdir, imagedir string, driver drivers.
rundir: rundir,
jsonPath: [numLayerLocationIndex]string{
filepath.Join(layerdir, "layers.json"),
+ filepath.Join(imagedir, "layers.json"),
filepath.Join(volatileDir, "volatile-layers.json"),
},
layerdir: layerdir,
@@ -1180,6 +1192,7 @@ func newROLayerStore(rundir string, layerdir string, driver drivers.Driver) (roL
mountsLockfile: nil,
rundir: rundir,
jsonPath: [numLayerLocationIndex]string{
+ filepath.Join(layerdir, "layers.json"),
filepath.Join(layerdir, "layers.json"),
filepath.Join(layerdir, "volatile-layers.json"),
},
@@ -1422,6 +1435,7 @@ func (r *layerStore) create(id string, parentLayer *Layer, names []string, mount
GIDMap: copySlicePreferringNil(moreOptions.GIDMap),
BigDataNames: []string{},
volatileStore: moreOptions.Volatile,
+ imageStore: moreOptions.ImageStore,
}
layer.Flags[incompleteFlag] = true
diff --git a/store.go b/store.go
index 053c31c42..88e7e088e 100644
--- a/store.go
+++ b/store.go
@@ -667,6 +667,8 @@ type LayerOptions struct {
UncompressedDigest digest.Digest
// True is the layer info can be treated as volatile
Volatile bool
+ // True is the layer info must be written to the image store
+ ImageStore bool
// BigData is a set of items which should be stored with the layer.
BigData []LayerBigDataOption
// Flags is a set of named flags and their values to store with the layer.
@@ -1545,6 +1547,9 @@ func (s *store) putLayer(rlstore rwLayerStore, rlstores []roLayerStore, id, pare
GIDMap: copySlicePreferringNil(gidMap),
}
}
+ if !writeable && s.imageStoreDir != "" {
+ options.ImageStore = true
+ }
return rlstore.create(id, parentLayer, names, mountLabel, nil, &options, writeable, diff, slo)
}
|
when an imagestore is used, a R/O layer must be written to the
layers.json under the imagestore, not graphroot.
The lock on the imagestore is already taken through the
multipleLockFile{} mechanism in place.
Closes: containers#2257
Signed-off-by: Giuseppe Scrivano <[email protected]>
|
opened a PR: #2258 |
|
Thanks for looking into it! I do have a bit of a hacky workaround in the meantime that anyone else in the same boat may be interested in. I populate WorkaroundStep 1. Give all the runners an Step 2. On job_completion, find all images that exist, and copy them into the shared storage by finagling images=$(podman images --format json | jq -c 'map(select((.Names | length > 0))) | group_by(.Id) | map({Id: .[0].Id, Names: [.[] | .Names] | flatten | unique }) | .[]')
pull_cache="/home/runner/.local/share/podman-pull-storage"
for image in $images
do
id=$(jq -r '.Id' <<< "$image")
names=$(jq -r '.Names | join(" ")' <<< "$image")
echo "attempting to cache image with id: $id"
if podman --root $pull_cache image exists $id; then
echo " image already exists in the cache"
for name in $names
do
if podman --root $pull_cache image inspect $id | jq --exit-status --arg imageName $name 'map(select(.RepoTags[] | contains($imageName))) | length == 0' > /dev/null; then
echo " adding missing tag to cache: $name"
podman --root $pull_cache tag $id $name
fi
done
else
echo " image not found in cache, writing it with tags: $names"
podman save $id $names | podman --root $pull_cache load
fi
doneStep 3. On subsequent runs, images will be pulled from the additionalimagestores if they exist (assuming v5.3.2) |
when an imagestore is used, a R/O layer must be written to the
layers.json under the imagestore, not graphroot.
The lock on the imagestore is already taken through the
multipleLockFile{} mechanism in place.
Closes: containers#2257
Signed-off-by: Giuseppe Scrivano <[email protected]>
when an imagestore is used, a R/O layer must be written to the
layers.json under the imagestore, not graphroot.
The lock on the imagestore is already taken through the
multipleLockFile{} mechanism in place.
Closes: containers#2257
Signed-off-by: Giuseppe Scrivano <[email protected]>
when an imagestore is used, a R/O layer must be written to the
layers.json under the imagestore, not graphroot.
The lock on the imagestore is already taken through the
multipleLockFile{} mechanism in place.
Closes: containers#2257
Signed-off-by: Giuseppe Scrivano <[email protected]>
when an imagestore is used, a R/O layer must be written to the
layers.json under the imagestore, not graphroot.
The lock on the imagestore is already taken through the
multipleLockFile{} mechanism in place.
Closes: containers#2257
Signed-off-by: Giuseppe Scrivano <[email protected]>
Issue Description
I suspect this was never an intended feature, but our use case is CI runners. We have hundreds of replicas, each of which has podman installed. Jobs always start with container pulls (which is the slowest part of the job). If a job pulls an image, we'd like for it to be written to a shared imagestore for subsequent jobs to reuse.
I created a
imgsvolume on the host, which I mounted into all of the runners. Then, all the runners were given a storage.conf with the contents:My expectation was that any podman pull inside of a runner would read/write this shared imagestore. However, the pulls step on each other, corrupting the imagestore/root.
Steps to reproduce the issue
Describe the results you received
Corrupted imagestore
Describe the results you expected
Non-corrupted imagestore
podman info output
Podman in a container
Yes
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: