401 Unauthorized pulling from private registry

[anthony-zawacki]

Issue #16 is closed indicating that public/private registries are supported, but project-copacetic/copa-action@main fails to be able to pull from an ECR registry. I've logged into the registry every way in the github action that I could think to do it. As far as I understand, there are no parameters to the buildx container that runs, so I'm not sure how it's supposed to authenticate with the registry.

The documentation at https://github.com/project-copacetic/copa-action?tab=readme-ov-file#ways-to-connect-to-buildkit seems to indicate buildx (Option 1) is the only supported way to connect to private registries.

Selected github action logs:
Example of being able to pull the image locally, can be done as a regular user or as super use, both are successful:

Run sudo docker pull ${REGISTRY}/${REPOSITORY}:${TAG}-0
  sudo docker pull ${REGISTRY}/${REPOSITORY}:${TAG}-0
  shell: /usr/bin/bash -e {0}
  env:
    TAG: [2](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:2)5.0.1
    REGISTRY: ***.dkr.ecr.us-east-1.amazonaws.com
    REPOSITORY: ironbank/opensource/keycloak/keycloak
    AWS_DEFAULT_REGION: us-east-1
    AWS_REGION: us-east-1
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
25.0.1-0: Pulling from ironbank/opensource/keycloak/keycloak
e[3](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:3)94ea8406c7: Pulling fs layer
0a[4](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:4)84c292e0f: Pulling fs layer
e2bc109f1f18: Pulling fs layer
f32fd4c317d8: Pulling fs layer
f32fd4c317d8: Waiting
0a484c292e0f: Verifying Checksum
0a484c292e0f: Download complete
e394ea8406c7: Verifying Checksum
e394ea8406c7: Download complete
e394ea8406c7: Pull complete
0a484c292e0f: Pull complete
e2bc109f1f18: Verifying Checksum
e2bc109f1f18: Download complete
f32fd4c317d8: Verifying Checksum
f32fd4c317d8: Download complete
e2bc109f1f18: Pull complete
f32fd4c317d8: Pull complete
Digest: sha2[5](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:5)6:a030889378d9a2c981f245ec35974[6](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:6)37b431796b84e0afca4693c7008e3f3df6
Status: Downloaded newer image for ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0

Running the copa action fails with 401 unauthorized:

Run project-copacetic/copa-action@main
  with:
    image: ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:[2](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:2)5.0.1-0
    image-report: report.json
    patched-tag: patched
    timeout: 5m
    output: vex.json
    format: openvex
  env:
    TAG: 25.0.1
    REGISTRY: ***.dkr.ecr.us-east-1.amazonaws.com
    REPOSITORY: ironbank/opensource/keycloak/keycloak
    AWS_DEFAULT_REGION: us-east-1
    AWS_REGION: us-east-1
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
Run # check for copa version input, else use latest
Unable to find image 'ghcr.io/project-copacetic/copa-action:v0.7.0' locally
v0.7.0: Pulling from project-copacetic/copa-action
f11c1adaa26e: Pulling fs layer
c802ad29fa74: Pulling fs layer
fd6e[3](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:3)8fed636: Pulling fs layer
d3d5f8ca516f: Pulling fs layer
d3d5f8ca516f: Waiting
c802ad29fa7[4](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:4): Verifying Checksum
c802ad29fa74: Download complete
f11c1adaa26e: Verifying Checksum
f11c1adaa26e: Download complete
d3d[5](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:5)f8ca516f: Verifying Checksum
d3d5f8ca51[6](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:6)f: Download complete
f11c1adaa26e: Pull complete
c802ad29fa[7](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:7)4: Pull complete
fd6e3[8](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:8)fed636: Verifying Checksum
fd6e38fed636: Download complete
fd6e38fed636: Pull complete
d3d5f8ca516f: Pull complete
Digest: sha256:1242566dfddc06682ce27d14e55d6[9](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:9)86846cc3e176d43c72c313e21366bb846a
Status: Downloaded newer image for ghcr.io/project-copacetic/copa-action:v0.7.0
+ image=***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
+ report=report.json
+ patched_tag=patched
+ timeout=5m
+ connection_format=buildx
+ format=openvex
+ output_file=vex.json
+ echo ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
+ cut -d: -f1
+ image_no_tag=***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak
+ [ -z vex.json ]
+ output=--format openvex --output ./data/vex.json
+ docker buildx create --name=copa-action
copa-action
+ docker buildx use --default copa-action
+ connection=--addr buildx://copa-action
+ copa patch -i ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 -r ./data/report.json -t patched --addr buildx://copa-action --timeout 5m --format openvex --output ./data/vex.json
#1 resolve image config for docker-image://***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
#1 ERROR: unexpected status from HEAD request to https://***.dkr.ecr.us-east-1.amazonaws.com/v2/ironbank/opensource/keycloak/keycloak/manifests/25.0.1-0: 401 Unauthorized
Error: unexpected status from HEAD request to https://***.dkr.ecr.us-east-1.amazonaws.com/v2/ironbank/opensource/keycloak/keycloak/manifests/25.0.1-0: 401 Unauthorized
+ echo Error patching image ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 with copa
+ exit 1
Error patching image ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 with copa
Error: Process completed with exit code 1.

This is a private registry and private git repository, but the full contents of the github workflow is:

name: Docker Image Build

on:
  workflow_dispatch:
    inputs:
      keycloak_tag:
        description: "The keycloak tag to patch.  Assumes that ECR already has the ironbank tag with `-0` appended in to use as a base image."
        required: true


jobs:

  build:

    name: Build Image
    runs-on: ubuntu-latest

    steps:
    - name: Configure environment
      run: |
        echo "TAG=${{ github.event.inputs.keycloak_tag }}" >> $GITHUB_ENV
        echo "REGISTRY=***REDACTED***.dkr.ecr.us-east-1.amazonaws.com" >> $GITHUB_ENV
        echo "REPOSITORY=ironbank/opensource/keycloak/keycloak" >> $GITHUB_ENV
        mkdir -p ${HOME}/.aws
        echo "[default]" >> ${HOME}/.aws/credentials
        AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}
        echo "aws_access_key_id = ${AWS_ACCESS_KEY_ID}" >> ${HOME}/.aws/credentials
        echo "aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}" >> ${HOME}/.aws/credentials
        aws ecr get-login-password --region us-east-1 | sudo docker login --username AWS --password-stdin ***REDACTED***.dkr.ecr.us-east-1.amazonaws.com

    - name: Configure AWS credentials us-east-1
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1

    - name: Login to Amazon ECR us-east-1
      id: us-east-1-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Get the Docker image
      run: |
        sudo docker pull ${REGISTRY}/${REPOSITORY}:${TAG}-0

    - name: Generate Trivy Report
      uses: aquasecurity/trivy-action@d43c1f16c00cfd3978dde6c07f4bbcf9eb6993ca # 0.16.1
      with:
        scan-type: "image"
        format: "json"
        output: "report.json"
        ignore-unfixed: true
        vuln-type: "os"
        image-ref: ${{ env.REGISTRY }}/${{ env.REPOSITORY }}:${{ env.TAG }}-0

    - name: Check vulnerability count
      id: vuln_count
      run: |
        report_file="report.json"
        vuln_count=$(jq 'if .Results then [.Results[] | select(.Class=="os-pkgs" and .Vulnerabilities!=null) | .Vulnerabilities[]] | length else 0 end' "$report_file")
        echo "vuln_count=$vuln_count" >> $GITHUB_OUTPUT
        echo "vuln_count=$vuln_count"

    - name: Run Copa action
      if: steps.vuln_count.outputs.vuln_count != '0'
      id: copa
      # using main for testing purposes
      # use a tag (such as v1 or v1.0.1) at a bare minimum
      # recommendation is to pin to a digest for security and stability
      # and rely on dependabot for digest/version updates
      uses: project-copacetic/copa-action@main
      with:
        image: ${{ env.REGISTRY }}/${{ env.REPOSITORY }}:${{ env.TAG }}-0
        image-report: "report.json"
        patched-tag: "patched"
        timeout: "5m" # optional, default is 5m
        output: vex.json # optional
        format: "openvex" # optional, default is openvex

    - name: Tag Copa image for push
      if: steps.vuln_count.outputs.vuln_count != '0'
      id: tag_copa_img
      run:
          docker tag ${REGISTRY}/${REPOSITORY}:${TAG}-0-patched ${REGISTRY}/${REPOSITORY}:${TAG}

    - name: Tag and push the Docker image to us-east-1
      if: steps.vuln_count.outputs.vuln_count != '0'
      env:
        ECR_REGISTRY: ${{ steps.us-east-1-ecr.outputs.registry }}
      run: |
          docker tag ${REPOSITORY}:${TAG} $ECR_REGISTRY/${REPOSITORY}:${TAG}
          docker push $ECR_REGISTRY/$REPOSITORY:${TAG}

In my case, I have copied the original keycloak image from ironbank to our ECR with a tag of 25.0.1-0 and I'm attempting to patch the image and update the 25.0.1 tag for keycloak with the output from copa. I don't think ironbank images have anything special about them, we could copy registry.access.redhat.com/ubi9/ubi-minimal:9.4-949 to our private ECR as registry.access.redhat.com/ubi9/ubi-minimal:9.4-949-0 and then attempt to run copa against it and have the same results.

[ashnamehrotra]

Hi @anthony-zawacki sorry for the delayed response, are you able to patch this same image locally using copa or does it result in the same error? Also, since you are able to pull the image in the workflow, can you try the containerd image store approach for local images to see if that also results in the same error?

[anthony-zawacki]

This is a touch painful. :)

At the top of my workflow, I've added:

    steps:
    - name: Set up crazy-max's Docker server
      uses: crazy-max/ghaction-setup-docker@v3
      with:
        daemon-config: |
          {
            "debug": true,
            "experimental": true,
            "features": {
              "containerd-snapshotter": true
            }
          }
    - name: Get socket path
      run: |
        url=$(docker context inspect | jq -r .[0].Endpoints.docker.Host)
        socket_path=$(echo "$url" | awk -F// '{print $2}')
        echo "$socket_path"
        echo "SOCKET=$socket_path" >> $GITHUB_ENV
        echo "DOCKER_URL=$url" >> $GITHUB_ENV

However, aquasecurity/[email protected] runs and builds a local image in docker before these steps run. Trying to use the crazy-max docker server fails on the trivy step because the local image they built as part of the set up to the action doesn't exist in that docker server. To work around this, in my environment configuration section, changed it to log into the ECR registry with both the "standard" docker server as well as the crazy-max docker server:

    - name: Configure environment
      run: |
        echo "TAG=${{ github.event.inputs.keycloak_tag }}" >> $GITHUB_ENV
        echo "REGISTRY=948564350653.dkr.ecr.us-east-1.amazonaws.com" >> $GITHUB_ENV
        echo "REPOSITORY=ironbank/opensource/keycloak/keycloak" >> $GITHUB_ENV
        mkdir -p ${HOME}/.aws
        echo "[default]" >> ${HOME}/.aws/credentials
        AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}
        echo "aws_access_key_id = ${AWS_ACCESS_KEY_ID}" >> ${HOME}/.aws/credentials
        echo "aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}" >> ${HOME}/.aws/credentials
        ECR_PASSWORD="$(aws ecr get-login-password --region us-east-1)"
        echo $ECR_PASSWORD | sudo -s docker -H $DOCKER_URL login --username AWS --password-stdin 948564350653.dkr.ecr.us-east-1.amazonaws.com
        sudo -s docker -H $DOCKER_URL info
        echo $ECR_PASSWORD | sudo docker login --username AWS --password-stdin 948564350653.dkr.ecr.us-east-1.amazonaws.com
        sudo docker info

Using that, I can see both servers, the crazy-max one running 27.2.0 and the "standard" one running 26.1.3.

I updated my image pull section to pull the target image to both docker servers:

    - name: Get the Docker image
      run: |
        sudo -s docker -H $DOCKER_URL pull ${REGISTRY}/${REPOSITORY}:${TAG}-0
        sudo docker pull ${REGISTRY}/${REPOSITORY}:${TAG}-0

This allows trivy to run using the "standard" docker server and copa to run using the crazy-max docker server, both accessing different copies of the same image.

The copa step now looks like:

    - name: Run Copa action
      if: steps.vuln_count.outputs.vuln_count != '0'
      id: copa

      uses: project-copacetic/copa-action@main
      with:
        image: ${{ env.REGISTRY }}/${{ env.REPOSITORY }}:${{ env.TAG }}-0
        image-report: "report.json"
        patched-tag: "patched"
        timeout: "5m" # optional, default is 5m
        output: vex.json # optional
        format: "openvex" # optional, default is openvex
        custom-socket: $SOCKET

This appears to work, I now get to where copa times out trying to update the image:

#12 sh -c /usr/bin/microdnf update python-unversioned-command cups-libs krb5-libs libcurl-minimal orc curl-minimal python3 python3-libs python3-setuptools-wheel && /usr/bin/microdnf clean all
#12 0.243 
#12 0.243 (microdnf:7): librhsm-WARNING **: 15:24:37.597: Found 0 entitlement certificates
#12 0.251 
#12 0.251 (microdnf:7): librhsm-WARNING **: 15:24:37.605: Found 0 entitlement certificates
#12 0.335 Downloading metadata...
#12 0.569 Downloading metadata...
#12 1.081 Downloading metadata...
#12 1.335 Package                                                        Repository               Size
#12 1.335 Upgrading:                                                                                  
#12 1.335  cups-libs-1:2.3.3op2-27.el9_4.x86_64                          ubi-9-baseos-rpms    270.2 kB
#12 1.335   replacing cups-libs-1:2.3.3op2-24.el9.x86_64                                              
#12 1.335  curl-minimal-7.76.1-29.el9_4.1.x86_64                         ubi-9-baseos-rpms    131.8 kB
#12 1.335   replacing curl-minimal-7.76.1-29.el9_4.x86_64                                             
#12 1.335  krb5-libs-1.21.1-2.el9_4.x86_64                               ubi-9-baseos-rpms    788.3 kB
#12 1.335   replacing krb5-libs-1.21.1-1.el9.x86_64                                                   
#12 1.335  libcurl-minimal-7.76.1-29.el9_4.1.x86_64                      ubi-9-baseos-rpms    233.7 kB
#12 1.335   replacing libcurl-minimal-7.76.1-29.el9_4.x86_64                                          
#12 1.335  orc-0.4.31-7.el9_4.x86_64                                     ubi-9-appstream-rpms 191.2 kB
#12 1.335   replacing orc-0.4.31-6.el9.x86_64                                                         
#12 1.335  python-unversioned-command-3.9.18-3.el9_4.5.noarch            ubi-9-appstream-rpms  10.3 kB
#12 1.335   replacing python-unversioned-command-3.9.18-3.el9_4.1.noarch                              
#12 1.335  python3-3.9.18-3.el9_4.5.x86_64                               ubi-9-baseos-rpms     30.3 kB
#12 1.335   replacing python3-3.9.18-3.el9_4.1.x86_64                                                 
#12 1.335  python3-libs-3.9.18-3.el9_4.5.x86_64                          ubi-9-baseos-rpms      8.2 MB
#12 1.335   replacing python3-libs-3.9.18-3.el9_4.1.x86_64                                            
#12 1.335  python3-setuptools-wheel-53.0.0-12.el9_4.1.noarch             ubi-9-baseos-rpms    480.1 kB
#12 1.335    replacing python3-setuptools-wheel-53.0.0-12.el9.noarch                                  
#12 1.335 Transaction Summary:
#12 1.335  Installing:        0 packages
#12 1.335  Reinstalling:      0 packages
#12 1.335  Upgrading:         9 packages
#12 1.335  Obsoleting:        0 packages
#12 1.335  Removing:          0 packages
#12 1.335  Downgrading:       0 packages
#12 1.335 Is this ok [y/N]: time="2024-09-05T15:29:04Z" level=error msg="patch exceeded timeout 5m0s"
Error: patch exceeded timeout 5m0s
+ echo Error patching image ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 with copa
+ exit 1
Error patching image ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 with copa
Error: Process completed with exit code 1.

This is project-copacetic/copacetic#714 which has been fixed, but not yet released. Is there any facility to run a nightly build of copa of some sort where I could test to see if it actually successfully makes it to the end of the patching, or do we need to wait for 0.7.1 release? I have gotten past the step with a custom-built copa binary where I had fixed the missing -y in the command. I used that in my jenkins pipeline where I ended up opening that issue and moby/buildkit#5244 which is stalled because everyone says the problem is someone else's. However, if the microdnf issue is fixed and copa-action works, I'll not worry about the buildkitd bug and just go this route. :)

[ashnamehrotra]

However, aquasecurity/[email protected] runs and builds a local image in docker before these steps run. Trying to use the crazy-max docker server fails on the trivy step because the local image they built as part of the set up to the action doesn't exist in that docker server.

This issue seems to be related to #46 and using the --docker-host flag within Trivy. We have also released a new version of copa with microdnf issue resolved :)

Let me know if theres any other issues!