`docker images`: Multi-platform builds for docker images using self-hosted runners #135

dsotirakis · 2024-05-28T08:47:53Z

This PR introduces a reusable workflow which can be used for multi-platform docker image builds using our self-hosted runners, to build images for both arm and amd architectures.

The reusable workflow calls a few composite actions that already exist as shared workflows.

Running example: https://github.com/grafana/grafana-ci-sandbox/actions/runs/9794036255/attempts/6

Notes:

Eventually this will be the only push to GAR workflow we will use, and will replace the already existing push-to-gar-docker action.

The workflow file can be as simple as: https://github.com/grafana/grafana-ci-sandbox/actions/runs/9794036255/workflow

Also, in case the workflow is complicated and we cannot use the reusable action as happened above, we can use the composite actions instead. Example workflow:
https://github.com/grafana/k6-cloud-github-workflows/pull/18/files
Result: https://github.com/grafana/k6-cloud-metrics-api/actions/runs/9877247789/job/27308095542

dsotirakis · 2024-07-19T07:16:31Z

actions/build-and-push-image-digests/action.yaml

+    default: "docker-container"
+
+outputs:
+  full_image_name:


Dirty but for now re-calculating the whole image name for GAR is difficult, we can get away with it using a simple output.

zerok · 2024-07-24T07:50:49Z

actions/build-and-push-image-digests/README.md

@@ -0,0 +1,56 @@
+# build-and-push-image-digests
+
+This is a composite GitHub Action, used to build and push image digests to GitHub, so then they can be picked up by the `push-to-gar-docker-multiarch` action, to create a manifest and push the actual image to GAR.


Suggested change

This is a composite GitHub Action, used to build and push image digests to GitHub, so then they can be picked up by the `push-to-gar-docker-multiarch` action, to create a manifest and push the actual image to GAR.

This is a composite GitHub Action, used to build and push image digests to GitHub, so then they can be picked up by the `push-to-gar-docker-multiarch` workflow, to create a manifest and push the actual image to GAR.

zerok · 2024-07-24T07:53:16Z

actions/build-and-push-image-digests/action.yaml

+    description: |
+      The dockerfile to use.
+    required: false
+  platform:


In the README that input is called platforms, which is more fitting IMO 🙂

zerok · 2024-07-24T07:56:08Z

.github/workflows/push-to-gar-docker-multiarch.yaml

+  workflow_call:
+    inputs:
+      platforms:
+        default: "['amd64']"


Would it perhaps be better to also use the same format as we have with the docker/build-push-action?

linux/amd64 linux/arm64 ...

zerok · 2024-07-24T07:57:59Z

actions/create-and-push-image-manifests/action.yaml

+        path: /tmp/digests
+        pattern: digests-*-${{ inputs.tag }}-${{ inputs.environment }}
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v3


zerok · 2024-07-24T07:58:13Z

actions/create-and-push-image-manifests/action.yaml

+      uses: docker/setup-buildx-action@v3
+    - id: meta
+      name: Docker meta
+      uses: docker/metadata-action@v5


zerok · 2024-07-24T07:58:52Z

actions/create-and-push-image-manifests/action.yaml

+  using: composite
+  steps:
+    - name: Checkout
+      uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11


Please add the version indicator to allow auto-updates.

iainlane

sorry for taking a while to get to this. great work 🚀. I think my comments will sound like a lot, but these are going to be some of the most important interfaces we offer from this repo so it's worth trying to design the interfaces to be the best we can. I'm up for pairing if you want.

here are my main thoughts in addition to the specific comments I left

Generality

It'd be best if we avoid repeating our earlier mistake of having parallel actions for GAR and Docker Hub which behave slightly differently. What you're building here could replace those. But as it is currently, the actions/workflows hardcode GAR quite deeply.

Flexibility

I think we should provide for users who need to run pre/post steps. We should provide building blocks for building/pushing multiarch images, so that if you need to do something before or after running the build it's possible. In addition, we can provide packaged-up versions as reusable workflows too for those who have simple needs.

Imagine I want to check the repo out, run some scripts (fetch generated artifacts, fill in variables with build info, whatever) and then build that state.

Or maybe I want to supply custom mounts, pass some weird Docker options or anything else. I could run the build myself.

tl;dr plenty of projects want to do custom stuff and if we are too packaged then we prevent that.

Layout

Maybe we should start grouping the actions under docker/ to keep all the related ones together. (I know it can't work for reusable workflows - maybe there they could all start with docker-?)

iainlane · 2024-07-24T17:24:41Z

.github/workflows/push-to-gar-docker-multiarch.yaml

+      - name: Checkout
+        env:
+          action_repo: ${{ github.action_repository }}
+          action_ref: ${{ github.action_ref }}
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+        with:
+          repository: ${{ env.action_repo }}
+          ref: ${{ env.action_ref }}
+      - name: Checkout
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+        with:
+          repository: grafana/shared-workflows
+          ref: multiarch-docker-builds-using-sh-runners
+          path: shared-workflows


Aren't these the same? I thought github.action_repository will be shared-workflows when you're running this, ditto action_ref and the selected ref? Or is it different for reusable workflows?

iainlane · 2024-07-24T17:25:27Z

actions/build-and-push-image-digests/action.yaml

+    - name: try printing env
+      shell: bash
+      run: |
+        echo ${{ env.platform_env }}


Suggested change

- name: try printing env

shell: bash

run: |

echo ${{ env.platform_env }}

iainlane · 2024-07-24T17:25:52Z

actions/build-and-push-image-digests/action.yaml

+      shell: bash
+      run: |
+        echo ${{ env.platform_env }}
+    - name: Prepare


Can you add a comment saying what this is doing, or make the name more descriptive?

iainlane · 2024-07-24T17:27:04Z

actions/build-and-push-image-digests/action.yaml

+    - name: Get repository name
+      id: get-repository-name
+      shell: bash
+      run: |
+        REPO_NAME=$(echo ${{ github.repository }} | awk -F'/' '{print $2}')
+        echo "repo_name=${REPO_NAME}" >> ${GITHUB_OUTPUT}


This could be merged into prepare I think

iainlane · 2024-07-24T17:27:40Z

actions/build-and-push-image-digests/action.yaml

+    - name: Resolve GCP project
+      id: resolve-project
+      shell: bash
+      run: |
+        if [ "${{ inputs.environment }}" == 'dev' ]; then
+          PROJECT="grafanalabs-dev"
+        elif [ "${{ inputs.environment }}" == 'prod' ]; then
+          PROJECT="grafanalabs-global"
+        else
+          echo "Invalid environment. Valid environment variable inputs: dev, prod"
+          exit 1
+        fi
+        echo "project=${PROJECT}" >> ${GITHUB_OUTPUT}


And this probably, or alternatively made into a standalone action since we repeat it all over the place.

iainlane · 2024-07-24T17:30:29Z

actions/create-and-push-image-manifests/action.yaml

@@ -0,0 +1,53 @@
+---
+description: Pulls image digests from GitHub and pushes manifests to GAR


I don't think it would be too hard to make this not specific to Artifact Registry.

Mainly removing the login step up to a higher level.

iainlane · 2024-07-24T17:31:20Z

actions/create-and-push-image-manifests/action.yaml

+        images: ${{ inputs.full-image-name }}
+        tags: ${{ inputs.tag }}
+    - id: login-to-gar
+      uses: grafana/shared-workflows/actions/login-to-gar@main


(will be obsolete if you adopt my last comment)

Everywhere else in this PR this repo is checked out and the action is used from there, to keep things in sync. That's what we should do here too.

iainlane · 2024-07-24T17:33:08Z

actions/create-and-push-image-manifests/action.yaml

+      shell: bash
+      run: |
+        docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
+          $(printf '${{ inputs.full-image-name }}@sha256:%s ' *)


Two suggestions:

Do the jq thing on a separate line for a bit better readability, also please add a comment showing what it's transforming (from: xxx to: yyy)

Put inputs.full-image-name through an env block on this step. Just so that you can't give it a value like ';) rm -rf /; or whatever and do mean things to the machine. (Applies to the "Inspect image" step too and maybe others where input is handled similarly)

iainlane · 2024-07-24T17:51:49Z

.github/workflows/push-to-gar-docker-multiarch.yaml

+          path: shared-workflows
+      - name: Build multi-arch
+        id: build-multiarch
+        uses: ./shared-workflows/actions/build-and-push-image-digest


Suggested change

uses: ./shared-workflows/actions/build-and-push-image-digest

uses: ./shared-workflows/actions/build-and-push-image-digests

dsotirakis requested a review from a team as a code owner May 28, 2024 08:47

dsotirakis marked this pull request as draft May 28, 2024 08:48

dsotirakis self-assigned this May 28, 2024

dsotirakis force-pushed the multiarch-docker-builds-using-sh-runners branch 4 times, most recently from 1bd94df to f313444 Compare May 28, 2024 14:54

dsotirakis changed the title ~~docker images: Multi-platform builds for docker images using self-hosted runners~~ [WIP] docker images: Multi-platform builds for docker images using self-hosted runners May 29, 2024

dsotirakis force-pushed the multiarch-docker-builds-using-sh-runners branch 8 times, most recently from 5998e3c to 7aeff14 Compare June 3, 2024 15:52

dsotirakis force-pushed the multiarch-docker-builds-using-sh-runners branch 5 times, most recently from 3c1eb95 to 219c6da Compare June 20, 2024 15:03

dsotirakis force-pushed the multiarch-docker-builds-using-sh-runners branch 9 times, most recently from 50b3970 to 2448fb2 Compare July 5, 2024 14:09

dsotirakis changed the title ~~[WIP] docker images: Multi-platform builds for docker images using self-hosted runners~~ docker images: Multi-platform builds for docker images using self-hosted runners Jul 5, 2024

dsotirakis added the enhancement New feature or request label Jul 8, 2024

dsotirakis force-pushed the multiarch-docker-builds-using-sh-runners branch 12 times, most recently from 7955733 to 0980241 Compare July 11, 2024 08:07

dsotirakis marked this pull request as ready for review July 11, 2024 08:19

dsotirakis added 6 commits July 11, 2024 17:14

Update .gitignore

c6f57c5

Add build-and-push-image-digests action

aa11cb5

Add create-and-push-image-manifests action

98f4728

Add reusable push-to-gar-docker-multiarch workflow

ff1cdf8

Run prettier

a429419

Update docs

879ba70

dsotirakis force-pushed the multiarch-docker-builds-using-sh-runners branch from 3f78e9e to 879ba70 Compare July 11, 2024 14:15

dsotirakis commented Jul 19, 2024

View reviewed changes

zerok reviewed Jul 24, 2024

View reviewed changes

iainlane reviewed Jul 24, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`docker images`: Multi-platform builds for docker images using self-hosted runners #135

`docker images`: Multi-platform builds for docker images using self-hosted runners #135

dsotirakis commented May 28, 2024 •

edited

Loading

dsotirakis Jul 19, 2024

zerok Jul 24, 2024

zerok Jul 24, 2024

zerok Jul 24, 2024

zerok Jul 24, 2024

zerok Jul 24, 2024

zerok Jul 24, 2024

iainlane left a comment

iainlane Jul 24, 2024

iainlane Jul 24, 2024

iainlane Jul 24, 2024

iainlane Jul 24, 2024

iainlane Jul 24, 2024

iainlane Jul 24, 2024

iainlane Jul 24, 2024

iainlane Jul 24, 2024

iainlane Jul 24, 2024

		@@ -0,0 +1,56 @@
		# build-and-push-image-digests

		This is a composite GitHub Action, used to build and push image digests to GitHub, so then they can be picked up by the `push-to-gar-docker-multiarch` action, to create a manifest and push the actual image to GAR.

		@@ -0,0 +1,53 @@
		---
		description: Pulls image digests from GitHub and pushes manifests to GAR

	uses: ./shared-workflows/actions/build-and-push-image-digest
	uses: ./shared-workflows/actions/build-and-push-image-digests

docker images: Multi-platform builds for docker images using self-hosted runners #135

Are you sure you want to change the base?

docker images: Multi-platform builds for docker images using self-hosted runners #135

Conversation

dsotirakis commented May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iainlane left a comment

Choose a reason for hiding this comment

Generality

Flexibility

Layout

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

`docker images`: Multi-platform builds for docker images using self-hosted runners #135

`docker images`: Multi-platform builds for docker images using self-hosted runners #135

dsotirakis commented May 28, 2024 •

edited

Loading