Error: buildx failed with: ERROR: failed to build: failed to solve: process /dev/.buildkit_qemu_emulator

[y-nk]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

I'm having a CI workflow which has had run for months, without any diff. Suddenly i got to see this error coming from the build-push-action:

Dockerfile:22
--------------------
  20 |     FROM node:${NODE_VERSION}-alpine
  21 |     
  22 | >>> RUN apk add --no-cache tini
  23 |     
  24 |     WORKDIR /app
--------------------
ERROR: failed to build: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apk add --no-cache tini" did not complete successfully: exit code: 1
Error: buildx failed with: ERROR: failed to build: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apk add --no-cache tini" did not complete successfully: exit code: 1

I have the setup-buildx-action@v3 added prior this build step.

Expected behaviour

It should build as usual

Actual behaviour

It stopped building without changes in workflow source

Repository URL

No response

Workflow run URL

No response

YAML workflow

It's a bit too big. There are composite actions involved. Expect something as:

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - uses: docker/setup-buildx-action@v3

      - uses: actions/checkout@v4

      - uses: jdx/mise-action@v2

      - run: pnpm install --frozen-lockfile

      - id: stack
        uses: ./.github/actions/build-stack

      - id: api
        uses: ./.github/actions/build-api

      - uses: docker/build-push-action@v5
        with:
          context: ${{ steps.api.outputs.build-path }}
          platforms: linux/arm64
          tags: |
            ${{ steps.stack.outputs.api-ecr-url }}:latest
          push: true

Workflow logs

No response

BuildKit logs

Additional info

No response

[chadlwilson]

Need to look at the rest of the log to confirm, but I'd guess this is almost certainly caused by the upstream apk-tools issue with QEMU user mode and apk V3 at https://gitlab.alpinelinux.org/alpine/aports/-/issues/17775 if you see apk errors from busybox trigger scripts mentioning execve this will be your issue.

There's a simple reprod at https://github.com/chadlwilson/docker-alpine-323-qemu-test

You will have issues on base images that use Alpine 3.23 with QEMU cross arch arm64 builds. (Including latest node, Alpine 3.23 base image, latest docker dind base image or anything that has updated to Alpine 3.23).

Workaround would be to lock on Alpine 3.22 base images for now.

[y-nk]

@chadlwilson thanks a ton for the informed feedback. i'll look at it later today and see if i have some progress.

[chadlwilson]

Ya, been hunting around waiting for people to report against the official docker images or actions. 😅

[chadlwilson]

The Alpine folks are onto it anyway, but I'm not sure if docker official images will roll back to Alpine 3.22, nor whether it can be fixed on Alpine (e.g with apk upgrade) without a new Alpine release.

At the moment there don't seem to be any known workarounds for whatever is going wrong in QEMU, but possibly that will come if one of the docker buildkit or QEMU experts take a look.

[chadlwilson]

Another alternative workaround on GitHub Actions is to split your workflow to use their native arm64 runners to build and push this platform separately from native hardware (no QEMU) and then push merged metadata and tag in a subsequent workflow job.

On docker-library/ruby#521 (comment) it is noted that adding docker/setup-qemu-action before setup-buildx-action may be a valid workaround but I have not tried this.