ARM, docker, linux, qemu

Running and building ARM Docker containers in x86

We already covered how Linux executes files and how to run ARM binaries “natively” in Linux in the last two posts. We ended up running a whole ARM root filesystem transparently in a chroot jail using QEMU user mode and binfmt_misc support.

Now that we have that covered, nothing prevents us from applying that to Docker containers. At the end of the day, containers are chroots in steroids, and they also share kernel with the host so all the basic mechanics remain the same.

Running ARM containers

If you haven’t yet, install QEMU user mode, by default it will install also binfmt_misc support.

# apt-get install qemu-user

Now,  we only need to place the qemu-arm-static interpreter inside the container. This is as easy as mount-binding it in the container invocation and voila! We are running an armhf container in our x86 machine.

$ docker run -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static --rm -ti arm32v7/debian:stretch-slim

Building ARM containers

This is a bit of an uglier trick, because we need to copy the QEMU interpreter to the container. The problem is that it adds ~5MiB to the container, but the benefit of being able to build foreign containers natively is immense!

Consider the following simplified Dockerfile to create an armhf nginx container

FROM arm32v7/debian:stretch-slim

COPY qemu-arm-static /usr/bin

RUN apt-get update;\
    apt-get install -y nginx;\
    echo "\ndaemon off;" >> /etc/nginx/nginx.conf

CMD ["nginx"]

We end up with a container that can be run both natively in an ARM device, and in an x86 system with a properly configured binfmt_misc support. Not bad!

We can even test it locally

docker build . -t nginx-armhf:testing
docker run --rm -ti -d -p 80:80 nginx-armhf:testing
firefox localhost

We are now running the ARM nginx web server locally.

This means that we can create a build pipeline with automated testing without requiring any ARM boards. After all automated tests are passing, we can create the production image.

FROM nginx-armhf:testing

RUN rm /usr/bin/qemu-arm-static
docker build . -t nginx-armhf:production
Reclaiming the space in Docker API 1.25+

If we really want to reclaim those extra 5MiB from qemu-arm-static, we can use the new experimental –squash  flag.

docker build --squash . -t nginx-armhf:production

Honestly, it was about time that they did something to help us keep the image size down. The piling up of layers and their refusal to add a simple –chown  argument to the COPY command made the creation of Dockerfiles a black art where you had to chain all the installation steps including the cleanup in a single RUN statement, therefore denying the benefits of caching for big blocks of code.

Thankfully, they are beginning to react and we now have not only  –chown , but also the –squash  flag for the build command. In order to use it, we need to enable experimental features. Add the following to the daemon.json file, which might not exist

{ "experimental": true }

Restart the Docker daemon, after this.

Reclaiming the space in Docker API <1.25

We are left with the old hacky way of flattening the images, by exporting the container

docker container create --name nginx-armhf-container nginx-armhf:production
docker export nginx-armhf-container | docker import - nginx-armhf:raw

, and finally rebuild the metadata

FROM nginx-armhf:raw

CMD ["nginx"]
docker build . -t nginx-armhf:latest

Author: nachoparker

Humbly sharing things that I find useful [ github dockerhub ]


  1. Thanks a lot for posting!

    I tried this on Ubuntu 16.04 and upon the first step I got an error saying `standard_init_linux.go:190: exec user process caused “no such file or directory”`. It seems that `/usr/bin/qemu-arm-static` was missing and I needed to install `qemu-user-static` using `apt-get install qemu-user-static`. After that I was set to go.

  2. Thanks a lot for those posts. Very useful.

    In the meantime the docker image format has gained some support for platform-specific images.
    For images that already specify those platform tags you now need to give the additional (experimental) `–platform` cli arg to docker run.

    docker run –platform arm64 -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static –rm -ti arm64v8/python:3.7

  3. Thanks for this really helpful article. On some machine I had to run `apt-get install qemu-user-static` instead of `apt-get install qemu-user`. Other wise I was getting a permission denied liked error during docker build.

  4. Hello, everybody.

    Thanks for the post. It’s serving me in the development I’m doing. Very useful.

    As a note, I tried to install it on Ubuntu 18.04 and needed to install qemu-user-static with the command:
    apt-get install qemu-user-static

    It has been necessary to be able to mount the volume inside the container because if the folder is not empty.

    Thank you and greetings.

  5. I was building Pytorch, and using -mfpu=neon-vfpv4 flag gives neon errors, and it says neon isn’t supported. Is this because a generic armv32 cpu was used? Is there anyway to regain support? btw, you’ve really saved me tons of time…took 3 days to compile pytorch for our use, took more like 20min in the docker container.

  6. Please bear with me, but I did not understand, when I look at the command:
    docker run -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static –rm -ti arm32v7/debian:stretch-slim
    I think it should run the default command of arm32v7/debian:stretch-slim in the container, and map /usr/bin/qemu-arm-static in the x86 machine to /usr/bin/qemu-arm-static in the container.
    How does the container get to run in the emulator ?

    1. I’m sorry, I wrote it before reading ” how to run ARM binaries natively” … which contains the answer and much more

Leave a Reply to Kiran Cancel reply

Your email address will not be published. Required fields are marked *