ARM, docker, linux, qemu

Running and building ARM Docker containers in x86

We already covered how Linux executes files and how to run ARM binaries “natively” in Linux in the last two posts. We ended up running a whole ARM root filesystem transparently in a chroot jail using QEMU user mode and binfmt_misc support.

Now that we have that covered, nothing prevents us from applying that to Docker containers. At the end of the day, containers are chroots in steroids, and they also share kernel with the host so all the basic mechanics remain the same.

Running ARM containers

If you haven’t yet, install QEMU user mode, by default it will install also binfmt_misc support.

Now,  we only need to place the qemu-arm-static interpreter inside the container. This is as easy as mount-binding it in the container invocation and voila! We are running an armhf container in our x86 machine.

Building ARM containers

This is a bit of an uglier trick, because we need to copy the QEMU interpreter to the container. The problem is that it adds ~5MiB to the container, but the benefit of being able to build foreign containers natively is immense!

Consider the following simplified Dockerfile to create an armhf nginx container

We end up with a container that can be run both natively in an ARM device, and in an x86 system with a properly configured binfmt_misc support. Not bad!

We can even test it locally

We are now running the ARM nginx web server locally.

This means that we can create a build pipeline with automated testing without requiring any ARM boards. After all automated tests are passing, we can create the production image.

Reclaiming the space in Docker API 1.25+

If we really want to reclaim those extra 5MiB from qemu-arm-static, we can use the new experimental --squash  flag.

Honestly, it was about time that they did something to help us keep the image size down. The piling up of layers and their refusal to add a simple --chown  argument to the COPY command made the creation of Dockerfiles a black art where you had to chain all the installation steps including the cleanup in a single RUN statement, therefore denying the benefits of caching for big blocks of code.

Thankfully, they are beginning to react and we now have not only  --chown , but also the --squash  flag for the build command. In order to use it, we need to enable experimental features. Add the following to the daemon.json file, which might not exist

Restart the Docker daemon, after this.

Reclaiming the space in Docker API <1.25

We are left with the old hacky way of flattening the images, by exporting the container

, and finally rebuild the metadata

Author: nachoparker

Humbly sharing things that I find useful [ github dockerhub ]

8 Comments on “Running and building ARM Docker containers in x86

  1. Thanks a lot for posting!

    I tried this on Ubuntu 16.04 and upon the first step I got an error saying standard_init_linux.go:190: exec user process caused "no such file or directory". It seems that /usr/bin/qemu-arm-static was missing and I needed to install qemu-user-static using apt-get install qemu-user-static. After that I was set to go.

  2. Thanks a lot for those posts. Very useful.

    In the meantime the docker image format has gained some support for platform-specific images.
    For images that already specify those platform tags you now need to give the additional (experimental) --platform cli arg to docker run.

    docker run –platform arm64 -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static –rm -ti arm64v8/python:3.7

  3. Thanks for this really helpful article. On some machine I had to run apt-get install qemu-user-static instead of apt-get install qemu-user. Other wise I was getting a permission denied liked error during docker build.

  4. Hello, everybody.

    Thanks for the post. It’s serving me in the development I’m doing. Very useful.

    As a note, I tried to install it on Ubuntu 18.04 and needed to install qemu-user-static with the command:
    apt-get install qemu-user-static

    It has been necessary to be able to mount the volume inside the container because if the folder is not empty.

    Thank you and greetings.

  5. I was building Pytorch, and using -mfpu=neon-vfpv4 flag gives neon errors, and it says neon isn’t supported. Is this because a generic armv32 cpu was used? Is there anyway to regain support? btw, you’ve really saved me tons of time…took 3 days to compile pytorch for our use, took more like 20min in the docker container.

Leave a Reply

Your email address will not be published. Required fields are marked *