Why we decided to use LXC instead of Docker for build environments in Snap

Posted by Akshay Karle

9 September 2015

This is part 3 of the series on “Bringing Docker Support to Snap CI”. In this article, we talk about security aspects around using containers for provisioning build machines in Snap CI.

Customization is an important aspect in a hosted CI environment. For example, users may need to install different packages, libraries, and services required by their builds that are not provided by default. In order to accommodate this, Snap CI allows users to run arbitrary commands, some of which may need to run as root.

But of course, providing root access within the container of the build environment has huge security implications for a multi-tenant, hosted build server.

The need for user namespace in containers on multi tenant systems

In part 1 of this series, we explained that we adopted OpenVZ containers for build environments back in 2012 because:

“With OpenVZ, we could provide root access within a container without being privileged on the host or other containers. This allowed users to have admin privileges required to customize their build environment on the container but at the same time secured our users against a major attack vector that other container technologies (including LXC) were vulnerable to at that time. [See user namespaces]”

In short, user namespaces basically afford the ability to run processes and access filesystems as a root user in the containers which can be mapped to a non-root user on the host. If a root user on a container were to find an exploit that allowed them into the host, the user wouldn’t be root on the host machine and so wouldn’t be able to cause as much damage as a root user on the host would be able to.

But as great as it is, Docker doesn’t yet support user namespaces. The Docker security page mentions leveraging kernel security features like capabilities to harden the host with systems like SELinux and AppArmor. These tools allow whitelisting of resources and capabilities that the container needs. There is also grsec which has simpler ways to manage the security policies. It requires patching the kernel however. This is something we may look into the future to further harden our hosts.

This DockerCon video has a good introduction to these concepts.

While those techniques to harden the system work well for trusted applications running on self-managed servers, they are not enough for a hosted build environment that executes arbitrary code because it is not possible to whitelist all the resources a user build may need in advance. Due to these security considerations, we decided not to use Docker containers as build machines.

On the other hand, LXC now has support for user namespaces starting with LXC 1.0 that was released in 2014.

We chose LXC containers for build environments

Due to these considerations, we selected unprivileged LXC containers for build machines on top of which users will be able to build and run Docker containers.

If you’re interested in using Docker on Snap CI, sign up here.


comments powered by Disqus