Inside Linux Containers (LXC) with Jenkins at CloudBees

This writeup by Caleb Tennis (infrastructure nut) shows the LXC setup at CloudBees - what it is and how we apply it.

One of the big advantages of using a hosted product like our Dev@Cloud Jenkins service is that Cloudbees takes care of the software, security, and maintenance updates required to keep the system highly available.  Recently we made an update to the security of our Jenkins infrastructure, which isn’t necessarily evident from an outside point of view, and wanted to take an opportunity to highlight this new architecture built around Linux containers using LXC.

LXC provides a lightweight way of namespacing off aspects of a running Linux operating system into smaller subsets of logical groupings that isolate resources. In a nutshell, this means that we can run multiple Jenkins instances together on the same Linux machine, but keep them isolated from each other in a way that doesn’t allow them access to any resources of other Jenkins instances on the same machine.

Since a picture is worth 1000 words, let’s just dive in and see how this is implemented:





Container Networking


Each Jenkins container receives its own private IP address that is specific to that container.
These containers are bridged together and accessible back from the main running Linux instance, which happens to be in a completely different network:
Firewalling outside the container is used to prevent Jenkins instances from talking with each other, keeping them strictly isolated.
In addition, within the container only a few ports are open:
In this case all that’s listening are 3 ports for Jenkins services, ssh, and mail services.  That’s it.  These ports are isolated into this namespace and only available on the private IP address listed above, so they don’t conflict with any of the other Jenkins masters running on the same machine.

Process isolation

A quick look at the processes running in the container:
Jenkins container running processes
(see gist)

This is the entire visibility of what’s available to see from Jenkins’ point of view.  In this case, some system initialization  to startup and manage the processes in the container using systemd, the java process itself, the davfs mounting program that provides the webdav /private directory, and sshd/bash/”ps aux” which is me logged in running the command to make the demonstration.

Compare this to the view of the exact same thing from outside the container:

Jenkins container view
The exact same processes can be seen, but with different process IDs.

Filesystem isolation

We’ve taken great care to isolate things at the filesystem level into the container to only provide/allow access to the minimum set of resources necessary to run Jenkins and its subprocesses.  For example, let’s look at the difference between the /etc directory within a container and without, on the same machine:
Filesystem isolation
Jenkins Container Filesystem isolation
Similarly, the system /home directories:
Outside the container:
Jenkins Container Filesystem isolation view
and inside the container:
Jenkins Container Filesystem isolation view


The testing and migration path to the use of LXC for containerized Jeknins has been a half a year process, from initial conception and drawings, to first round implementations, rollouts in our test environment, to a casual rollout across all production accounts.  As we learn more tips and tricks that better implement the security and isolation we want to provide we’ll continue to add them to our infrastructure, and you will continue to reap the benefits of using our hosted Jenkins service. 
Caleb (infrastructure nut at CloudBees).