Disaster-proofing agents with Docker Swarm and the CloudBees Jenkins Platform
Standardizing build environments is a best practice for improving Jenkins resiliency, since generic and easily replaceable build environments reduce the impact of an outage within a build farm. When a agent is configured from a standardized template and Jenkins jobs are configured to install required tooling at runtime, any agent in a given pool can seamlessly take on any downed agents’ workload. This concept is known as fungible agents and was coined by Andrew Bayer at a Jenkins User Conference.
The problem with such a setup is not the setup itself but the process to achieve it. Configuring a machine to act as agent inside your infrastructure can be tedious and time consuming. This is especially true when the same setup has to be replicated on a large pool of agents.
Tools for configuration management or a pre-baked image can be excellent solutions to this end, while containers and virtualization are also popular tools for creating generic agent environments. Containerization has risen to prominence for this purpose, with Docker being the fastest rising in popularity and ultimately most popular among Jenkins users.
What is Docker?
Docker is an open-source project that provides a platform for building and shipping applications using containers. This platform enables developers to easily create standardized environments that ensure that a testing environment is the same as the production environment, as well as providing a lightweight solution for virtualizing applications.
Docker containers are lightweight runtime environments that consist of an application and its dependencies. These containers run “on the metal” of a machine, allowing them to avoid the 1-5% of CPU overhead and 5-10% of memory overhead associated with traditional virtualization technologies. Docker containers can be created from a read-only template called a Docker image.
Docker Swarm is a clustering tool for Docker which unites Docker pools into a single virtual host.
Scalable and resilient agents
Docker images are an easy way to define a template for a agent machine and Docker containers are lightweight enough to perform almost as well as a “bare metal” machine, making Docker a good candidate for hosting fungible agents.
But what happens when an organization has scaled horizontally, with many controllers in their installation and each needing their own agent pool?
The open-source Docker Plugin allows controllers to create and automatically tear down agents in a Docker installation, but the configuration to connect to the Docker host will need to be re-created on every existing Jenkins controller and again when a new controller is onboarded.
Multiple Docker hosts may also exist, so to ensure the most efficient use of these resources, all Docker hosts in an organization should be pooled together and agent containers run in on an otherwise idle host if possible to maximize the performance of the container (similar to the logic behind the Even Scheduler plugin ). This sharing between Docker hosts allows controllers’ jobs to be built with minimal queue time and prevents some hosts from sitting idly while others are overloaded.
This pooling is possible with Docker Swarm, but job scheduling and Swarm configuration sharing still require special integrations with Jenkins.
Shared Docker Cloud Configuration
As a part of the next release of CloudBees Jenkins Platform, Docker Swarm may be configured as a agent pool whose configuration may be shared between all managed or “client” controllers in an organization. This prevents the pain of having to configure a Swarm for each controller and updating all such configurations should the installation change in any way (location, FS, max containers, etc).
Like other shareable clouds, the Docker Swarm cloud can be created as an object in CloudBees Jenkins Operations Center:
From there, you can configure the location of the Docker Swarm host, any credentials for connecting to it, as well as which Docker image(s) should be used for creating agents and how many containers should be up at any given time.
Where do I start?
This feature is included in CloudBees Jenkins Operations Center, part of the CloudBees Jenkins Platform. Contact sales@cloudbees.com for more information.
Other plugins complement and enhance the ways Docker can be used with Jenkins. Read more about their uses cases in these blogs:
Tracy Kennedy
Associate Product Manager
CloudBees
Tracy Kennedy is an associate product manager for CloudBees and is based in Richmond. Read more about Tracy in her Meet the Bees blog post and follow her on Twitter .
Stay up to date
We'll never share your email address and you can opt out at any time, we promise.