How Jenkins® admins can provide scalable continuous integration (CI) for software delivery teams
How Jenkins admins can cultivate developer productivity
I write a lot about making sure the developers on my team are working as productively as they can. I try to stay on top of the latest ideas and tools to help them with the continuous integration (CI) portions to keep them happily buzzing around coding. But I also work closely with our shared services team. This group goes by many names—platform services, shared services, DevOps engineers—but for us, it was the team that made CI and continuous delivery (CD) turnkey. I sometimes refer to this team of superstars simply as “admins” but understand they do so much more. It is up to them to centralize, curate, and deliver technical tools, services, and capabilities to multiple internal teams. In other words, they oversee the management and governance of our pipelines and keep our environment up to date and configured correctly. Let’s face it—we need to start somewhere with these templates and configurations for software delivery, and it’s up to our admin team to keep us honest and in compliance with our company standards. One less thing to worry about!
I know what you’re thinking—pipeline management isn’t that hard. In fact, at one of my companies, we put the management of pipelines directly onto the developers, which worked well for our small isolated groups. But it’s one thing to set up a CI pipeline for your fellow developers, and it’s another to create CI pipelines that can scale seamlessly and are easy to manage across all teams. Needless to say, we felt some growing pains as our company grew. That’s why simply deploying CI software isn’t enough. Admins who manage CI pipelines also have to take steps to ensure that the processes for deploying, scaling, and managing those pipelines are as efficient as possible. I want to share some growing pains I saw when it came to working with our admins for CI— and, as they overcame these challenges, how it benefited my dev team.
Challenge 1: Slow pipeline setup (Don’t repeat yourself!)
Setting up pipelines can be one of the biggest drains on efficiency. If admins have to create each pipeline that developers need from scratch, they’ll spend enormous amounts of time configuring pipelines. They’re also likely to end up with pipelines that are inconsistent and unpredictable because the pipelines won’t be configured in a uniform way based on centralized standards.
To meet this challenge, the admin team utilized Pipeline Templates and Pipeline Template Catalogs. Using these features, admins can define pipelines using code, then make those pipelines available to whichever developers or development teams need to use them. Templates can be categorized by the team to minimize wading through a sea of templates to choose from.
Pipeline Templates and Pipeline Template Catalogs mean that admins don’t have to reinvent the wheel each time a development team needs a new pipeline. They can instead reuse pipelines they’ve already created by making the templates easily available through the catalog. The result is much less time spent by admins in creating pipelines and much less effort on the part of developers to find and deploy the pipelines they need.
Challenge 2: Maintaining instance consistency (Consistency isn’t boring when it automates the setup)
Typically, admins want to give each development team or project its own CI instance. A common challenge that our admins faced was that each instance is configured differently. If the instances are set up separately and manually, there’s a good chance that their configurations won’t be consistent. In addition, even if you do manage to set up all of your instances in a uniform way initially, changes that your developers request over time may lead to configuration drift, resulting in inconsistent instances. Our admins tackled this by taking advantage of Configuration as Code (CasC). CasC captures the configuration of each of your instances and stores this in Git.
In this way, CasC solves two key challenges related to achieving instance consistency:
It lets you apply new instances using code. This makes it fast and easy to reproduce identical instances across your enterprise environment.
Instance configurations can be managed as code, which means you can version it, test it, and validate it. Code-based configuration makes it much easier to identify configuration inconsistencies, as well as track how your instances are configured.
Since we used CloudBees CI to tame our Jenkins install, the core of the CasC configuration data is the jenkins.yaml file, which defines the Jenkins configuration of each instance. In addition, CloudBees CasC includes several other YAML files:
bundle.yaml (which defines the CasC configuration files themselves)
rbac.yaml (which defines root-level Role-based Access Control [RBAC] configuration for individual managed controllers)
plugins.yaml (which defines which plugins to install or update on managed controllers)
This made it easier to bundle the full configuration together so teams can get moving quickly and not have to worry about what plugins to install. It’s all captured in the approved CasC bundle.
Challenge 3: Maintaining pipeline governance standards (Automatic governance means focus on code, not policy)
I get a pain in my eye whenever I see the word “governance.” But let me tell you—without this, it is chaos. Chaos that seeps into weekends and makes me want to shake my fist at various inanimate objects. When a CI workflow fails to run properly, it threatens to disrupt not only the operations of whichever team depended on it, but also other teams, whose pipelines may be negatively impacted by the buggy pipeline. For this reason, our admins had to take steps to ensure that they can identify and address situations where pipeline execution does not conform to organizational standards. They also needed to detect pipelines that violate regulatory requirements. One way to do this is to monitor each pipeline as it runs. But that, of course, would require tremendous effort. It’s not practical if you have dozens or more pipelines across your organization.
Our admins used Pipeline Policies to tackle this one. Pipeline Policies let admins define runtime validations that are performed automatically on both scripted and declarative pipelines. Using YAML, admins can impose conditions that govern pipelines at runtime and automatically take action when pipelines violate the rules.
For example, this policy will trigger a warning message in the event that the entire pipeline or a pipeline agent times out for more than 20 minutes or if there is a paused action for more than 20 minutes. The policy could alternatively be configured with the action “fail,” which would stop the build completely.
Using a policy like this, our admins can prevent execution delays with one pipeline from creating instability that affects other pipelines. They can also automatically warn developers about pipelines that are not executing as required. And they can do all of this without having to monitor each pipeline manually or generate feedback by hand when something goes wrong during pipeline execution.
Challenge 4: Enforcing access controls (Security beyond source control management)
Admins typically don’t want every developer in their organization to have complete access to and control over every resource in a CI pipeline. Instead, they want to configure access rights granularly, such that a developer can view build jobs but not trigger them, for example. They may also want an individual developer to have one set of permissions for one CI instance but a different set for a different instance.
For us, we turned to the granular access control permissions, also referred to as the RBAC model, to grant access to specific folders or actions depending on roles in our enterprise universe. For example, we had one instance owned by one cross-functional team. The team consists of multiple people with different roles. The most common roles in this case are as follows:
Developer: will want to be able to kick off builds and modify job configuration
Tester: will want to be able to kick off builds, access build artifacts, and create SCM tags
Manager: will want to be able to manage user roles in the team and create projects
Admin: will want to be able to administer the system
Read only: will require read access to the system but not be able to download build artifacts
Our solution was simple: to create one role for each role of user and add specific permissions for each role (job configure, build, tag, delete, manage, view, etc.). This worked well for this setup, and as noted above, the CasC configuration bundles that generate the CI instances can optionally include an rbac.yaml file. When applied as part of a CasC bundle, an rbac.yaml file like this allows admins to enforce a consistent set of granular access control permissions across all of the CI instances they manage.
Summary: Tie it up into a bow
Creating pipelines that are easy to manage and scale is hard—if you rely on manual configuration processes and workflows, that is.
Since we’re a Jenkins house for our CI orchestration, we used CloudBees CI to turn our unruly Jenkins open source into the “enterprise-ready” CI/CD platform we have today. We removed the time needed to create instances or pipelines, we automated and streamlined pipeline management tasks, and we removed risk with pipelines that were set up with governance and compliance baked in. As a recap, this is how we did it:
Used Pipeline Templates and Pipeline Template Catalogs to reuse and share template configurations across your organization
Maintained CI instance consistency and reproducibility using Configuration as Code
Automatically enforced organizational and regulatory standards during pipeline execution through Pipeline Policies
Configured granular access controls using the rbac.yaml file in your CasC bundle (or through the CloudBees Jenkins Operations Center if not using CasC)
When you take advantage of tools like these, you gain the ability to create new pipelines, modify pipelines, and share pipelines across your organization quickly and easily. At the same time, your developers get fast and easy access to the pipeline configurations they need while also benefiting from more consistent pipeline operations and a lower risk of disruptions to pipeline execution. And the business as a whole gets shared services admins and developers who are happier and more productive. A bit of happiness at work goes a long way.
Download the How Jenkins® Admins Can Provide Scalable CI for Software Delivery Teams, your guide to mitigating and solving CI pipeline challenges to learn more.
Stay up to date
We'll never share your email address and you can opt out at any time, we promise.