You heard a bit about managing Jenkins in my first blog in this series centered on bringing order to the unruly. For this blog, I dug a bit deeper into my velocity inefficiencies, bringing up some security, governance, and compliance challenges in our workflows. And no, these aren’t just DevOps buzzwords you see on websites or banners at trade shows—they are key in keeping up the speed of software delivery. What I found was that, without consistent governance or compliance for our security protocols, the automation became less, well, automated.
Here are two more woes I identified with my unmanaged Jenkins that perhaps you’ve found as well.
Woe #3: Inconsistency
At this point, you may have noticed a theme emerging from my previous blog: decentralized control and lack of visibility generate chaos. With too many parties manipulating too many variables, it’s rough getting everyone and everything on the same page. This creates significant roadblocks.
Lack of governance inhibits best practices. When any number of people can spin up, configure, or modify a server, you effectively have no capacity for governance—and little chance of consistently implementing best practices (e.g., regular security scans, prohibitions on unsafe plugins, etc.).
Plugins divide servers, workflows, and teams. Without plugin management, your teams may populate their servers with any number (and quality) of plugins. This opens considerable compatibility and interoperability issues between teams who will inevitably use conflicting plugins and update shared plugins on different schedules.
Lack of consistency leads to more bugs. Pipeline consistency is critical for avoiding bugs. With differently configured controllers, disparate plug-in sets, variable workflows, and largely disconnected teams, unmanaged Jenkins can increase the time your engineers spend hunting bugs.
Backups can be sporadic. Backing up data—and verifying the integrity of those backups—requires discipline. Without meaningful governance, your backup strategy is likely to have holes, and, if that’s the case, you’re always at risk of losing data.
Woe #4: Insecurity
Cybersecurity is an endless challenge. Cloud and business transformations make for highly fluid (and expanding) attack surfaces, and the rapid pace of innovation requires constant vigilance to thwart would-be attackers. So, how does that work for enterprises subject to the woes we’ve already covered?
No one is empowered to enforce security policy. Without centralized authority or reliable governance, you’ll struggle getting teams to adhere to security best practices (or even follow basic policy). This brings risky plugins, risky code, and risky practices to your pipeline.
Your security is only as strong as your least safe plugin. As plugins proliferate across your controllers, any security holes in those plugins become your liabilities. Whether plugin developers plug those holes, and whether your teams reliably update their plugins to take advantage of security fixes, are security wildcards.
Vulnerability remediation can be slow to nonexistent. With too many cooks in the kitchen and a generalized lack of visibility and coordination, implementing fixes for any security threats you do discover will be cumbersome.
Chaos plus weak security makes compliance difficult. Compliance is all about demonstrating that you are in control, have reliable safeguards, and can meet security standards. For all the reasons listed above, this is tough without a managed Jenkins implementation.
Enter Standardization, Automation, and Optimization!
You’ve already read how CloudBees CI helped me rein in the chaos of my unmanaged Jenkins instances, but it was time for me to get serious about improving efficiencies across the board. That meant consistency and automation. Here are a few items that I identified as helpful to me and my organization:
All dev teams start with a trusted, verified, and supported version of Jenkins and a curated set of plugins tested for stability and security. Support was the big one for me. I had someone by my side for questions—no matter how simple they were.
Repeatable components are provisioned utilizing custom CasC bundles, enabling teams to start with preconfigured and tested environments managed from a single source.
Pipeline template catalog to enforce consistency and stability operations-wide
You have a collection of pipelines based on best practices so teams never have to start from scratch, and changes to templates are pushed to all team pipelines from a single point.
Up-to-date template code is fetched every time a job is run, so jobs always reflect the most current standard. Optionally, developers could be restricted to certain templates to enforce governance goals.
Governance developers could avoid starting from scratch by modifying existing templates to suit new agendas.
Custom Marker Files allow for instant build-template selection based on software configuration management (SCM) identifiers.
Automated backup and recovery
Eliminate human error from your backup strategy. We have a lot of independent thinkers, which is magic for feature development but not good for consistent environment setup. This helped create an environment for them to spin up and just work with, as opposed to creating something from scratch every time.
Manage events across multiple controllers with a powerful publisher/subscriber model, where teams subscribe to pipeline events and act on them regardless of location.
Pipelines work in sync, with different teams able to work collaboratively across pipelines using synchronized automation execution.
Teams can be isolated by controllers, but they can still work together to automate your pipelines.
Every team should be able to safely use different plugins and technologies without the risk of impacting other teams.
Developer productivity enhancements
Receive granular, actionable build data directly in GitHub and Slack.
Security and Compliance Made Easy
Security and compliance was a big challenge for us. After all, it’s difficult to achieve either if you aren’t in total control of your development environment—and few developers are. Fortunately, with centralization and standardization from CloudBees added to the mix, security and compliance become relatively easy. With every team using a secure build and approved pipeline configurations, I had guardrails supporting my SDLC across the board. Here are some items that made my job a lot easier:
Centrally managed security policy
Controllers are authorized to run only tested, validated, and secure versions of Jenkins. If issues arise with a build, it’s easier to replicate, fix, and update that build globally.
Manage authentication and authorization features from a single location:
Use single sign-on (SSO) across all controllers.
Define security credentials using the role-based access control (RBAC) model, and group access at the control plane, folder, or individual controller level.
Enforce access to controllers, agents, or items using folder hierarchies.
Combine RBAC with folder restrictions for added security, and decide who can create, modify, delete, and use credentials at each place in the folder hierarchy.
Scope security on a per-team basis.
Use templated workflows to build security into your pipelines.
Simply build compliance by using CasC bundles and pipeline templates.
Minimized risk from plugins
Limit teams to a curated set of plugins (CloudBees Assurance Program - CAP) that are tested for stability and security and confirmed to meet compliance goals.
Scalability to Match Your Growth
Automating at scale is the Jenkins grail quest. Unmanaged Jenkins is great at the automation part—not so much at the at scale part. And this speaks to not managing a team, but rather, the management of teams. I needed to provide room and support for growth within my team and plan for the creation of new teams as our portfolio expanded. The human scalability of growth needs to be top of mind, because what is the purpose of software if someone isn't putting it to use? Someone needs to make sure everyone has what they need to get their job done. Some of the features that worked for me above have a hand here in scalability.
Deliver your Jenkins at scale by automating provisioning of new teams/controllers, propagating best practices (pipeline templates, CasC, etc.), and managing resources for right-sized infrastructure.
Teams are empowered to quickly spin up new controllers and workspaces on their own while ensuring that the SDLC remains consistent, governed, compliant, and secure.
If you are using Kubernetes, optimize infrastructure resources when scaling by leveraging Kubernetes provisioning. That way, controllers can be spun up as needed, and then they can be hibernated when not in use.
Finally, to truly make Jenkins enterprise ready, we have to address the elephant in the room: lack of structured support with Jenkins. Community assistance is one of the best things on Earth, but when disaster strikes, you don’t have time to dig around forums to find a solution. To bake resiliency into your game plan, you’ll need rock-solid, professional support so you can concentrate on building apps rather than chasing down fixes.
Let the CloudBees team of Jenkins experts guide you. You don't need to rely on Rick—your lovable Jenkins expert who collects limited edition 1980s Star Wars figures (we share the same passion, Rick). Rely instead on the experts to help, and let Rick work his magic elsewhere.
CloudBees CI is built on Jenkins and adds additional features to meet the needs of an enterprise. We know Jenkins, and because we contribute a lot of code to the open-source project, we are uniquely situated to fix issues—should they arise.
Additionally, CloudBees proactively addresses vulnerabilities in the Jenkins build to ensure they never reach your installation, and we also built the Jenkins Health Advisor. The health advisor provides health-status monitoring across your Jenkins environment, delivering notifications before issues (security vulnerabilities, performance bottlenecks, plugin defects, etc.) can affect your users. The health advisor is available to all Jenkins users and not just CloudBees CI users, so if you have an elite team dedicated to supporting Jenkins, you can use this too.
In my next blog, I’ll tackle the dreaded infrastructure problem. Infrastructure is like foundation, the crawl space under your house, the basement. I want to know when there is a problem, but I also don't want to know—because I know it will be a lot of work to fix. This was all me, denying I had infrastructure issues as we limped along. Sort of like when I know there is a spider on the ceiling… but maybe if I ignore it, it will go away. No such luck—those little creatures will show up again somewhere.
In the meantime, download the Massive ROI with Managed Jenkins eBook to learn more.