Capital One

Capital One Invests in Continuous Delivery to Automate Software Development Pipelines

SUMMARY

Capital One brings highly-customized financial products to market faster with the power of CI/CD and CloudBees

CHALLENGE

Accelerate delivery of business applications while maintaining the highest quality and security standards

SOLUTION

Use the CloudBees Jenkins Platform™ to provide a stable, scalable CI infrastructure, automate repeatable build processes and manage CD pipelines from commit to deployment

RESULTS

» 90% of pipeline automated

» Deployment frequency increased 1,300%

» Engineers focused on application development, not infrastructure

» Quality and security ensured through repeatable processes

SOFTWARE

» CloudBees Jenkins Platform

As one of the most well-known brands in America and among the top 10 largest banks in the nation by deposits, Capital One offers a wide range of financial products and services to commercial clients, small businesses and individual consumers, both online and in-person at branch locations. The company was founded on the vision that the power of information, technology and testing could be harnessed to bring highly-customized financial products directly to consumers.

The thousands of software engineers, testers and other professionals in the Capital One software development organization are making this vision a reality with continuous integration (CI) and continuous delivery (CD) powered by the CloudBees Jenkins Platform.

“A few years ago, visionaries at Capital One began embracing CI and CD practices, and those practices have been growing organically ever since,” says Brock Beatty, director of software engineering at Capital One. “We see numerous advantages to CI and CD with the CloudBees Jenkins Platform, including shorter time-to-market, improved quality through repeatable processes and a reduction in the cognitive load on our developer community, who are now focused more on the software they are creating and less on the pipeline that creates it.”

CHALLENGE: INCREASE AUTOMATION WHILE ENSURING STABILITY, SCALABILITY AND SECURITY

The initial grass-roots adoption of CI tools and practices at Capital One emerged from the company’s culture of encouraging innovative thinking and a recognition among some teams that there was an opportunity to employ automation to improve software quality and speed development. Teams began using Hudson, Jenkins and other CI tools to capitalize on this opportunity.

As CI practices were becoming increasingly mainstream, Capital One formed a shared CD tools team as part of a restructuring of the company’s technical organization.

“Our team was created to support thousands of developers by providing a solid service that helps them deliver software applications more quickly and in a more agile fashion,” says Beatty. From the outset, the team had several criteria for the platform that would underpin the service they provided. “From a platform perspective, the technical makeup had to be extremely flexible to meet a wide variety of use cases, but it also had to be both stable and scalable,” Beatty notes.

Further, as a financial services company, Capital One needed its platform to be secure itself and promote improved security for the applications built with it. “Security is absolutely of the utmost importance to us as an organization,” says Beatty. “We value developer autonomy and at the same time we have an obligation to our brand, to our customers and to regulatory agencies to ensure that security is well managed.” With these objectives in mind, the new shared CD tools team sought to implement CI and CD practices that supported security practices without stifling developer innovation with reference to speed and security.

SOLUTION: ESTABLISH A PLATFORM FOR CI AND CD TO DELIVER QUALITY SOFTWARE WITH SHORTER TIME-TO-MARKET 

The shared continuous delivery tools team at Capital One established an internal platform for automating builds, testing and deployment based on CloudBees Jenkins Platform.

The team used the Role-based Access Control (RBAC) plugin to implement an authorization strategy in which security roles are assigned to groups of users. “We map our users to Active Directory groups,” Beatty explains. “We then use the RBAC plugin to map the Active Directory groups to Jenkins groups. Using the Folders plugin, we associate the Jenkins groups with folders that are ultimately mapped to the infrastructure.” 

After the initial platform rollout, the team made incremental refinements over time to improve operational efficiency. They decoupled their Jenkins agents from the masters, giving them the ability to scale masters and agents independently using Amazon Web Services (AWS). The team began using CloudBees Jenkins Operation Center™ to manage multiple masters and reduce administrative time by ensuring plugin compliance and version consistency with update centers.

The team also settled on a single default environment for all groups. “In the beginning, we were so customer-focused that we ended up with numerous one-off environments all with unique configurations,” says Beatty. “In an organization of our size, this can lead to massive headaches. Now, all of our worker agents are the same – they all have Docker, they all have the same Java version. It’s much easier to maintain and keep stable.”

It’s not unusual for changes to tooling and development practices to be met with resistance and skepticism by development groups that prefer the status quo. The culture of innovation at Capital One mitigated this reluctance, as did the approach of Beatty’s team.

“We’re lucky in that we work with a lot of really good engineers who are passionate about what they do,” he explains. “Instead of asserting our opinion, we began by understanding their problems. As we began to solve those problems with Jenkins, we found the engineers were eager to partner with us.”

Atop the infrastructure the shared continuous delivery tools team created with Jenkins, each Capital One division creates a common implementation pipeline layer. Using the GitHub plugin, Pipeline plugin and Pipeline Multibranch plugins among others, groups have automated the progression of code commits, builds and functional and non-functional tests throughout five highlevel activities: development, QA, integration, performance and production.

“For these five high-level activities, we’ve defined 16 gates and 29 behaviors that embody how pipelines should function in a mature DevOps plan – and Jenkins is at the core of those pipelines,” says Beatty.

The team is working on a dashboard that will show up-to-theminute progress of projects and the more than 27,000 jobs that run on CloudBees Jenkins Platform at Capital One. “The ability to see, in real-time, our various pipelines as they progress through different Jenkins stages will be a key part of a DevOps dashboard created to increase transparency of our CD pipelines,” Beatty explains.

The team also has plans in place to make it easier for individual development groups to make changes as needed to meet their project’s specific requirements. To add a new plugin, for example, a group will fork the existing environment, update the plugins file and begin working. Issuing a pull request will set up a Jenkins process that builds an environment with the needed plugin, and the team can begin using it right away. Later, if the shared CD tools team agrees that the plugin should be merged into the main environment, they will do so and continue to support it from then on.

RESULTS 

90% of pipeline automated. 

“Jenkins is at the heart of our automation, from GitHub commits to deployment with Ansible, Terraform or Chef,” says Beatty. “From commit to deploy, Jenkins manages nine-tenths of our pipeline.”

Deployment frequency increased 1,300%. 

“With CloudBees Jenkins Platform we’ve created a service for our developers that’s scalable and stable,” says Beatty. “As a result, the time they would’ve spent managing infrastructure is now spent developing business applications. That has contributed to our ability to increase deployments from a couple per year to now deploying every two weeks.”

Quality and security ensured through repeatable processes. 

“With CloudBees Jenkins Platform, we’ve implemented consistent, repeatable processes that run through automated regression suites every time a developer commits or before each deploy,” says Beatty. “That repeatable process not only produces better quality, more secure software, it enables us to deliver it faster.” 

 

 

Capital One developers use a variety of tools for CI and CD, but with 27,000 jobs running on it, the CloudBees Jenkins Platform is by far the most heavily used. That fact alone leads me to believe it is either the easiest to use, it has the most community love or it is simply the best from a technology perspective.
Brock Beatty
Director of Software Engineering, Capital One