CARFAX

CARFAX Reduces Risk and Improves Automation with CI, CD and CloudBees Core

SUMMARY 

CARFAX builds and deploys a new billing and finance system using CloudBees Core, hitting a business-critical deadline. 

CHALLENGE 

Rebuild and replace a decade-old monolithic finance and billing system, without additional resources, to enable hundreds of millions of monthly transactions. 

SOLUTION

Use CloudBees Core and Jenkins Pipeline to automate build and deploy processes that meet audit requirements. 

RESULTS 

» New build set up times reduced by 96%, from one day to 15 minutes 

» Ramp-up time minimized 

» Manual processes automated 

» Pipelines created 3X faster

PRODUCT 

» CloudBees Core 

» Professional Services

Every year millions of car shoppers worldwide trust CARFAX to provide them with accurate vehicle history information to reduce the stress and risk involved in purchasing a used car. Software is at the heart of the CARFAX offering, assembling 20 billion vehicle background records from more than 112,000 data sources to continually improve and expand its database and making reports available to dealers and consumers quickly, easily and accurately.

Data is the difference between two cars that look the same on the outside but are completely different “under the hood.” The CARFAX applications services group develops back-end billing and financial systems in support of the company’s consumer-facing and dealer-facing systems. The software developed by this team captures CARFAX transactions and bills them back to dealers. Other applications interact with data providers, like motor vehicle agencies and repair shops, to help fix errors and discrepancies that could mean hundreds or thousands of dollars difference in a car’s value.

The CARFAX team was tasked with replacing a decade-old billing and finance system while rebuilding quote functionality, creating a product catalog and maintaining all transaction monitoring and management. With the magnitude of the project, the team knew they couldn’t use the existing approach and process to accomplish their goals, especially without additional personnel. Instead, the team adopted continuous integration (CI) and continuous delivery (CD) practices powered by CloudBees Core to deliver this business-critical update on time. 

“At CARFAX, we are focused on doing more with less. Because our team is small we have to take advantage of automation so our developers can focus on delivering business value,” says Matt Cervarich, director of application services at CARFAX. 

To tackle the project, CARFAX deployed CloudBees Core and engaged professional services to quickly remove and clear hurdles. “CloudBees Core was the tool that enabled us to leap to doing things the right way, so we could meet our deadline and satisfy the auditors,” said Cervarich.

A DevOps approach using CloudBees Core for CI and CD helped the CARFAX team create pipelines three times faster and reduce build set up times by 96 percent.

CHALLENGE: DELIVER NEW BUSINESS-CRITICAL FINANCE SYSTEM ON TIME WITHOUT MORE RESOURCES 

When the original billing and finance system was built, CARFAX was handling a few million transactions per month. That’s now hundreds of millions per month, necessitating the need for the new system. The complex project was coupled with an unmovable deadline. 

“Hitting the deadline was not optional,” says Cervarich. The new system had to be deployed in its third quarter or it would be pushed out another six months. In addition, the team had to be ready for questions from auditors about the controls they had put in place. 

The CARFAX team was making a wholesale change in philosophy, approach and process. The development landscape at CARFAX had been far different. Previously, monolithic systems were developed using a combination of ColdFusion and PL/SQL stored procedures. Today, that monolithic software has been broken down into 18 components, most of which are running in the cloud with Heroku and Amazon Web Services. 

As the team transitioned to using technologies like Grails and the Spring Boot framework to modernize existing systems, they began to notice conflicts and inefficiencies stemming from their manual build and deployment processes. 

“A lot of our builds and deploys were done with Ant scripts that we ran manually,” recalls Eugene McKissick, solutions architect at CARFAX. “We ran into issues because we had multiple people working on the same code, going to the same place at the same time as another person’s build.”

Eventually, the Ant-based system broke down and the team set up an alternative based on Gradle and Jenkins. The application services team, which represents about 10 percent of the CARFAX development organization, was among the first to begin using Jenkins. 

“Most teams were deploying one system to one place, but we needed custom automation that we could tailor ourselves,” says Cervarich. The requirement was to build and deploy five or six different systems with separate deployment paths and different security models while integrating with packaged apps and internal services.

Over time, the complexity of the builds increased as did the maintenance burden. “We had at least three Jenkins servers installed, and as the project load increased, maintaining them was time consuming,” says Becca Gaspard, software developer at CARFAX. Keeping Jenkins versions, backups and plugins up to date and applying patches was taking valuable time away from higher-value tasks. 

With a looming deadline, CARFAX chose CloudBees Core to improve automation via CI, begin adopting CD practices, reduce maintenance overhead and deliver the new billing system on schedule. 

SOLUTION: USE CLOUDBEES CORE AND JENKINS PIPELINE TO AUTOMATE BUILD AND DEPLOY PROCESSES 

To get a fast start, CARFAX engaged CloudBees professional services for training and assistance with the initial deployment. “Having Jenkins experts come in and spend time with us derisked the learning curve,” says Cervarich. “I could time-box the task for two people who had lots of other tasks on their plates. We had the server up and running within hours on the first day, and we were off and running with setting up our deploys.”

Beyond assistance with the initial set up, the team also received training on Jenkins Pipeline and best practices for creating pipelines. “In the first week, we learned how to write Jenkinsfiles. In the second week, we got deeper into creating a shared library and deploying to AWS with the AWS Command Line plugin,” says Gaspard. 

Building upon their early momentum, the team created pipelines for Docker, Amazon Elastic Container Service (ECS), AWS Batch and AWS Elastic Beanstalk, using a pipeline as code approach in which all pipeline definition files were versioned in source control.

“Those first pipelines covered about 75 percent of everything we needed,” says McKissick. “Before we could schedule training, our team members were already using the pipelines we had created as templates to create their own. And they just worked.”

To reduce maintenance overhead associated with unstable plugins, the team now vets the Jenkins plugins it uses with the CloudBees Assurance Program, which verifies compatibility with each other and specific Jenkins instances. They also use CloudBees Jenkins Operations Center as a central operations console for managing Jenkins masters and plugins.

For the multitasking developers on the application services team, a key advantage of CloudBees Core is its ability to scale quickly. “One Jenkins box was never enough for all of our teams,” said McKissick. “In the past, we had to manually request and create a new box for each new domain. With CloudBees Core, we can spin up multiple workers to handle those multiple domains while still having one instance. That’s an enterprise-level capability that we have to have.”

Having successfully delivered the new billing system on time, Cervarich’s group is planning further improvements to their environment, including integration with ServiceNow Change Management and Incident Management for Jenkins projects. Using Jenkins pipelines to automatically track software changes and create incident tickets will enable the group to further improve its controls and the auditability of its build and deployment processes.

“We’re still in the midst of our CI and CD story right now, but we know we’re on the right path with CloudBees,” said Cervarich. “For anyone getting started, my advice is to find a project and jump in so you can see the value. Don’t try to roll everything out across the enterprise all at once. We started with one project; we’re learning how it all works, making improvements and creating a new way to build and deploy that other teams will want to use.

RESULTS 

New build set up times reduced by 96% (from one day to 15 minutes). 

“In the past, setting up a new application meant finding a job on Jenkins that was similar to what you wanted, cloning it, changing out all the custom information and then hoping it all worked,” says McKissick. “If it didn’t work, it could easily take a day. Now that we’re using CloudBees Core, we simply take an existing Jenkinsfile, tweak it for our AWS CloudFormation template, and within 10 to 15 minutes, we have it building and deploying to the environment.” 

Ramp-up time minimized. 

“It was clear that the deployment paradigm we had been using wasn’t going to work for our new billing system unless we added more staff,” Cervarich says. “With CloudBees professional services, we were able to move quickly and get on a path that made sense from the start, instead of muscling through months of doing things the old way.”

Manual processes automated. 

“If we had continued with our old process, showing the controls we had in place to auditors was going to be very manual – running database queries to find anything that was touched and then manually trying to tie the results together,” says Cervarich. “Now we have automation tailored to our needs with CloudBees Core and we can quickly and easily show compliance to our auditors.”

Pipelines created three times faster. 

“Having help from CloudBees experts on pipelines was a big advantage,” says Gaspard. “We worked with CloudBees for two weeks; we could easily have spent triple that time if we were trying to figure deploys out on our own, and the results would likely not have been as consistent.”

CloudBees Core was the tool that enabled us to leap to doing things the right way, so we could meet our deadline and satisfy the auditors.
Matt Cervarich
Director of Application Services