CARFAX has the most comprehensive vehicle history database available in North America, with more than 23 billion records and counting.
Summary: CARFAX builds and deploys a new billing and finance system using CloudBees CI, hitting a business-critical deadline.
Challenge: Rebuild and replace a decade-old monolithic finance and billing system, without additional resources, to enable hundreds of millions of monthly transactions.
Solution: Use CloudBees CI and Jenkins Pipeline to automate build and deploy processes that meet audit requirements.
New build set up times reduced by 96%, from one day to 15 minutes
Ramp-up time minimized
Manual processes automated
Pipelines created 3X faster
Product: CloudBees CI, Professional Services
Every year millions of car shoppers worldwide trust CARFAX to provide them with accurate vehicle history information to reduce the stress and risk involved in purchasing a used car. Software is at the heart of the CARFAX offering, assembling 20 billion vehicle background records from more than 112,000 data sources to continually improve and expand its database and making reports available to dealers and consumers quickly, easily and accurately.
Data is the difference between two cars that look the same on the outside but are completely different "under the hood.” The CARFAX applications services group develops back-end billing and financial systems in support of the company’s consumer-facing and dealer-facing systems. The software developed by this team captures CARFAX transactions and bills them back to dealers. Other applications interact with data providers, like motor vehicle agencies and repair shops, to help fix errors and discrepancies that could mean hundreds or thousands of dollars difference in a car’s value.
The CARFAX team was tasked with replacing a decade-old billing and finance system while rebuilding quote functionality, creating a product catalog and maintaining all transaction monitoring and management. With the magnitude of the project, the team knew they couldn’t use the existing approach and process to accomplish their goals, especially without additional personnel. Instead, the team adopted continuous integration (CI) and continuous delivery (CD) practices powered by CloudBees CI to deliver this business-critical update on time.
"At CARFAX, we are focused on doing more with less. Because our team is small we have to take advantage of automation so our developers can focus on delivering business value,” says Matt Cervarich, director of application services at CARFAX.
To tackle the project, CARFAX deployed CloudBees CI and engaged professional services to quickly remove and clear hurdles. "CloudBees CI was the tool that enabled us to leap to doing things the right way, so we could meet our deadline and satisfy the auditors,” said Cervarich.
A DevOps approach using CloudBees CI for CI and CD helped the CARFAX team create pipelines three times faster and reduce build set up times by 96 percent.
"CloudBees CI was the tool that enabled us to leap to doing things the right way, so we could meet our deadline and satisfy the auditors."Matt Cervarich
Director of Application Services
Deliver New Business-Critical Finance Systems on Time Without More Resources
When the original billing and finance system was built, CARFAX was handling a few million transactions per month. That’s now hundreds of millions per month, necessitating the need for the new system. The complex project was coupled with an unmovable deadline.
"Hitting the deadline was not optional,” says Cervarich. The new system had to be deployed in its third quarter or it would be pushed out another six months. In addition, the team had to be ready for questions from auditors about the controls they had put in place.
The CARFAX team was making a wholesale change in philosophy, approach and process. The development landscape at CARFAX had been far different. Previously, monolithic systems were developed using a combination of ColdFusion and PL/SQL stored procedures. Today, that monolithic software has been broken down into 18 components, most of which are running in the cloud with Heroku and Amazon Web Services.
As the team transitioned to using technologies like Grails and the Spring Boot framework to modernize existing systems, they began to notice conflicts and inefficiencies stemming from their manual build and deployment processes.
"A lot of our builds and deploys were done with Ant scripts that we ran manually,” recalls Eugene McKissick, solutions architect at CARFAX. "We ran into issues because we had multiple people working on the same code, going to the same place at the same time as another person’s build.”
Eventually, the Ant-based system broke down and the team set up an alternative based on Gradle and Jenkins. The application services team, which represents about 10 percent of the CARFAX development organization, was among the first to begin using Jenkins.
"Most teams were deploying one system to one place, but we needed custom automation that we could tailor ourselves,” says Cervarich. The requirement was to build and deploy five or six different systems with separate deployment paths and different security models while integrating with packaged apps and internal services.
Over time, the complexity of the builds increased as did the maintenance burden. "We had at least three Jenkins servers installed, and as the project load increased, maintaining them was time consuming,” says Becca Gaspard, software developer at CARFAX. Keeping Jenkins versions, backups and plugins up to date and applying patches was taking valuable time away from higher-value tasks.
With a looming deadline, CARFAX chose CloudBees CI to improve automation via CI, begin adopting CD practices, reduce maintenance overhead and deliver the new billing system on schedule.
Use CloudBees CI and Jenkins Pipeline to Automate Build and Deploy Processes
To get a fast start, CARFAX engaged CloudBees professional services for training and assistance with the initial deployment. "Having Jenkins experts come in and spend time with us derisked the learning curve,” says Cervarich. "I could time-box the task for two people who had lots of other tasks on their plates. We had the server up and running within hours on the first day, and we were off and running with setting up our deploys.”
Beyond assistance with the initial set up, the team also received training on Jenkins Pipeline and best practices for creating pipelines. "In the first week, we learned how to write Jenkinsfiles. In the second week, we got deeper into creating a shared library and deploying to AWS with the AWS Command Line plugin,” says Gaspard.
Building upon their early momentum, the team created pipelines for Docker, Amazon Elastic Container Service (ECS), AWS Batch and AWS Elastic Beanstalk, using a pipeline as code approach in which all pipeline definition files were versioned in source control.
"Those first pipelines covered about 75 percent of everything we needed,” says McKissick. "Before we could schedule training, our team members were already using the pipelines we had created as templates to create their own. And they just worked.”
To reduce maintenance overhead associated with unstable plugins, the team now vets the Jenkins plugins it uses with the CloudBees Assurance Program, which verifies compatibility with each other and specific Jenkins instances. They also use CloudBees Jenkins Operations Center as a central operations console for managing Jenkins controllers and plugins.
For the multitasking developers on the application services team, a key advantage of CloudBees CI is its ability to scale quickly. "One Jenkins box was never enough for all of our teams,” said McKissick. "In the past, we had to manually request and create a new box for each new domain. With CloudBees CI, we can spin up multiple workers to handle those multiple domains while still having one instance. That’s an enterprise-level capability that we have to have.”
Having successfully delivered the new billing system on time, Cervarich’s group is planning further improvements to their environment, including integration with ServiceNow Change Management and Incident Management for Jenkins projects. Using Jenkins pipelines to automatically track software changes and create incident tickets will enable the group to further improve its controls and the auditability of its build and deployment processes.
"We’re still in the midst of our CI and CD story right now, but we know we’re on the right path with CloudBees,” said Cervarich. "For anyone getting started, my advice is to find a project and jump in so you can see the value. Don’t try to roll everything out across the enterprise all at once. We started with one project; we’re learning how it all works, making improvements and creating a new way to build and deploy that other teams will want to use."
New build set up times reduced by 96% (from one day to 15 minutes).
"In the past, setting up a new application meant finding a job on Jenkins that was similar to what you wanted, cloning it, changing out all the custom information and then hoping it all worked,” says McKissick. "If it didn’t work, it could easily take a day. Now that we’re using CloudBees CI, we simply take an existing Jenkinsfile, tweak it for our AWS CloudFormation template, and within 10 to 15 minutes, we have it building and deploying to the environment.”
"It was clear that the deployment paradigm we had been using wasn’t going to work for our new billing system unless we added more staff,” Cervarich says. "With CloudBees professional services, we were able to move quickly and get on a path that made sense from the start, instead of muscling through months of doing things the old way.”
Manual processes automated.
"If we had continued with our old process, showing the controls we had in place to auditors was going to be very manual – running database queries to find anything that was touched and then manually trying to tie the results together,” says Cervarich. "Now we have automation tailored to our needs with CloudBees CI and we can quickly and easily show compliance to our auditors.”
Pipelines created faster.
"Having help from CloudBees experts on pipelines was a big advantage,” says Gaspard. "We worked with CloudBees for two weeks; we could easily have spent triple that time if we were trying to figure deploys out on our own, and the results would likely not have been as consistent.”