This post is written by Kurian John, Systems Development Manager and Karan Khera, Hands-On Software Architect at CareFirst FEP Operations Center.
When a company undertakes a DevOps transformation, it is often described as a journey. Like any journey, a DevOps one has a starting point and an expected end point. The path between the two is rarely a straight line, and in fact, the end point is more of an evolution. Each company’s path is unique.
At CareFirst FEP Operations Center, we operate in a highly-regulated industry, so we’re not pushing to production tens or hundreds of times of day like some of the industry trailblazers – nevertheless we forged our own path with continuous integration, continuous delivery and DevOps to meet the specific needs of our customers and our business.
At CareFirst FEP Operations Center, we have focused our DevOps journey in terms of reimagining software development. Adopting continuous integration, continuous delivery and DevOps is not something that happens overnight even for relatively small companies. As the largest health care insurer in the Mid-Atlantic region, serving 3.2 million members, we are far from small. In fact, CareFirst FEP Operations Center is a technology partner for the Blue Cross and Blue Shield Federal Employee Program which serves 5.3 million members nationwide, so when we reimagine software development, it is a years-long, large-scale commitment with the opportunity for transformative impact.
The factors that led us to begin our DevOps journey will likely sound familiar. First, the number of our ongoing software projects had been growing rapidly – in the past five years new development initiatives had more than tripled compared to the previous five years. Complexity had increased as well. Past projects were mostly mainframe oriented, but now they incorporated distributed platforms, big data, mobile, microservices and more. As everything scaled, there were more hand-offs, more unplanned work and more stress put on teams. On top of this, we wanted to reduce time to market for our software programs and set a goal of cutting software release cycles by 83%, from six months to one month. We needed to push new features out faster, more predictably and more consistently.
Among the first steps we took was a jump to agile from waterfall development. While the move was important, we soon found that delivery agility requires more than embracing Scrum. There was still a great deal of friction in the process, lots of manual steps and lots of waiting. Deployments were taking up to two hours. In part, this was due to large, monolithic applications that we were working with. We have since started decoupling these applications in earnest. In just two years, we have gone from zero microservices to 30.
Focusing on the application stack was only the start. Solving the delivery problem holistically required us to apply systems thinking, and to focus on every aspect of delivery. Building on early successes, we then introduced four practices at the enterprise level: continuous integration (as a first step toward continuous delivery), test automation, deployment pipelines and infrastructure as code.
We kicked off our CI/CD initiative by improving our tooling, including version control and code review tools, and followed that with an overhaul of branching processes. The big shift, however, was centered around Jenkins. Whereas some teams had been using Jenkins for basic builds and unit tests, we decided it was time to make Jenkins a “first class citizen.”
As more and more of our process was incorporated into and automated with the CloudBees Jenkins Platform, our developers
saw their overhead drop sharply. Developers were no longer spending 30 minutes to two hours just trying to get a deployment to take place.
Instead, they could use that time to actually write code and deliver business value. Over time, we saw teams that used this platform
not only develop code faster and with better quality, but the developers were actually happier.
We gradually took full advantage of the CloudBees Jenkins Platform including enterprise support and the Beekeeper Upgrade Assistant for verifying plugins, CloudBees Jenkins Operations Center for centrally managing masters and enterprise plugins including Folders Plus and Role-based Access Control to facilitate self-service.
As our teams gained momentum, two-hour deploys were streamlined to a single button click. Transparency increased, and feedback loops were
strengthened. Teams that embraced DevOps practices were seeing cycle times that were five to ten times faster than before,
with concomitant increases in throughput and confidence in the work they were producing.
A successful DevOps journey is as much about culture as it is about tools and processes. From the start, we sought to build a culture of trust and mutual respect both within and across teams. Increased transparency and empowerment enabled by self-service helped foster that culture and gave teams a foundation upon which they could move forward and solve problems together.
In just the first two years of our DevOps journey, we have more than 60 applications in deployment pipelines,
all of them with the ability to be push-buttoned to production.
We’re not resting on what we’ve accomplished thus far, however. CareFirst FEP Operations Center teams are currently pushing forward with the journey, actively working on improvements to reduce technical debt, increase test coverage, bring more legacy applications into the pipeline and nurture a culture of experimentation and innovation. Our journey is far from over and we’re enjoying the ride.
Want to learn more about CareFirst FEP Operations Center’s journey to and through DevOps? Check out this webinar recording, “Reimagining Software Development at CareFirst FEP Operations Center.”
Kurian John, Systems Development Manager
CareFirst FEP Operations Center
Karan Khera, Hands-On Software Architect
CareFirst FEP Operations Center