Consistency, Faster Deployments, and Less Duplicated Effort: The Immediate Impact of Modern CI/CD and DevOps

Contributed by Rebecca Rickert, DevOps Manager, Ventech Solutions

At Ventech Solutions, innovation is our passion. Our dedicated teams of experts and engineers help our clients deploy high-quality IT infrastructure that advance their missions. We leverage key partnerships and cutting-edge technologies to deliver powerful platforms and products in an ever-evolving IT landscape.

Our visionaries work with private sector and government clients to provide IT solutions that balance speed, efficiency and security.

One of our major customers is a large federal health care organization. As their infrastructure contractor, we provide the servers and the development tools that they need to create websites and applications. We also manage their help desk, on-premise data center and cloud-based data center on AWS.

Ventech Solutions provides support for the enterprise-level tools used by the contractors that design and code these websites and applications. These entities are known as ADOs—application development organizations. Each of the 20-plus ADOs is responsible for a piece of the IT puzzle for our customer. Our mission is to create a DevOps environment that will keep everyone under the same roof and moving in the same direction.

In the past, developers did their own thing. They were all using unmanaged servers, and everyone was off on a separate planet. Compliance was spotty. We had no way of knowing how individual ADOs had configured their DevOps environments. We didn't know whether their plugins and APIs were up to date, or whether their servers were secure.

When it came to managing ADO output, there were a lot of duplicate efforts. We had to recompile and manually deploy every new app and every update to meet contract requirements. We had no way to ensure every app was future-proof. We could not be certain whether all of the ADOs were using current DevOps environments and standards, and therefore, there was no assurance we could still support an app a few years from now. We tracked compliance with tags and spreadsheets, and the process relied on self-reporting, which made it error prone.

We needed to implement some form of version control before we started building new enterprise tools, and the first step was to move all the ADOs to GitHub Enterprise. We now had a lot more oversight, but we were still in the wild west because everyone was running unmanaged Jenkins servers. To this day, I find some such machines that continue to operate.

Once started the process with GitHub, our next step was to adopt an orchestration tool that would give us a general overview of the ADO’s work.

We went with CloudBees CI, the only enterprise implementation of the Jenkins DevOps environment. CloudBees CI combines the flexibility of Jenkins' open-source codebase with the security and stability of CloudBees' enterprise-level tools and support. We could now take advantage of the wisdom of the wider Jenkins open-source community while benefiting from the reliability of CloudBees-verified Jenkins distributions and integrations, a curated library of plugins (including security tools like SonarQube and Nexus IQ) and CloudBees support.

CloudBees CI is clustered and designed for high availability. If an updated app or part of the codebase fails, we can quickly revert to an older version while we troubleshoot. CloudBees CI also makes it easier to share resources. When an ADO builds an app, a function, or code that can be leveraged by other developers working for our customer, we can easily pass it along to them.

When we launched our CloudBees implementation in October 2017, CloudBees was right there beside us. Their Professional Services team sent an engineer to sit with us in a room to help us get everything up and running smoothly. Over four days, we stood up our test and production environments and got all ADOs started on their journey to fully utilizing CloudBees CI.

Our new DevOps environment was fully operational within a week, and so the next step was training the ADOs. We started by introducing everyone to the CloudBees Jenkins Pipeline. This suite of plugins enables the implementation and integration of a continuous development workflow as code. We presented it as a better way to maintain all their jobs and then launched our Deployment Automation Working Group (DAWG).

DAWG met once a week. We'd bring all the ADOs together and go through tips, tricks, best practices, and mini-trainings to pull everyone into using the tool. Since we were on-boarding all the contractors at the same time, we were able to get buy-in on the new rules we'd created, and consistency shot through the roof. Everyone was following the same methodologies, and this helped streamline, standardize, and secure the DevOps process.

This new level of standardization had an immediate impact on code maintenance and support. When every app uses the same framework and is documented using the same set of guidelines, it makes it easier for another contractor to step in and take over a project if the ADO that developed an app is too busy with other projects or has stopped working with the customer entirely.

There's great peace of mind in knowing that we now have a maintenance and succession plan in place for every app and website the ADOs build for our customer.

Knowing what everyone is doing at all times means we no longer have to recompile new apps to ensure they meet customer standards. We now have access to all ADO servers and can go in, look at their builds, read their documentation, verify the plugins, and see that they've done everything correctly.

As a result, we now permit ADOs to deploy their own apps. Since they have automated their deployment process, they can push out new apps and updates faster. CloudBees CI provides a high level of repeatability and consistency because our developers run the same blocks of code every time.

Our ADOs have gone from deploying updates every 3–6 months to every few weeks. Downtime for major updates has been improved from two or three days to less than two hours. One of our ADOs deploys patches every Tuesday and Thursday after 8 p.m. Their downtime is only as long as it takes to reboot the server.

Another benefit of CloudBees CI is the speed with which we can fix issues. Fixes can be deployed up the stack and sufficiently tested at a much faster rate. Often, we find a resolution in a day or two. If it's a particularly thorny issue, we can call on the expertise of the CloudBees technical support team.

Recently, our customer, a large federal health care organization, decided to move all its servers—including all ADO servers—to AWS. We were given just 18 months to shut down everything at a data center in Ashburn, VA and move it to the cloud. There was just one problem. The ADOs working with applications that target the physical data center must remain on-premise.

With CloudBees CI, we've been able to migrate the ADOs whose apps target the cloud to AWS and to also keep the rest at the Ashburn data center until the services they target also move.

The ADOs that have gone to the cloud have reduced their data transfer costs, downtime and server spin-up time. A typical on-premise ADO suffered two to four hours of downtime a couple of times a month. In the cloud, you're looking at twenty minutes once a month, at most.

With CloudBees CI and AWS, we can spin up a new Jenkins server in a matter of hours. It previously would take two weeks to add a new machine at the on-premise data center.

Another win is the ability to manage credentials. We are dealing with a lot of different ADOs at any given time, and individual developers come and go. CloudBees Operations Center allows us to set up service accounts that give ADOs access to all the necessary tools and all the target servers. We can then track what these accounts are doing across multiple tools, including GitHub Enterprise, Ansible Tower, Jenkins, Nexus Repository Manager and CloudBees CI.

If someone makes a mistake that crashes an entire system or application, we now can see what happened and how to fix it.

When you combine these service accounts with automated deployments and the increased speed of the cloud, we have the perfect solution that has allowed Ventech Solutions—and the ADOs—to shift from traditional development cycles to agile and CI/CD.

One of our contractors saved hundreds of person hours during a two-week sprint thanks to automation using CloudBees CI. That, to me, says everything.

CloudBees CI is helping Ventech stay ahead of the curve. Instead of struggling to keep up with developers, we are now asking, "What are you doing next?" Now we can offer a suite of tools that allows them to work smarter and faster, as well as the peace of mind that comes from a DevOps environment that operates as a managed service.

It’s the best of both worlds. We look after infrastructure and security so developers can focus on coding.