WatchGuard is a global leader in network security, secure Wi-Fi, and network intelligence products and services for SMBs and Distributed Enterprises worldwide.
Summary: WatchGuard embraced a DevOps approach with CloudBees CI to deploy high-quality, secure software faster, strengthening its market position and competitive advantage.
Challenge: Establish a new DevOps culture, transition away from waterfall development to spur innovation and speed delivery of a new security product
Solution: Embrace agile methods together with DevOps practices and CI/CD automation powered by CloudBees Jenkins Enterprise
Competitive advantages strengthened
Large-scale projects completed in weeks instead of months
Teams on boarded 95% faster (in 4 hours versus 2 weeks)
Security lapses avoided; enterprise support gained
Product: CloudBees CI
Over the past two decades, WatchGuard has built a reputation for providing enterprise-grade network security appliances and wireless security hardware with an emphasis on simplicity, visibility, performance and extensibility. With its eyes on growing market share and gaining competitive advantage, WatchGuard continues to innovate in ways that will complement its security solutions.
In developing new solutions, the company recognized an opportunity to move away from its traditional waterfall development approach toward one based on DevOps and agile development, continuous integration (CI) and continuous delivery (CD). Powered by CloudBees CI, this new approach has enabled WatchGuard to innovate faster, improve its market position and further cement its standing as a leader in security for small- and medium-sized enterprises.
"We can’t afford to be complacent about what we have already offered; it’s about what else we have to offer our customers,” says Jack Waters, senior vice president of engineering, WatchGuard. "Having a healthy CI/CD pipeline based on CloudBees CI has really helped us stay competitive, maintain the level of quality that WatchGuard is known for and deliver new products with a high degree of confidence.” Jai Misra, vice president of engineering at WatchGuard, agrees that a DevOps approach with CloudBees® solutions is a competitive advantage. "It helps us add more value quickly with an assurance of the quality that is associated with the WatchGuard name.”
"Powered by CloudBees CI, this new approach has enabled WatchGuard to rapidly innovate, which is key to maintaining our position as a leading provider of security solutions for small and midsized enterprises."Jack Waters
Senior Vice President of Engineering
Move Aways From Waterfall Development to Innovate Faster
As WatchGuard began its cloud initiative and began the shift from waterfall to agile methods, management knew that wholesale changes would be required.
"The company had to go through a cultural shift in moving from waterfall to agile,” says Misra. "Agile is not just a process – it’s about architecture and everything that is needed to build quickly, test and release.” Waters adds, "When you’re trying to bring an agile way of thinking to an organization, you need to keep everyone aligned with a common goal: to achieve something meaningful and ensure it is really done and done well.”
To facilitate the transition and the company’s long-term objectives, the CI/CD practices that WatchGuard put in place needed to prioritize scalability, flexibility and security.
Kolby Allen, AWS architect at WatchGuard, explains, "We view CI/CD automation as a way to enforce our security model. At the same time, we wanted a CI/CD tool that was flexible, not one that forced us to do builds its way. And we needed a solution that integrates well with industry standard artifact management systems, one that we can run ourselves and one that could grow with us.”
Beyond these fundamentals, WatchGuard also wanted a solution that enabled them to construct custom pipelines and that offered enterprise-class support. "We needed something that was simple, yet would allow us to create pipelines as complex as we wanted – including pipelines made up of smaller pipelines that we could run independently,” Allen adds. "We also needed professional support behind our CI/CD platform because it’s the core of everything we do.”
Implement DevOps and CI/CD Practices With Flexible, Scalable and Secure Automation
WatchGuard selected CloudBees Jenkins solutions to implement DevOps practices and CI/CD automation, starting initially with CloudBees CI and later upgrading to CloudBees CI.
From the start, Allen and his team embraced an everything-ascode approach, developing Shared Libraries in Groovy and hybrid Jenkins files to construct sophisticated continuous delivery pipelines. All Groovy scripts and infrastructure code goes through the same processes – including versioning, QA and code reviews – that are applied to production microservices.
The team also embraced a security-first mindset and set a goal of using no burned-in or hardwired credentials in their setup. To help achieve this goal and securely manage access to jobs, they used the Role-Based Access Control (RBAC) and Folders plugins together with AWS Identity and Access Management (IAM).
"We have different jobs running at different permission levels. RBAC lets us lock down access to the jobs and then enable fine grained access to them based on an individual’s role and need to interact with the job. For example, some roles have the ability to run jobs, while others only have permission to review the logs,” Allen explains.
Following its initial success with the CloudBees CI, WatchGuard transitioned to CloudBees CI, expanding the team’s ability to fully implement its CI/CD plans with more advanced pipelines.
"The move to CloudBees CI really opened up the next evolution of pipeline for us,” says Allen. "With more executors and auto-scaling of executors, we now have the speed and flexibility to move from single, self-contained pipelines to pipelines that contain other pipelines. Now, if we only want to run unit tests, we can run just that pipeline with fine-grained control instead of running the whole thing to deploy the application.”
As pipelines run, WatchGuard developers use Blue Ocean to track their progress and Allen’s group uses CloudBees Jenkins Operations Center™ to identify the root cause of any issues that may arise.
"All of our developers use Blue Ocean to watch their jobs,” says Allen. "If there’s a failed build, we use CloudBees Jenkins Operations Center to troubleshoot it and determine if it was a tooling issue, an Amazon issue, a developer issue or something else.”
Using CloudBees CI together with Docker enabled the team to create immutable executors custom tailored to specific needs. “We are big proponents of immutable infrastructure. CloudBees Jenkins Enterprise unlocked the ability for us to auto-provision immutable executors instead of having to pre-provision them,” says Allen.
With Docker, the team has created a set of customized images for specific purposes. "Rather than having one controller or executor with every possible framework or tool on it, we can add only what is needed, which helps us isolate problems, minimize potential security issues and use resources more efficiently,” says Allen.
WatchGuard has moved from a single controller with five executors to four controllers with as many executors as needed. The four controllers are set up for development, QA, staging and production environments. "We have fully automated CI/CD up through QA,” says Allen. "We have a separate controller that controls access to staging, which only my team can access. The multiple controller set up we have with CloudBees CI enables us to wall off environments for added security.”
Though limiting access to certain jobs had increased security, it also imposed a burden on Allen and his team, who were responsible for creating all new branches as needed. The team used the Multibranch Pipeline plugin to empower developers to create branches on their own without compromising security.
"Due to separation of duties, we often found ourselves copying and pasting jobs, which was tiresome,” says Allen. "Multibranch Pipeline was a great fit for us because it enabled developers to bring on their branches automatically with minimal effort from our team.”
Like WatchGuard itself, the DevOps team is still focused on the future rather than what they have already achieved in automating CI/CD. "We’re constantly learning more and trying to get better. The pipeline we have now looks very different than the one we had last year,” he says. "CloudBees CI lets us continue being innovative in our build process – because it’s more than just building software. That innovation allows us to be more consistent and more reliable so when we deploy we know exactly what is going to happen, every time.”
WatchGuard’s customers are ultimately the beneficiaries of the software development advantages achieved with the company’s DevOps approach. "With CI/CD, it’s only done if it’s been deployed, it’s being used and all of that has happened seamlessly,” says Waters. "And that’s what we’ve been able to accomplish with Jenkins® and CloudBees and the approach we’ve taken.”
Competitive advantages strengthened.
"If the business hadn’t adopted a CI/CD methodology with CloudBees, everything would be slower,” says Allen. "If you’re not doing CI/CD, you’re not going to be able to respond to the market quickly.”
Large-scale projects completed in weeks instead of months.
“In thinking about how we can deliver better products faster for our customers, we adopted a service-oriented mindset – breaking systems and workflows into smaller pieces that can be delivered or executed quickly with an automated pipeline,” says Waters. “We’re now able to get big things done noticeably faster. Whether it’s implementing encryption or changing the way our back-end databases are set up. Activities that would take months are now taking days to weeks to complete.”
Teams onboarded 95% faster (in four hours versus two weeks).
"We recently brought on an entirely new Java development team, and in the past, it would have taken a couple of weeks to get them up and running,” says Allen. "With CloudBees CI and our commitment to infrastructure-as-code and deployment-automation-as-code, we had the build for their microservice set up in about four hours.”
Security lapses avoided.
"When organizations experience security lapses due to exploits or other issues, it’s often because there was no team responsible or accountable for the issue. And if you’re not doing CI/CD, you cannot respond to these types of events quickly,” says Allen. "With the way that we approach DevOps using CloudBees CI, we built everything around the idea that we, at any moment, can patch, deploy, update and rollback changes with minimal effort and high confidence that what we’re doing is correct.”
Enterprise-level support gained.
"I was tasked with testing all the different systems on the market,” says Allen. "When we got down to it, the support that we have from the account side and from an engineering side with CloudBees, I was able to say unconditionally that we need to make this investment.”