By Shawn Ahmed, senior vice president and general manager, Product and Solutions Marketing CloudBees
When you bring customers together, magic happens. It’s one of my favorite things to do, especially when I can sit back and watch the conversation unfold.
I recently had the opportunity to speak with three CloudBees Feature Management customers. Each embarked on a journey to improve release cycle times to deliver innovation to market faster, but a lack of controls and testing kept getting in the way. Bugs slipped through the cracks, impacting uptime and their end customers. Rollbacks were painful and developer morale was hit too. What all three of these thought leaders figured out is that while innovation is critical, it involves a delicate balance of minimizing release risk.
They had a lot to say about the challenges each faced before discovering the power of feature flag technology. It’s my hope in sharing their stories that the insights and lessons learned can have a ripple effect that helps many others. Here’s a window into our conversation.
How have Feature Flags Changed the Way You Manage and Release Features?
It’s a common scenario in enterprises: development teams struggling with release cycles, testing and rollbacks. According to Oren Ben-Shaul, vice president of software engineering at PeerStreet, used CloudBees Feature Management in a previous role. "We aimed for a two-week release cycle, and each release was deployed to millions of devices, covering 11 different families of devices. We were spending so much time on testing; we were missing our release dates. On top of that, if we had an issue, rollbacks were taking as much as 72 hours, which was negatively impacting our customers.” Ben-Shaul and his team implemented CloudBees Feature Management and improvements came quickly. "We started to see improvements in our release processes and governance, and it became much easier to release new software to production.”
Brydin Eckert, product manager at Macmillan Learning explained that releases were very stressful. "It was a true nightmare to be honest,” she said. "We had 10-hour maintenance windows with all-hands-on-deck. It was a really expensive, challenging process, especially when you’re trying to tie together your back end, your front end, mobile clients and web apps,” Eckert said. After Macmillan Learning started using CloudBees Feature Management, releases went from a 20-person, all-night work effort to a one-person, one-hour time frame. Rollbacks are now executed at any time without a maintenance window.
Has Feature Flagging Positively Impacted Your Operating Model and Culture?
Russ Wilson, director of software development at Coyote Logistics, said they often had to use "the oh crap button” to turn off code releases. According to Wilson, "The cultural shift has been massive. We have well over 100 feature flags in production right now, and it grows with every sprint we do.” He also explained how CloudBees Feature Management has become a safety net for the development team. "No one wants to get that call at 2:00 a.m. and then get asked why we didn’t use a feature flag,” Wilson said.
Use of feature flags has also improved morale. "We’re not nearly as stressed about releasing so many different components,” said Eckert. "It’s been a great opportunity for the dev and product teams to collaborate and make sure we’re all on the same page. It’s given the product managers more ownership in the development process. Now, it’s now the product teams’ responsibility to flip the flags, which takes some of the stress off the developers.”
How Do You Use Metrics and Data to Overcome Challenges?
All three customers agreed that before CloudBees Feature Management, their teams had difficulty tracking specific activities such as the length of time to complete a pull request. According to Ben-Shaul from PeerStreet, "I am often asked how the engineering team is doing. Before, I could provide anecdotal answers, but I had no data to back it up. We couldn’t easily tell where we had bottlenecks and how to improve the KPIs from sprint to sprint. Now, we’ve shifted from storytelling to data-driven continuous improvement cycles, and that’s what we wanted to achieve.”
Eckert from Macmillan Learning has seen the same evolution. "We’ve brought more data to our processes,” she said. "We look at product data to understand how it’s being used, which drives new product features and helps us better understand how our organization is functioning. This information dictates how we optimize our team for success.”
To learn more about CloudBees Feature Management, visit https://www.cloudbees.com/products/feature-management.