In a recent episode of Continuous Discussions (#c9d9), we were joined by expert panelists who discussed how to best collect and leverage the data that is generated from the software delivery pipeline. The panel included: Juni Mukherjee, a uthor of Continuous Delivery Pipeline - Where Does It Choke? and The Power Of Continuous Delivery In DevOps; Manuel Pais, DevOps and CD consultant; Mirco Hering, a passionate Agile and DevOps change agent trying to make software delivery a more humane place to be; Paula Thrasher, director of digital services at CSRA; Torsten Volk, managing research director for hybrid cloud at EMA; and CloudBees’s Sam Fell and Anders Wallgren. Continue reading for some of their top takeaways!
It’s important to discuss metrics before jumping in and starting to measure, suggests Thrasher : “When I'm talking to teams about what to measure I have some standard skits, and I like to hear from the team about what problem they're trying to solve before jumping in with, ‘You've got to measure all these things.’ You can go absolutely bonkers with numbers and create chaos without actually solving for things.”
Focus metrics around bottlenecks as well, advises Pais : “What I recommend is besides those core metrics that give you an overview of the state of your delivery of your software operations, also look at the value stream. Look at the bottlenecks and come up then with a metric, which acts as a proxy to evaluate if you are getting better, are you reducing that bottleneck, and working to remove it."
Put metrics in place that will help you find defects earlier, says Wallgren : “Earlier is usually better for finding defects. The root cause part is pretty important because otherwise, you're just playing whack-a-mole with the release side.”
The most important metric according to Mukherjee ?: “’Check-in to go live’ is how long does it take for a check-in to go live in the hands of a customer? That is pretty much the basis of everything. Check-in to go live is a subset of feature lead time, which is how long it takes to get a feature out. And then feature lead time's a subset of concept to cash, which means how long it takes for a concept to actually make money. But again, going back the heart of it is really check-in to go live.”
Pipeline analysis should give people a shared context of what the dashboard is telling them - #MTTR , for example, should mean the same thing to everyone @manupaisable #c9d9 #devops pic.twitter.com/oW8cK5fVE8— Electric ClouVokd (@electriccloud) March 6, 2018
Everyone in the organization should be clued in to what is being measured, per Volk : “Keep each other honest. Have the presales guys, the operations guys and the sales guys in the meetings where we discuss the gates. That is absolutely critical no matter what the metrics are.”
It’s important to evaluate the technical metrics, too, says Thrasher “There's cycle time metrics that can tell you something about how the team's performing in the work, but there's also some value in the tech metrics of time to test, test efficiency, cyclomatic complexity, etc. All of those things can be really valuable in teasing out a core problem that you're trying to solve for.”
Visibility and understanding of the metrics that matter most to the business is key, says Pais : “Regardless of the dashboard, the most important thing is to have a shared physical view of the core metrics that everyone gets to look at it and discuss. Understanding the business metrics and having a visible dashboard sparks conversations and important discussions.”
While dashboards can provide good visibility into data, it’s important to interpret it accurately and keep the metrics that matter current, suggests Hering : “Because you have dashboards you basically have an executive view where you put the metrics that currently matter. On a day-to-day basis you're looking at different things, and things can change. You’re looking at what is the current bottleneck. But that previous bottleneck might well become the bottleneck again in a months’ time.”
Often times the C-suite doesn’t immediately see the value in tracking technical metrics, so bake business metrics into your pipeline as well, advises Mukherjee : “I like to trend the business KPI’s along with tech metrics. So, if somebody's bothered about a number of downloads of a game, put it on the same canvas. Make this data available for everybody.”
With disparate teams working with disparate systems, it’s critical to keep metrics easily visible and explained, per Volk : “If there is no culture of DevOps, then everybody wants to guard themselves and wants to hide as much as they can. They won’t necessarily integrate their systems with their overall system because all the dirty laundry comes out. So, no exceptions allowed. If the metrics go to the dashboard there's no explanation needed from somebody.”
Best practices on test metrics, from Hering : “Everyone is talking about automated tests and code coverage. As this increases, what happens to our defects that we find in integration testing or in production? We can actually see at what percentage, an increase doesn't make an impact anymore. This allows you to make better economic decisions.”
It can be time consuming to get all your systems to talk to one another, but CloudBees Flow can help, per Wallgren : “One of the things we do in CloudBees Flow with DevOps Insight is collect all of the data, not just what see in the pipeline but talking to Jira, Git, etc. so that you can correlate and then hopefully find some causation.”
Watch the full episode:
Want more Continuous Discussions (#c9d9)?
We hold our #c9d9 podcast once a month on Tuesday’s at 10 a.m. PT. Each episode features expert panelists talking about DevOps, Continuous Delivery, Agile and more.
DevOps Insights with CloudBees Flow
Learn how you can automate data collection and from all your point-tools, environments and tasks throughout your pipeline and gain instant visibility into the metrics across your entire release with CloudBees Flow.
Stay up to date
We'll never share your email address and you can opt out at any time, we promise.