Gary Gruver - An Adventure into Agile

Andre Pino: Welcome to DevOps Radio. In today's episode we're joined by Gary Gruver, author and president of Gruver Consulting. Previously, Gary was the director of laser jet firmware development lab at Hewlett Packard and a vice-president of quality, release and continuous delivery at macys.com. Welcome Gary.

Gary: Thanks. It's great to be here.

Andre: So Gary, you've written a couple of books, you've done some amazing DevOps transformations; tell us a little bit about your background and how you got to where you are today.

Gary: I guess I've always been into continual improvement and trying to get things better. Started as manufacturing process engineer, Kaizen before – Kaizen was just coming out in manufacturing – and I moved over into product development with the laser jet business and I sat on the outside of the firmware organization for a long period of time and was frustrated in terms of our ability to deliver new products and new capabilities because the firmware organization was a bottleneck for the organization for years. In 2007 I got an opportunity to take over leading a large organization – 800 people worldwide – doing embedded firmware development for the laser jet printers and I figured there needed to be a better way and we were in the middle of a re-architecture and so I stepped in and really started every day trying to figure out how to get more effective, how to get more productive in terms of how we developed software. And went on a journey – because this organization had spent two decades trying to spend their way out of a problem, hiring engineers around the world and we needed a better solution, then 2008 hit – downturn in the economy – we had to figure out how to get it done with less people and we drove a fairly continuous improvement to get the new architecture out and get it done because we had to, to get the business working.

Andre: So in your books I noticed that a lot of what you talk about is that in order for a DevOps transformation to be successful there has to be a business driver that everyone can really relate to. What was the business driver at the HP printer unit?

Gary: At HP the firmware had been the bottleneck for the business for two decades. We couldn't add a new product to our plans and we couldn't add new features or capabilities to the printers without checking with firmware and all too often the answer was no. So our objectives was we no longer wanted to be the bottleneck for the business and we wanted to free up capacity for innovation and go after that piece. But it drove a focus for us to do a 10X improvement in productivity and my co-author would say we had 10X in terms of number builds and number of lines and that sort of stuff, but from a business result I think it's easy at 2 to 3X improvement productivity. And as I go around working with organizations around the world I see more organizations doing development the way we were doing it before the transformation then after the transformation, and software's become so important. You know, in the laser jet business we started with competing based on print speed and resolution and the next new product and that's why customers were out there buying them. But after a while you only need so fast a printer and so high a resolution on your desktop and we needed to compete with authentication and all these firmware features, and I've seen that happen across the industry as company after company needs to start competing on software and then becoming software companies and they've got to figure out how to get good at it.

Andre: That's great. So, can you walk us through some of the implementation challenges or the transformation challenges that occurred at HP?

Gary: Sure. It was – you know, we started by just trying to bring up this new architecture and then previously we'd gone into this effort of when the group had done it before they'd gone into pre-arranged plans and spent a lot of time planning in detail. And I had a program manager come up to us, "Let's just do kind of what we need done in the next month." So we started writing that down and figuring that out and some people wanted long-range plans and we were just bringing up the new architecture and we went into it month by month trying to figure out what we could get done and what we couldn't get done and it just seemed like natural common sense to me. And then our process was always trying to continuously improve and it felt like a lot of what I did in manufacturing was a process engineer. It wasn't until later that I realized Mike was using some concepts he started to call agile and all that time I just thought I was doing common sense and it was really agile principles being applied at scale.

Andre: So in many ways, a software project requires a lot of project management skills. Project management's been around for many years in many industries and many companies got very, very good at it. What makes software delivery so different?

Gary: There's three things that's fundamentally unique about software. The first one is that it's unlike anything else you do. If you're going to build a new manufacturing plant, if you're going to build a new department store if you're going to build anything like that – you've done it before – you're going to maybe make it a little different size, you're going to make it a little different shape, but project management applies to that reasonably well because you can estimate the task and you can know what it's going to take and you can do that over a period of time. So it's easier to accurately project what's going to get done. The other thing is if you get it wrong with most of traditional hardware, mechanical project type of things it is extremely expensive and takes a long time to change. Software, if you're developing it right with a good deployment by applying, can be very flexible but can be very responsive to change and doing that. And then the third thing that's unique about software is as an industry 50 percent of everything we developed is either never used or doesn't meet its process intent. So if you were going to use traditional, what we would call in software waterfall planning, which is project management that works for everything else, what you end up doing is you take your most flexible asset that's your most valuable thing, which is where companies are starting to differentiate, and you lock it into committed plans to deliver features that won't make their business intent or provide value or even be used, it doesn't make sense, but everywhere else in your organization waterfall planning is used because it makes sense for those things. And if you get software-focused companies that are software from the ground up and you have software people: they get that. But the thing that's different is software has become embedded in how all these different companies compete and they want to manage it like they manage everything else and if they do they're not set up for success.

Andre: Just doesn't work, does it?

Gary: And the biggest change that I'm trying to drive is how do you help executives across the industry get that so they can run it as well.

Andre: That's interesting.

Andre: Gary, you've written two books so far, can you tell us about them?

Gary: Yeah, the first one was really a case study. It was after we finished the transformation at HP. We felt like it was such a big fundamental breakthrough for the business we had to share what we'd done with everybody else in the industry, and it's A Practical Approach to Large Scale Agile Development written by myself and Mike Young and Pat Fulghum, and it really captured the journey and a case study of what we did at HP. And it didn't sell a lot of volumes and it wasn't out there a lot but a lot of feedback that it's one of the better books that was written about a transformation like this case study in the industry. If you listen to a lot of people speak it's referenced by a lot of groups because, you know, it was done with embedded firmware and apparently embedded firmware is a lot harder than everything else. I guess if I would have known that going into it maybe I wouldn't have tried as hard but it was a problem we had and what we had to get fixed and we started doing it and continuing to improve it. The second one is Leading the Transformation: Applying Agile and DevOps Principles at Scale. And it was really a culmination of everything I learned about what do you have to do for embedded firmware to make it effective and improve its productivity? And also taking that to apply it to the website. As I moved to working at Macy's I got an opportunity to lead the continuous delivery effort there, worked pretty closely with Jez Humble going back and forth learning how to do that, what he was thinking, how to apply a scale and go back and forth. And it's a summation of everything I wish I knew as an executive before I started leading these but didn't know anywhere to go ask. It's easy for engineers to go to conferences and learn how to develop software differently but if the executives don't understand their role and play that role, that spark of enthusiasm is going to get put out by the wet blanket of organizational resistance to change. And you need the executives engaging and to help to figure out what are the things you want to transform and then getting people aligned that we're going to do things different.

Andre: Sounds great. So your key transformations that you led – were at HP printers and then at Macys.com – were there any differences between those two transformations?

Gary: Oh yeah, huge. Well, yes – and the transformations – but just in the technology. You don't have a database in a laser jet printer. I mean you've got a file system and some other things. Deployment's FTP-ing to a printer. I'd never done a SOC call, I'd never done a launch call, I'd never realized the issues associated with deploying into a large database and getting how that works. I learned a ton in the process applying it, but a lot of the principles are the same, right, you want to have the automated testing, you want to have fast feedback to the developers so they become better developers you want to make sure that includes security and performance issues so that you're on top of all those things. So the basic principles apply, and you know the fun thing about Macy's was there's not very often that you get a job where you're able to contribute and teach an organization how to do things different at the same time that you're learning so much. And so that was fun for me to get an opportunity to learn as much as I could at the same time that I was helping to lead the organization in terms of its approach and in terms of how to develop software.

Andre: It sounds like it's quite a learning experience.

Gary: Oh yeah, no, it was great. If I'm not learning I get bored and I want to go try different things, which is really fun for me now because I'm working with a lot of different companies and I'm seeing a lot of common problems, but I'm also seeing the same problems over and over again. So it's easy for me to go, "Well, this is getting solved in this way in a lot of different companies. You might want to avoid this in your journey." And then – but I get to learn about new and unique things every day that different companies are challenged with.

Andre: Nice. Nice. So one of the key principles in your book Leading the Transformation is that really the executives need to get involved and they need to provide the support and create an atmosphere where it's okay to fail and learn and to improve. How do the executives sort of learn where the key challenges are so that they can actually lead that transformation and make those changes?

Gary: Yeah. What you're trying to do is solve the problems of how teams come together to deliver value, not how the individuals work. And lot more of how even the individual teams work. And one of the things that we saw at HP at the end of three years is we had a 2 to 3X improvement in business productivity but we didn't focus a lot on the team effort and what the teams do and the rituals of agile at the team level: we focused on applying those principles. And as we did that, we saw a 2 to 3X improvement in productivity but we didn't see a lot of differences with the team. And what we came to the conclusion is the first order effect is how teams come to deliver value, the second order effect is how the individual teams work in large organizations that are tightly coupled. And if you're going to change how people come together and deliver value across the organization, that's the executive's role and they need to figure out what things we need to prioritize, approve and need changing, and they don't really necessarily know what's getting in the way of the organization. So if you would get into these types of things – you know, I had metrics, I could see every morning where we were and the progress we were making and I could quickly look into that and figure out that, "You know Andre, last night at 11:00 you broke in – you checked in some stuff that broke all the facts tests," and I'd spend my time out in the organization trying to figure out why, what happened, what you'd learn, or you'd signed up for something at the beginning of this iteration that you thought you could do and it wasn't getting done. And I would spend my time out in the organization and my entire management chain was playing the role of investigative reporters where we were spending time out in the organization trying to figure out what was getting done, what wasn't getting done and what was giving the people delay. And you know, I'd walk into your cube and say – and it wasn't, "Andre, why aren't you getting this done?" It was more, "You know Andre, we both agreed this was important," and we both go, "Why isn't it getting done?" And so it wasn't attacking you, it was attacking why it wasn't getting done. And the first time I walked into your cube it was like, "Dude, what are you doing here? Am I going to get fired?" The second time I walked into your cube it was like, "Well, the last time you were here you actually fixed something. Let me tell you what happened this time." And it was all this – and it was that process of understanding what was getting in the organization's way and prioritizing those things and getting it fixed and we'd spend the time getting that out there. And then once you agree you're going to do it differently in a large organization and nine teams agree they're going to do it one way and a tenth team doesn't want to do it –

Andre: So it's breaking down those natural shields that prevent collaboration and an open discussion about what the real problems are.

Gary: And at some point it's – as an executive it's not optional. Like that one team that doesn't want to do it it may become terms of employment. And if you're trying to drive a transformation bottoms-up or peer across you don't have that ability to influence people very well and it's hard to – harder to influence up, which is why I think the biggest thing missing in the industry right now is engaging executives who are willing to spend the time working with the teams, learning how to lead the transformations and getting out there and doing the process. And what I've realized is there weren't any resources for me either – there's nothing out there. And I published the book, which was the second book Leading the Transformation, which was everything I wish I knew before I started doing this. Didn't have a place to get it.

Andre: So it sounds like – you know, I've been to a lot of DevOps and continuous delivery-related conferences and one of the questions I hear over and over again as people want to understand how to get started is, you know, should we start bottoms-up or top-down? It sounds like what you're saying it really needs to start at the top and executives really need to be engaged with this entire transformation.

Gary: I think they need to be chartering it, leading it. It depends on what level in the executive you get to. But if you're going to transform it and you're going to follow through and you're going to make it happen you can't be doing DevOps to do DevOps: it's got to be how they think they make their business better and you've got to start with that piece. So I always start with the business objectives and when you look at what those business objectives are then you start to look at your toolbox and say, "What's the most biggest inefficiencies in my software development process and what tool will help me solve that?" And you go down that path, the continuous improvement, continuing to work through that and taking up one thing at a time and working down that path. And that I think the executives can do. You can implement tools, you can implement changes, you can make some improvements at the bottom up, but in almost every bottoms-up effort I've seen, they make a certain level of progress and then they get stymied and frustrated and they're challenged to do it. And I think it's – you can do the bottoms-up thing and it helps and it does a lot – it gets so much easier if you have that leadership and drive from the executives and coordination, and it's not managed and, "Let me tell you how to do it," and it's not going to be the same for every group but it's, "Let's work with the team to agree on the priorities of what we're going to fix and change and let's learn and adjust together and we'll figure it out along the way."

Andre: So that top-down approach does that mean that you constrain the individual teams and how they do things?

Gary: No, there's guardrails. There may be certain things that you say, "As an organization we're going to improve these things." But you know these things are all about organizational change management and getting fingerprints on it, getting ownership for what you do, and you know if I go tell you to go do it this way and it fails you're going to come back to me and say, "Gee, Gary, I tried but it didn't work. You kind of had a really bad idea." But instead if I say, "Andre, you know, we kind of need to solve this thing in this area. How do you think we should do it?" And you come up with a plan, you're not going to come back to me and say, "Gary, I had a really stupid idea. This doesn't work." You're going to modify and adjust your plan and you're going to do the extra effort to figure out how to make it work and it's that ownership for organizational change management. You know, when we applied the agile principles at HP we applied them at the executive staff level we didn't get focused on the rituals at the team level. And while we didn't see dramatic differences in productivity between the ones that implemented all the agile rituals and the ones that didn't, what we did see is they were empowered to pick how they were going to use it and what they were going to do and they made their approaches successful. So I think that's a key part of it. And, you know, what – we had the priorities that we drew every sort of monthly goal-setting period that we had were those were the top priorities for the organization. If you had something on the list, you know, it was your top priority – if you could help somebody that had one of those priorities on the list you knew it was your top priority. If you didn't have anything on the list you had to deliver then you're okay to work on your team level priorities, and it's good that you do it but I see what happens – in too many organizations is their plan is an aggregate of the teams and it's hard to drive change across the teams so you need this balance of teams having some of their capacity and bandwidth to sort of work on the things that they need to but if you're going to fundamentally move the needle in terms of how you do things across teams there's got to be a set of priorities that are driven by the executives. And your business plan is not an aggregate of all the teams – it's a strategy that you're driving across the company – your transformation plan can't be an aggregate of all the teams: there's got to be some sort of overreaching view across how to optimize your deployment pipeline above that.

Andre: Right. One of the things I found interesting in the book Leading the Transformation was the discussion around the HP printer unit and the before and after views of where they were applying the time and resources. And before the transformation they had virtually 5 percent of their resources available for new innovation: after the transformation they had 40 percent available for innovation. What did that do to the business?

Gary: Oh, it fundamentally broke it free, right? We had never been able to add a new capability or feature to it and they fundamentally – the last I checked you know they were – you know, in the beginning marketing had almost given up asking for new features. The last I checked in with Mike Young, who is still there, they were running out of ideas.

Andre: Wow, that's really interesting.

Gary: Have you ever seen a marketing organization working with a software team that's running out of ideas? And you know part of that is the laser jet business maturing –

Andre: So what you're saying is the software delivery organization was outpacing the business in terms of what it could deliver.

Gary: Of ideas. You know the last I talked to him the engineering teams were getting together to have brainstorming sessions about what they felt like they could do to come up with different ideas and they could go back to the marketing team and said, "What if we did this? What if we did that?"

Andre: Wow.

Gary: Where before they were just completely buried and giving up and just shipping products.

Andre: So the software team is actually getting more integrated with the marketing team and helping to determine what could be good for the business next.

Gary: And coming up with ideas.

Andre: Sure.

Gary: Right? Nobody said marketing got a corner on the market on good ideas.

Andre: There you go.

Gary: My favorite quote from Bill Hewlett when I was at Hewlett-Packard was "Marketing was way too important to be left to the marketing organization." So I think everybody needs to be thinking about how you can solve problems for the customer and help your business be successful. And what happens with this transformation is that there was enough capacity for you to open it up and enough energy for innovation that they were able to come up with, or to that capacity to start spending time thinking about that stuff.

Andre: Sure. So as you go out – I know you do a number of conferences and you engage with a number of companies – what are some of the common things that you see people doing or attempting to do that you think is, you know, not wrong but perhaps not the way – on the right path towards the transformation that they really need to accomplish?

Gary: A lot of them get too focused on rituals or get too focused on tools. Or I guess my favorite one lately in DevOps is you've seen a bottoms-up success story and it's like we have one team that did all these things successfully and they figured it out and they had huge benefits and so now they want to go scale across the organization – they want to go tell everybody else how to do it and say, "You need to use these tools, this process: this is exactly how you have to do it," and you don't get the other people's fingerprints on it, you don't get their ownership and you don't necessarily solve the biggest problem in their deployment pipeline and what they're trying to get out the door. It's that scaling piece. And the other piece is the assumption is if I'm going to do DevOps and I'm a CIO and I decide to do it then I'm going to assign it to somebody. Am I going to assign it to a leader in ops to drive it across the company, am I going to assign it to a release manager to drive it across the company? Where does that fit? And I think it needs to go to the people who are responsible for leading and driving the technology for those businesses and they need to believe this is how they get better at delivering software to engage and drive the cultural changes. Because if you have an organization trying to drive a change into a product team and the product team leadership is saying, "I need you to deliver this stuff. Oh, and by the way, because the CIO or somebody said it, I've got to do some of this stuff to check the box," and they don't truly fundamentally believe that's how they're going to transform the business they're not going to be driving the cultural organizational change management behaviors that need to occur and you're not going to get that alignment and you're going to ask, "Do I work on what the product team said or do I work on the transformation –?" and you get that conflict. If the leaders over that piece of the business agree that this is how they're going to get better and they're going to do things this way and we're going to look at our delivers that we have to the business and within that we're going to interleaf our continuous improvement efforts because we believe this is how we get better to delivery to the business. I think we need to get that focus as opposed to somebody over in the corner trying to drive across the organization – that's what you need – but I see just too many organizations assigning it to somebody to come into staff and give updates on how DevOps is going or how agile's going or…

Andre: Right. What about organizations that actually have a team called DevOps Team? Is that something you see that is beneficial or something that, you know, is opposed to the whole concept of DevOps?

Gary: I think it depends. If you have a loosely coupled architecture where small teams can independently develop qualifying deploy code then the DevOps team doesn't make a lot of sense because what you're trying to get is everybody else – the development and the operation – you own it cradle to crave and you collaborate – and when you've got a team of 10 to 20 people you can do that – everybody can wear a pager, everybody knows a little bit about what's going on and can solve a problem to figure that out – and that team can own it cradle to grave. When you have a tightly coupled architecture that requires hundreds to thousands of people to work together to develop qualify and deploy code – that whole idea of having a developer take ownership for operations, he doesn't know what the 900 other people did and how to do it so he can't do that effectively. And in those situations you need to set up a deployment pipeline, you need to figure out – you know, a developer is going to be responsible for checking in their code and making sure it works, any changes to the environment making sure that's done in a scripted environment making sure that's done, there's an orchestrator that orchestrates the code moving from this end all the way out to a customer, the automated test – who manages and maintains that deployment pipeline in a large organization and who's responsible for making sure the operations people are using the same processes, the same tools, the same scripts in dev and operations and across that pipeline? I fundamentally believe in large organizations somebody needs to set that up and maintain it – call them what you want – call them a DevOps team, call them a release team – I'm not a big name guy – call it Bob if you want to – but somebody needs to do that.

Andre: So your first two books were a great success. What's coming next for you?

Gary: I've been starting to write a little bit. I'm not sure if I've got another book in me. We'll see how it goes. But I'm starting to write it down, and I don't know if it's just collecting my thoughts and clarity or the next presentation out of it but what I'm seeing is in large organizations everybody has a different view of what DevOps is and because of that it's hard to get this thing started. As I go in and work with companies a lot of what I do is kind of investigate and evaluate where they are so they can start with the improvement that's going to have the biggest impact on the business results. And I'm learning a lot through that process and I'm getting clarity in my methodology, but a lot of it really comes down to the deployment pipeline is how code gets checked in and value gets delivered to your customer. And I'm spending a lot of time helping customers get a common view of that piece and putting metrics and things around it so that they can focus in and understand on fixing the thing that's their biggest issue and then going after the next one and continuous improvement. You know, you get into some companies and they'll say, "Wait. Well, wait. I heard DevOps means that you have to have developers push code into production." And that's really good. If you're a really quick, loosely coupled organization with seven people that can own it end-to-end and, you know, waiting a day for approval is the long lead time in your post because you've got all automated testing and you can get an environment quickly that's a good thing to go after. But if you're in a large complex organization where it takes you weeks to get an environment, your automated testing takes weeks to run, it takes you a few more weeks to get your code base and all your defects taken out of it, taking that day approval time out of the process is not significant and doesn't matter, so the cultural change associated with wanting dev pushed directly into production could be met with a huge amount of resistance and it's going to make no significant change because you really should be working on test automation or environments on demand or keeping your code base more stable on an ongoing basis or those different things. So the key is going in and focusing on the things that will improve your business the most – the quickest – and looking at your value chain that way and going down your deployment pipeline. And then too many organizations go to a DevOps conference and come back and say, "I heard DevOps was this. I need to go do this," and everybody in the organization hears something different. And if you have a CIO or somebody come in and say, "I just tried the Phoenix project. Everybody go do DevOps," you end up with this uncoordinated effort that's not necessarily improving things. In a large organization I think getting everybody to look at that deployment pipeline, put metrics around it and have a common view, and you're not going to get that at the individual team layer: you need the executives to paint that view and get the team to come together to agree on where the priorities are – go after that in a systematic manner. In a large organization you may have ten of those different teams that don't have to develop qualifying deployed code together and what they need to do to optimize and fix – may be different for all ten of them. And if you go out in the organization and said, "Everybody's going to do it this way," nine of those teams aren't going to be seeing benefits very soon, very well. And it's going in and customizing your transformation plan based on the issues that the deployment pipeline has for each of those ten things differently and going after and fixing different things. And it's – I enjoy doing that part and I think that's a value and I don't know that there's that many people out there doing that, and do I go through the effort of writing it down in a book or am I just clarifying my thoughts so I can help people better? I don't know. Publishing a book is a pain in the rear and I'm not sure I've got another one in me.

Andre: That's an arduous process at best.

Gary: Yeah.

Andre: So Gary, any closing thoughts?

Gary: Well, thanks for having me. It's great to be out here with CloudBees and opportunity to do this and we've got a lot of these DevOps summits going around the world and I'm looking forward to getting out and meeting new people and understanding what challenges they have and how I can help.

Andre: That's great. It's been great talking to you today. Gary, author of Leading the Transformation: Applying Agile and DevOps Principles at Scale. And I suggest that anybody that's interested in learning how to do that – getting their organization inspired to do that – pick up a copy of the book. Thanks Gary.

Gary: Yeah. Thanks for having me.