Gene Kim - DevOps by the Book

Andre Pino:You’re listening to DevOps Radio, the podcast series that dives into what it takes to successfully develop, deliver and deploy software into today’s ever changing business environment. Today’s episode features the well-known speaker and author of The Phoenix Project book, Gene Kim. Gene, welcome. It’s great to be talking to you again.

Gene Kim:Andre, it’s great to be with you here today. Thanks for having me on.

Andre:Yeah. Gene, I remember last year when you delivered a keynote at the Jenkins User Conference how everyone was just thrilled to have the opportunity to hear you and to be inspired by your talk.

Gene:Oh my goodness. That was such a fun time for me just because of not only the people and the community that you’ve assembled there, but also to be able to hang out with KK again. I mean, wow. You can’t have an interaction with KK without walking away a great deal smarter, so thank you for allowing me to be a part of that journey.

Gene: You know, so I guess my introduction to the DevOps movement I mean I guess when I was formally introduced into the community – formally – informally – when I got introduced into the DevOps community was at the DevOps days in – on the LinkedIn campus in Mountainview in 2010 and that’s when I got to meet the famous John Allspaw, John Willis and Damon Edwards and it was just such a fabulous experience for me because over the previous 13 years in my study of high performance technology organizations, there was this horrible suspicion that there was something terribly wrong that this downward spiral would occur when DevOps and information security couldn’t really work together as a cohesive unit to solve the organizational goals and to see the success stories that were being created within the DevOps community and this radically different way of working where Dev and Ops and information security were genuinely working together as both this adversary relationship. It was just such an eye opener for me and I think my previous part of the journey had made me ready and so receptive to see the better way when the better way finally introduces itself in the form of DevOps as we know it. And so it’s been such a privilege to work with so many of these people who have been pioneering these principles and practices. In fact so many of those learnings went into The Phoenix Project, but the real goal was always to create another book really where The Phoenix Project was meant to be a narrative that really describes what does a downward spiral feel like whether we’re a developer, a tester, an Ops person, InfoSec and someone from the organization that we serve. The goal was always to set the stage for a non-fiction descriptive book that could actually say, "How do you replicate the amazing transformation that Bill Palmer and team did at Parts Unlimited, the fictitious organization that’s depicted in the novel?" How do we actually replicate that so that the rest of us can replicate those amazing outcomes? That’s what DevOps Handbook is.

Andre: Yeah. So after almost five and a half years, you’re now coming out with that prescriptive book, the DevOps Handbook.

Gene: Yeah. In fact I actually serve – actually the real plan back in 2011 when we’re actually starting the book was actually to have this book come out before The Phoenix Project. So, the goal was not to have that come out before The Phoenix Project and this is just for a variety of reasons and I think the primary reason is that the surface area of the book just kept on growing in terms of the things that we realized we didn’t know. It was just so large and so the book kept on changing shape and finally we had to – The Phoenix Project got released and that journey continued for another two and a half years. So now finally in October of 2016, almost five and a half years after the handbook – DevOps Handbook Project started, it’s finally coming out, so you can imagine how elated I am that that is now finally within sight.

Andre:Well, congratulations. I know how much hard work goes into producing a book. So, The Phoenix Project book, if you can talk about that for just a minute, was a huge success. You must be thrilled with how widespread coverage that book got and leadership that you developed.

Gene: Oh, you can’t imagine. You know it is really – there maybe just to paint the motivation. You know one of the books that has influenced me the most in my professional career is a book called The Goal. So that was written by Dr. Eliyahu Goldratt in 1984 and that was a novel about a plant manager – a manufacturing plant manager who had to fix his cost of due date issues in 90 days, otherwise they would shut the plant down and so you got to see the problems that were happening in every manufacturing plant through the eyes of a plant manager and so that book is really credited for being one of the main reasons how the whole notion of just in time manufacturing and theory of constraints and the whole set of - _____ would fit into there. How that took hold into – now it’s part of modern management practices. And so that book has been integrated into almost every mainstream MBA curriculum and business school curriculum over the last two decades. But there’s this – in the Goldratt lectures and you can hear this in the audios book that he put out called Beyond the Goal where Dr. Goldratt lectures for 6 hours about his 30 year journey and his contributions to the space. He talked about the letters that he would get after The Goal came out of people writing to him saying, "It’s like you’ve been living in my manufacturing plant. I know these characters. The exact problems that we have are those that are depicted in the book, The Goal." And so apparently this has happened to him for going on 15 years, he’s now – unfortunately he passed away I think two years ago, but I think like me many others were moved by just the form that The Goal took – this narrative that really takes the readers along in terms of describing what the problem looks like and feels like and that was really the inspiration for The Phoenix Project and you can imagine my surprise and delight when the string of emails starting coming in saying, "Holy cow! I just read The Phoenix Project. It’s like you’ve been living in our IT organizations. I just had the same meeting. It’s happening to us." Right? And just having that be an on ramp for people to enter the DevOps community already knowing what the problem that DevOps is intended to solve. That’s – as you can imagine – just incredibly gratifying that what I think has just been a great vehicle for people to have their own DevOps aha moment just like I had in 2010.

Andre:Well it must be extremely gratifying to know that your book is resonating with people and it’s actually impacting the lives and careers of folks, so congratulations.

Gene:Thank you so much. And of course I have to credit my fellow co-authors of The Phoenix Project – Kevin Behr and George Spafford. I was one of three authors on that project.

Andre:Of course. So, the DevOps Handbook is more prescriptive. Who’s it actually targeted at?

Gene: We specifically targeted at technology leaders and practitioners from all parts of the value stream, so that would certainly include Dev, Tests, Operations, InfoSec, but we also think it’s relevant for product owners and any business leader who is extremely reliant on technology to achieve their goals and I think whereas The Phoenix Project was really intended to have people sort of get that sort of visceral aha moment. This is meant to really show: where do you start? What are the principles and that make up the DevOps practices and where do those principles come from, going back almost a century of known and trusted manage and science and then where the sort of practices that you can put into place to replicate the amazing outcomes that we see in those unicorn organizations of Google, Amazon, Facebook, Netflix, Etsy. But also targeted at the horses – large complex organizations that have been around for decades or even centuries, and so one of the things that we put into the book was three years of learning that came out of the DevOps Enterprise Summit whereas you and I have talked about before, there’s a place where leaders of large complex organizations who are embarking on DevOps transformations. We ask them to tell their stories in the forms of experience reports. In other words, in a very specific form. Here’s our organization. Here’s what we do. Here’s the business problem we’re trying to solve. Here’s where I fit in the org chart. Here’s what we did. Here’s where we started and why. What were our outcomes? What did we learn and what challenges still remain? And so many of those case studies got integrated into the DevOps Handbook. I’m looking at some stats on it. In the DevOps Handbook we include 48 case studies – 500 endnotes, 192 footnotes and all that is really intended to arm the reader to give them as much knowledge as we think that we’ve been able to assemble that can help them on their journey to replicate those amazing outcomes that we see not only just in the unicorns, but increasingly in the horses as well.

Andre: Sounds like a tremendous compilation of information and knowledge about what works. So what’s your favorite part of the book, Gene?

Gene:There’s a couple of favorites that I think are near and dear to my heart just because they were – I had such a profound aha moment in learning about them. I think one of the sections that I’m just really delighted by is on Conway’s Law. So, Conway’s Law I think has been popularly paraphrased by saying, "If you have a – compiled a team that has four teams on it, you’ll end up with the four pass compiler," and so it’s been increasingly introduced in the DevOps discussion about the notion that what Dr.Conway put down on paper almost four decades ago was that the organizations really matched the communication paths and vice versa – the communication paths will mirror the organizational structure, but there’s another sort of profound implication which is that this has profound implications on the architecture and so if we have a large monolithic architecture where it becomes impossible for small teams to independently develop, test, and deploy value to the customers independently. In other words, in order for them to develop – to deliver value to the customer, they have to work and then wait for the next integration tests that might happen in months or quarters from now and so the notion that using Conway’s Law to really precisely dictate the conditions that must exist for small teams to work independently and to be independently able to develop, test and deploy value to the customer. It was just such an aha moment for me and there’s a specific case study that I really love, which is the Sprouter story at Etsy. So back in 2009, this was back when they were very much a horse. Every – as Michael Rembetsy and Patrick McConnell said as part of the Etsy transformation and then went through a retail season in 2009 that left everyone thinking that they might not be able to get to 2010 and one of the problems that they confronted was this thing called Sprouter. So what Sprouter was was this thing that sat between the application and the database. So Sprouter stood for the store procedure router and so it was developed really to buffer the developers from the database, but the unexpected outcome of this was that now before Sprouter they would need to have to two teams work together to deliver values. So developers would write a new piece of business logic. The DBAs would have to write the new store procedure and with the advent of Sprouter, now the Sprouter team would have to also deliver a piece of code and this as they said in the book required a level of synchronization and coordination that was rarely achieved. The end result was outages and so forth and so there was this three year journey to basically kill Sprouter and this was when they moved to object relational – ORM – and the application developers were able to directly make changes to the database and the result was that they had this incredible breakthrough. Not just in stability, but also developer productivity and so I think this was just a great example of showing how Conway’s Law can hurt us as well as how Conway’s Law can help us. So I think that that’s something that hasn’t been verbalized as clearly – I was just very proud of the way that story came out. The other one that I’m just really delighted about was just hearing about the Google testing culture. This was based on a presentation that Mike Bland did who back in 2005, he described the point to Google.com was actually a very dangerous activity. Even though it didn’t cause outages, it would often cause many unexpected negative outcomes like slow-downs, inaccurate searches, impacts on ad revenue and so forth and just to be able to capture the three year journey that the small team went on to stabilize Google.com and force continuous integration and continuous testing practices and not only create this world class group that was Google.com, but also expand that across the Google enterprise that allows them to do – that has allowed them to become the exemplar of continuous integration and continuous delivery. So that was just another story that I just learned to so much out of. And I think that’s so relevant. I think what I love about both of those stories is that even the unicorns were at one point in their lifetime, horses. Right? And to be able to capture that journey was I think important and valuable because those are relevant lessons for any organization embarking on a DevOps transformation.

Andre:Yeah. So virtually any organization can learn from those stories and be inspired by those stories to make changes within their machine.

Gene:Absolutely. And so those were two things that I learned so much out of. I’m hoping that other people will also have – maybe not the same aha moment that I had, but also find that valuable in their own work.

Andre:So you also mentioned that you really like how the book records Jez Humble’s new thing on continuous delivery versus continuous deployment. Can you talk a little bit about that and what that actually means?

Gene: Yeah. Absolutely. And I think what’s been so great was that in the community, there’s no doubt that continuous delivery/continuous deployment has taken a life of its own and I think one of the big questions that comes up is what is really the difference between continuous delivery and continuous deployment and to be able to sit down with my fellow co-author, Jez Humble – and by the way, the authorship team on DevOps Handbook is myself, Jez Humble, Patrick Debois and John Willis and so one of the sections that I just got incredible delight out of and also learned a lot out of was to hear his updated thinking on what continuous delivery is versus continuous deployment and how this changed since continuous deployment came out three or four years ago. And so the way that he verbalizes it now is that when all developers are working in small batches on trunk, when everyone is working off of trunk in short lived featured branches that get merged into trunk regularly, when trunk is kept in a deployable state and when we can do deployments on demand, that is continuous delivery, so these are the conditions that allow developers to get fast feedback on their work. So instead of learning about errors they make months later during integration testing, we can rapidly find out when our continuous pipeline can give us fast feedback from automated unit tests, exception tests and so forth, right? And there’s – we heard so many stories about that at the Jenkins Users Conference last year and what a phenomenal place Jenkins has in enabling that journey for so many people. And so examples of this would be certainly Facebook and Google where they are – trunk is always kept in a deployable state even though they’re not deploying every time someone commissions a trunk. So, this continuous deployment is when we’re doing all of those things that we just discussed, but we’re deploying to production at least once per day and maybe even upon every developer commit and so when we’re doing that, that would be considered continuous deployment. And so I guess in that definition, Facebook would fit that deployment at least twice a day these days. Amazon would certainly fit that model. Netflix would fit that model. So here’s I think the best – a really precise definition that really makes a distinction between continuous delivery and continuous deployment which I found incredibly satisfying to record.

Andre:I think that those are good clarifications and that organizations as they look to move more towards a continuous deployment type of environment, the testing aspect becomes critical, right? Because now you’re taking that next step. You’re actually allowing the automation to push the application into production and so I think that’s an evolution that every organization has to feel comfortable with and be ready to embark on at the right time for themselves.

Gene: And you know what we found in our benchmarking work that now spans over 25,000 IT professionals as embodied by the state of DevOps report that we’ve done over the last four years, that capability – the notion of being able to quickly get code into production as well as architecture and culture norms was one of the best predictors of employee satisfaction, that we get joy out of our work. It was one of the best predictors of whether employees would recommend their organization as a great place to work to friends and colleagues. So it’s not just great for developers and operations, but ultimately it’s great for the organization that we serve as well. So I agree, this is such an important capability for any technology organization.

Andre:Fabulous. Fabulous. So, I don’t think you can talk about DevOps without talking about Jenkins because it’s typically in virtually every reference architecture. How do you think Jenkins has changed the landscape with continuous delivery?

Gene: As I think I was eluding to before, I think just continuous delivery has taken a life of its own and I think so much of that is enabled by the work that KK has done both in terms of his work at Jenkins and even before that and Hudson. I think by making the deployment pipeline accessible and buildable by any engineer on the planet easily and quickly and to be able to have a community of kindred spirits and peers who are working on the same problem, you know there is no doubt in my mind that that has been a mass accelerator in the adoption of DevOps patterns and DevOps success stories. So, you can go into almost any unicorn and at the heart of it will be Jenkins and so my congratulations to KK and the entire Jenkins community for making that happen. This is just such a privilege to have been able to hear from KK the whole story of that and to be able to see that story unfolding at the Jenkins Users Conference.

Andre:Yeah. KKs an amazing guy. That’s for sure. And the Jenkins community is an amazing community. It’s extremely active and every day the growth of the community and the growth in the use of Jenkins I think is a great indicator in the adoption of continuous delivery DevOps around the world.

Gene: By the way, just maybe one comment on that and this is not in the handbook. This maybe just a reflection – maybe a deeply held belief of mine. I think in any sort of community, so much of it is based on the person – the leader of that community, so whether you’re looking at Linux and Lin Torvalds or you look at Patrick Debois and the DevOps community and how you see that in KK and Jenkins user community, I think what’s so remarkable about Patrick Debois and KK is that they’re such generous people. They’re open, warm. They have such – they have unquestionable integrity. I think the Jenkins user – the Jenkins community has benefitted so much from that just from the values that KK represents and verbally espouses so I don’t think that’s often talked about, but yet that’s one that I believe very sincerely.

Andre:I couldn’t agree more. Gene, so the DevOps Handbook – we wish you great success with that and I think it sounds like it’s going to be a phenomenal success and maybe even exceed the success you had with The Phoenix Project book.

Gene: Well, thank you so much and the book is coming out on the first week of October and we’ll be making available instructions on how to get the DevOps Handbook and so forth maybe in the show notes.

Andre: The DevOps Handbook is going to be available from the DevOps Radio landing page, so if you’d like a copy, you can download it there at no cost.

Gene: Woohoo! Yes. Thank you so much, Andre. That’s fantastic.

Andre:Thanks, Gene. Thanks for joining us today. It’s been wonderful talking to you again.

Gene: Andre, it’s always a pleasure and I look forward to seeing you hopefully in a couple of weeks.

Andre:Like what you’ve heard today? Don’t miss out on our next episode. Subscribe to DevOps Radio on iTunes or visit our website at Cloudbees.com. For more updates on DevOps Radio and industry plugs, follow Cloudbees on Twitter, Facebook and LinkedIn.