Jacob Tomaw - Your Ticket to CD and DevOps

Andre Pino: You're listening to DevOps Radio. The podcast series that dives into what it takes to successfully develop, deliver, and deploy software in today’s ever-changing business environment. This show is sponsored by CloudBee, the enterprise Jenkins company and continuous delivery leader. This week we’re joined by Jacob Tomaw, principal engineer with Expedia. Jacob, you’ve had an interesting journey getting to where you are now with Expedia. Tell us a bit about how you got into software development and what prepared you for your current role at Expedia.

Jacob Tomaw: Well, I’ve always been quite a natural with computers. I was lucky enough to go to school in a school district that had lots of computers throughout all of my classes and so that was – I was always gravitating towards that, and using computerized learning to get more out of school. When I went to university, there was really only one choice I had for major. It was going to be computer science and I wasn’t really sure – I didn’t really have a whole lot of knowledge of what people did with computer science degrees other than do stuff with computers. And so it was nice that I took a wide variety of computer science classes and learned a wide variety of things. I mean, all the way down to circuitry and that kinda stuff. And then it was kind of – turned out I really liked computers, but I wasn’t very good at job hunting and I didn’t really cast a wide net into what career to look for within computers. I went to work for a software consultancy that I worked on a wide variety of different projects. I did some interesting stuff, did a lot of stuff I didn’t like. I learned a lot about what kind of job I didn’t want to work for. I went to another job that I learned even more what I didn’t want to do. And what I realized what – as I going along – probably in the beginning of my career, I didn’t work anywhere more than, say, a year-and-a-half at any one location because I was learning a lot about what I didn’t want to do. But then I realized what I want to do is work for a place that is really a technology company where technology was forefront and then the business was a vehicle for deliverying good technology. In 2006, when I started looking at somewhere new to work in Chicago, Orbitz was really the top place to work in Chicago. It was known as a real strong technology company with really bright minds working here. And that really excited me. It would be a place that id’ really want to go to work. So I started at Orbitz in our Orbitz for Business development team, working on business-to-business travel solutions just as a sernior engineering working on teams, really passionate about getting things done the right way, making sure things were architected well. And really beginning to understand my own frustrations with what blocks people from being able to get things done with a point of writing software is to deliver value to the customers. And we have a tendency to create barriers that are supposed to be good valid checks to make sure bad things don’t happen, but they can really slow down the process. After working on Orbitz for Business, I went to work for another line of business where, as we were transitioning from a more waterfall development process into a more scrumlike Agile process, and I really saw a lot of the – a lot of the pitfalls of that last waterfall project really cemented the value of Agile software development, which, you know, is all about getting barriers out of the way and making sure we’re delivering value and only delivering value and nothing extra. And I went on to lead a team where I tried to implement what I thought was right for Agile software debelopment, and really I was working on a lot of the core founational plug-ins and applications that we have at Orbitz. After that I – and so from doing that, working on those core things, I was working with all the other development teams. They were our consumers. And so it was really that we wre delivering high quality software to them ‘cause we wanted them to take our new versions of the software that we were creating and I wanted them to really be confident in what we were delivering. But we were always delivering always really big bang releases. It was – we weren’t doing continuous delivery of these internal libraries. It was really the beginning of continuous delivery - at Orbitz and I think, really, throughout the industry. And –

Andre: Nice.

Jacob: And so I wanted to get a – I wanted to learn more about why – the things that would motivate these other development teams to take up our changes – the core team’s changes. And so I went to work for one of our featured development teams, leading up the package development. So if you buy more than one product on a travel site, that – we call that a “package” – our package development site to learn more about what the pressures they were seeing to why they wouldn’t pick up our new core library changes. And what I saw was that there was a lot of fear because we were making big changes. It was hard to express the value of these changes to the business that they were answerable to. And that really shed a lot of light, that I think if could move to much smaller change sets, reliable change sets, and knowing that we were going to support and rollback any bad changes, it wouldn’t really be a game changer on how those core libraries were picked up. At the same time I saw – I got exposed to all their applications at Orbitz. And saw that there was a lot of value that could be gained by delivering value more quickly and more reliably. Ultimately, my skill set wans’t in management. It was – it’s more in leading through influence and modeling good behavior for engineers. I became principal engineer at Orbitz, and started becoming the champion for continuous delivery in order to help developers not have a lot of barriers to delivering the software that they want to deliver.

Andre: That’s awesome. That’s a great story. So, Jacob, help set the stage for us. I’m sure most people know who Expedia is. But maybe you tell us more about Expedia, and of course, Orbitz as well.

Jacob: Yeah. So Expedia is a leader in the online travel agency space. We have many different brands, many different lines of businesses. One of those is Orbitz now. And we – within Expedia, we have several different platforms that all work together in various different ways. For ten years or so, I worked on the Orbitz platform and now, in a much broader sense, I work on the brand Expedia platform.

Andre: Great. Jacob, you’ve talked a bit about, sort of, what was happening in what led you to continuous delivery. Let’s take a step back in time and talk about what was happening at Orbitz prior to you going down this continuous delivery journey and path, and was there something – maybe you can talk a little bit more about whether any business pressures that were influencing that decision as well. As well as just the need to make IT and development more efficient, effective, and faster.

Jacob: Yeah. In some bit of irony, the primary motivator for me was how reliable we were. And that’s because we were reliably very slow, and reliably very wasteful. From really when I started at Orbitz until 2013 we were migrated onto what we called the “global platform.” We were building out this platform that’s about 200 different job applications that all communicate with each other. It’s thousands of virtual machines. And we had gotten into a regular cadence of deploying all of these applications every two weeks. We started out that we had to take a big side _____ and just play them all at the same time, and then we slowly decoupled those. So every application could be delivered on its own, but they still had set windows they were going to be deployed and that was roughly every two weeks. If an application didn’t change within two weeks, it wouldn’t be deployed. But, if – you know, most of our applications – say, half of them or so, were changing all the time and they would be deployed every two weeks. And we had – like most organizations, I think, we had one big monolithic web app also. And that web application would have lots and lots of changes. It would be deployed every two weeks. But it would also – it was great. It was very reliable. It was better than monthly releases, better than every six week releases, quarterly releases. So we’re deploying every two weeks. But we would also – no matter how big that change was over that two weeks, we were doing the same amount of testing every time. The deployment was still the same arduous manual process. And we would always have about the same level of critical bugs and patches after the release. We would deploy, say, version 1, 2, 3 into production and then we would look at – everybody would have to check out that production deployment, make sure it worked how they want to. And then Tuesday morning, after the release – so on Monday, we’d have a big call and easily two to three to four product areas would need to have a patch. And so we’d have to release 1, 2, 3.1 or 1, 2, 3.3 or something like that because of all these patches. And that wasn’t right. And it was definitely requiring a lot more – much more manual intervention than was necessary to get software out in a modern way. I also saw through this slow arduous process that our developers were taking shortcuts or not doing the refactoring or investment in the product that was necessary to really deliver a high quality job application. Because we knew – because the barrier to releasing anything was so high, we weren’t doing any opportunistic refactoring. One thing – I think what’s useful for organizations to look at is to measure what would be the – what’s the time it takes to deliver any arbitrary software change. And, say, a new product – so a new feature that you're going to implement – and what’s the time it takes if I make a spelling correction in a comment? So if I find a comment that has a spelling correction, I should – _____ use the spelling correction, I should be able to fix that right away. And that should be the easiest thing to get out. It’s production because it’s not going to have any impact. It’s just a comment change. And look at the difference between those two things. And if there’s no difference at all, and they’re both arduous processes, that’s bad. If there is a difference, then maybe that’s worth it ‘cause you have a way of doing test analysis to know that you need to do more tests for a real product change versus a comment. Or ideally, you have fully automated tests and they’re really short, and the change goes through really fast. But we went to remove all barriers possible to developers doing that kind of opportunistic refactoring that enables the codebase to stay healthy and for them to know that they can make quality improvements that no one’s every going to see, but it makes their life better.

Andre: Right, right. Really interesting. If you go back to that – the previous way you were doing things, one of the things that came to mind as you were talking about that was not only did you put out that in a release, but you know, the patch releases that you needed to do afterward kept all the developers focused on that release rather than thinking forward and what’s coming up next, right?

Jacob: Yeah. Absolutely. I mean, our working process level was really high. At one point, you know, I don’t remember exactly how many stories I calculated that people had to keep in mind, given – if you think about nothing’s really done until customers are using it and there’s no known bugs in it. So you have to keep in mind all the stories that you worked on this sprint. You have to – and because the release hasn’t gone out. It’s still – the previous sprint’s work isn’t gone out to production. You have to think about all those stories also. And there’s probably – the way you were on support for the earlier release. You had to keep in mind three whole change sets all once, and they would – you really could never get them out of your mind. So you could imagine that – that’s a tax on your mental ability to think about the current thing at hand.

Andre: Right. Right. Okay. So now you’ve decided that you need to do things a different way. Walk me through the process of transitioning to continuous delivery and what I’m really interested in, and what I think our audience would be really interested in is learning what you learned through that process, and what you would do different today if you had to do it again.

Jacob: Mm-hmm. The first thing that we – that I realized was that we were a learning organization. We were the type of organization that was ready to learn from our failures, and we’re already learning from our failures ‘cause this, like any project, was not going to go smoothly. And because it didn’t have super clear value to add, other than going faster, it was likely – if we didn’t accept that there was going to be failure at the beginning, some kind of – things that we would have to learn from – then it wasn’t really going to be worth starting it. After that, the very first real practical thing I did was make the thought work of delivering software be physical in some way. I took the – I fixed around an ideal task – some kind of really small story or development task that you could get done in half the day, say, three hours. And then if I make that change, a three-hour change, what are all the other stages of the delivery pipeline that that change is going to go through and how long does it sit in each stage? I think a lot of people, when they hear “delivery pipeline,” they think automated delivery pipeline. But every application has some kind of delivery or deployment pipeline. It’s just it might be manual. And it might have a whole bunch of steps. I mapped out ours. And ours had 18 steps that would take you 18 days. If you went from – if you injected a story into a sprint at the exactly the right time, which would be four days before the release that that change will go into, it would go through all of the stages and that would take 18 days. And some of these were huge stages. Like, we had a huge testing cycle, and some of them were rly small things, like, updating our version numbers for applications was harder and more manual than you think it might have to be. But we were really honest. We went through and looked at all those different things, measured out how long they were, and then I gave an estimate of what was the most important ones for us to get rid of first or to shrink first, and how much effort would it take to do all those things. And what did I want to do? Did I want to fully automate the step or did I want to remove it? Did I want to change it into something different? And we had a step or two that we actually wanted to implement. They were missing stages. And so after that we – the next thing is I put together a community _____ to talk about this throughout the organization. One of the real benefits we had at Orbitz is that all of our applications have a lot of commonality around some rules of the road that we had set up. They all worked in the same container. They were all deployed the same way. They were all monitored the same way. We had kinda had some of these fixed things that when we made improvements to one delivery pipeline, it would – it would improve everyone else’s. We really met as a full team – a full development organization to say, “Okay. What’s the thing that we can all work on?” And I get to take credit and talk to people about this wonderful stuff that we did, but it’s really the hundreds of engineers that have worked at Orbitz over the last couple of years that made the real improvements with just a improve – a ten-minute improvement here, an hour improvement there, eliminating this task, adding this feature to the delivery pipeline that really made it all happen.

Andre: That’s interesting, really interesting. Let me ask you a question. Going back to that 18-day process, how exactly did you go through the process of analyzing that existing – that previous existing process? Did you get everybody in to a room? Did you put it up on the wall? How exactly did you do that, and how did you go about making determination whether a particular step could be eliminated, could be automated? Did you scale – create a scale that would indicate how much work and effort you thought it was to automate it? Perhaps you can help us understand how you actually went through that.

Jacob: Yeah. So what I had some trusted senior engineers that I knew would give me – we’re going to catch any of your answers, so I would go talk tot hem and say, “Okay. Here’s what I think the delivery of any one piece of software is. What am I missing here?” And they would add a stage or say, “I don’t think that’s really a separate stage. It’s combined with this other thing.” And then we would take estimates of what they thought that would go through. Because of a lot of it, it’s not – it’s not something I can look up in some change management database and say, “Oh, this is how long it is.” There were things that were like that.

Andre: There’s a number of these steps that were basically in the heads of people that you had to go extract.

Jacob: Well, the – yeah. Or define whether we actually interpreted them as separate steps or not. ‘Cause some of them were – this work isn’t all – the stage of it isn’t held in one system. We would use our bug tracking system as our story system. There might be – a story might be in there and in different stages. And then stuff on the board – on our Agile boards might be something else. And then it might go into limbo for a while where, like, the product owner has to remember that this thing needs documentation or something like that. It was just gathering it all and putting it all in one place that everyone could actually see it. And that’s what I want to make – when I say “physical” I don’t mean I wrote it down on a Wiki. I mean, I had actual Post-It notes that people could see that spanned over two of our desks so that I could take it – bring people from release management over and say, “See. This is what we have to do to get our software.” And developers would see; and team leads would see; and testers would see; everybody could see what was actually – what the code had to go through. When you see it all like that, all lined up – because all those stages are happening all the time everyday when we had our deployments all staggered. No one could actually – you saw all the work that was happening in one day, but seeing it all spread out over 18 days was really an eye-opener.

Andre: That’s pretty amazing. So getting the whole team to visualize and visually see everything that had to take place was a bit of an eye-opener.

Jacob: Yeah. Now how I knew – I kind of – when I estimated what would be the value or which one of the things we should work on, it was really just a swag. Lots of times people can get kind of paralyzed by not providing any kind of an estimate. I’ve always found if I provide some kind of estimate that people can knock down or improve, that’s better than not doing anything. I kind of just picked what I thought would be the best for our organization to fix first, knowing that there was no way that I was going to be able to do all of it anyway. The most important factor on what to fix first was what people were passionate about fixing. For instance, one of the things that I thought would be the last thing we would fix was reducing the amount of time that it takes for us to test our software. But it turned out that people really wanted to do that, and it was much easier than I had estimated it would take.

Andre: Great. Throughout the discussion you’ve been talking about the team and getting the team – how difficult was it getting everybody to rally around this notion of making these types of improvements and automating this process?

Jacob: It was not as – I would break people into four different classes – maybe five different classes. We have our developers who – you know, they wanted to go faster. But they weren’t quite sure how to do it. Sometimes the problem seemed bigger than they could handle on their own. They really _____ needed someone to rally them to get the work done ‘cause they’re brilliant people who can get this work done. Then we have our managers who – they wanted stuff to go faster too, but they were concerned about making sure we were still delivering value. And same with our product owners. They wanted – what they want is value and what is really easy to convince them that what I’m say – what I want to do is give you the value that you want faster. And that we are making our goal, delivering software faster than you can think up new things for us to do. In an idea world, you –a product owner thinks up something, we implement it and we test it and we see if it works instantaneously. And getting – doing continuous delivery gets us closer to that ideal. Another group that we had were our testers and they’re the people whose lives I was going to change the most because they were doing – they had a really predictable job. They come into work and they would execute a bunch of test scripts. We would underutilize their skills, but we weren’t pushing them very hard. In this new model change delivery, we were using their skills, which means they have to work differently than they were before. But the most important group that I had to work with was release management, problem management, change management, whatever kind of management you have. We have lots of different management – star management, I call it in our operations – that I think a lot of time people see as a barrier to going faster or getting changes in that they – developers might see their – the change management’s job as stopping changes from happening. But they just want – they want to make sure that you're putting good changes in – reliable changes in, and that you have a plan for when changes go south. It was really – one of the – you know, you asked earlier, “What are the things that I would have changed?” and what I would have changed was getting release management right on board from the very beginning. It wasn’t like in the process that I got them on board, but I was kind of putting off talking to them, thinking I’m going to have to do a lot of influence to get them on board. But once they saw our pipeline spread out in physical form, they were right on. And they understood right away that we’re not talking about delivering bad changes faster. We’re talking about delivering changes faster and being able to revert the change faster also.

Andre: Right. So that’s a really interesting point. The idea is that you want to be able to go faster, but recognize that you might need to pull it back much more quickly as well so that should you encounter problems, you can revert back quickly.

Jacob: Yeah. Exactly. And ideally you're never actually reverting. You're just always rolling forward.

Andre: Exactly.

Jacob: If you have a problem, you're either patching it or you're – maybe you are putting in a reversion commit, but the application is always moving forward.

Andre: Right. So when you compare where you are today with those very early days, what are some of the improvements that you’ve realized from the adoption of continuous delivery, and both at the software development delivery level, but also at the business level?

Jacob: Well, we’ve – I mentioned that we were a learning organization, which started just as internally we’re learning from our failures, but we’ve – like most e-commerce systems, we’ve been doing a ton of test and learn, where we want to put different changes in front of our customers, see how they react to that, see if we’re providing value to them, and if we are, that’s great. If we’re not, then we’ll roll back that change and we’ll think of something else. And being able to deliver software faster means that we’re able to execute on those tests much, much more quickly. That’s one real tangible business value that we _____ _____.

Andre: So if I understand you correctly, it – what you're doing is creating this feedback loop to the business that enables them to understand the business impact of a change and then decide what to do from that point.

Jacob: Yeah. Exactly. ‘Cause we measure every change. There’s no point in making a change if you can’t measure what the value is going to be for me, and if that value was actually realized. Now we were able to do that up and down our software platform. It also means that now our - when our software engineers say something is done and our product owner say something is done, it’s the same thing. It’s the same done. It’s not just committed to master – since the pipeline is really small and fast moving from master into production, that means that done means done. Being a master and a production are nearly equivalent. And everyone is getting the psychological benefit of saying something is done at the same time.

Andre: It sounds like the side benefit that is that you got really good alignment between what you're doing in IT and what the business is looking for as well.

Jacob: Yeah. Absolutely. I mean, there’s not – we don’t really see a big difference between our product organization and our _____ organization. They’re – we’re all going towards the same goal of delivering value.

Andre: That’s awesome. That’s awesome. How has Expedia or Orbitz business and business operations changed as a result of this ability to accelerate the delivery of new features and this whole notion of testing that feature, getting more immediate feedback? How has it changed the business a bit?

Jacob: It’s hard looking back to see how things were before. We’ve really become accustomed to delivering changes quickly, being able to deliver things in a more timely manner. I guess one big change is we haven’t’ had _____ that feels like a big giant project in a long time. Nothing feels big and giant if you're deploying little bits of it all the time. And I don’t have a way of measuring it, but it seems like everyone is happier and happier with this software that we’re delivering.

Andre Pino: Right. Let me ask you this, and feel free to say, “I don’t care to answer,” but when you look at the competition, do you feel that they can react as quickly as you do? Do you feel like you’ve got an advantage there?

Jacob: Well, you know, I don’t know whether the competition can react faster or now than we can, but I know that we’re always pushing towards going faster. In –

Andre: Your journey’s not complete.

Jacob: No, absolutely not. And in fact, now – so a lot of the changes that we did were within Orbitz worldwide and now Orbitz Worldwide is part of this larger Expedia, Inc., which means we have a lot of – a lot more field to work in. And we have more benefits that we can realize from implementing continuous delivery even wider – you know, in a larger organization. And then because we have a much larger base to grow from, the benefits are going to be even greater for us and that’s what – we’re constantly driving towards – ‘cause I think most organizations are towards – we want to deliver faster. The marketplace is constantly changing. Software is constantly improving. And we’re doing everything we can to enable us to – for that to be an advantage, that the changing is an advantage for us.

Andre: That’s awesome. That’s awesome. What – so – you’ve discussed how your teams – your business teams and your IT teams are very much aligned and you're making these small iterative improvements constantly in a continuous fashion. Can you talk about how you’re organized to do that?

Jacob: Well, the organization is that we don’t have a separation between our technology teams and our product teams. And while the actual reporting structure has change – there’s been a couple different ways that we _____ _____ organized within Orbitz and then within Expedia, we’re organized yet another way. But it’s still one team. I’m on team that is the leadership team for our line of business and that includes analysts. It includes managers. It includes tetical folks like me. And it includes product strategy and product implementers. So it’s a – the key there is we’re all one team. And that’s actually a broader thing that we have within Expedia, that we’re all one team working on this. There’s not competing priorities and we’re not trying to one up one other side or prove product wrong or product’s not trying to prove technology wrong. We’re all working towards one goal. I think what’s helpful is that we’re – we’re all on this boat that’s going really, really fast so we all have to work together.

Andre: That’s very impressive. That’s very impressive, and I think that’s a goal that many organizations are looking to achieve and strive for. So as you look to the future, Jacob, what’s on your radar screen? Your personal radar screen? The Expedia radar screen?

Jacob: Well, as much as I try to get excited about other stuff, figuring out how to make developers’ lives easier and for them to deliver value faster is something that I just can’t stop caring about. And so now within – I’ve been able to lead this amazing change within the Orbitz Worldwide global platform and now I’m figuring out how to do that within the broader Expedia platform, which is really exciting. There’s lots of opportunities. We’re in more lines of business than we were – than Orbitz was in. We affect a whole order of magnitude more travelers. Whenever I give a talk on this, I talk – you know, I have an About Me slide and I start with that – what I do is I help people take vacations. And that’s a pretty awesome thing to do, and I also get to work on software that I really enjoy. I’m wanting to figure out how I can help all of the developers. We’re a whole magnitude more than I was before. Deliver software faster so that people can take their vacations they’ve been wanting to take.

Andre: I love that. That’s such a great way to position it. You help people take vacations. I love that.

Jacob: Well, really, it makes it easy to – you know, enjoy your work. I’ve done a variety of things. I’ve helped states collect revenue. And I’ve helped loans be collected. But helping people take vacations is a pretty fun way to spend your day.

Andre: That’s pretty amazing. You mentioned earlier that, you know, things are always changing. New technologies. Any new technologies on your radar screen?

Jacob: Well, you know, I mean the cloud is always out there. The cloud is ever changing. I think most organizations that have been around – I mean, Orbitz and Expedia, we’ve been around since the dot-com boom. We’re starting – in the grand scheme of things, maybe we’re a legacy organization. We’re still early on cloud and anything new that we’re doing, we want to be cloud first, but we have things that we’ve been working on for decades that clearly aren’t cloud first ‘cause the cloud didn’t exist. That’s – a lot of the interesting stuff that we’re doing is how do we take these applications that perform really well – ‘cause we’ve tuned them really well to work in a datacenter environment – and how do we get them to – so we can accelerate their delivery velocity. We can scale them in a different way by moving to the cloud and take advantage of some of the best of breed cloud services that are provided.

Andre: That’s fabulous. It’s a whole new area for you to work on.

Jacob: Yeah. It enables some really interesting stuff. It allows – there’s new technology, new development disciplines that we have to embrace.

Andre: Right. Great. Well, Jacob, as we finish up today, any final thoughts?

Jacob: Well, one thing – so whenever I give a talk, people – I think – I’m always amazed that people find what we’ve done amazing because it really is just lots of little changes that we’ve made. And I think that any – there’s nothing – I worry that people might get discouraged that, “Oh. Orbitz. A large organization with a large platform,” or, “Expedia with an even larger platform. They’re doing all of this, but there’s no way that my organization can do this.” And I don’t think there’s any organization that should be discouraged from adopting continuous delivery. And we should make continuous delivery first, part of what we do. If you can't deliver software, it’s not really valuable. I was thinking about what’s the opposite of continuous delivery? What’s the opposite of continuous integration? It’s really useless delivery and useless integration. If you're not delivering software, you're not really doing anything.

Andre: Right. Right. You’ve used the term “continuous delivery” quite a bit today. Do you use the term devops at Expedia?

Jacob: We do. I don’t – I try not to use the word devops very much because devops means a lot to a lot of different people. I know – for instance, for a lot of people devops means continuous delivery. For some people, devops means operations people who are writing codes. They’ve _____ or _____. For me, what my ideal devops would be would that we don’t have an operations team and we don’t have a development team. We just have people who are delivering software in an automated or manual way. But the devops is that there’s not really a distinction between developers and operations folks. We are just delivering software together.

Andre: That’s awesome. Jacob, thanks very much for your time today. I’m sure that everyone will find this very interesting and very inspiring to their own organizations. Thank you.Jacob: I hope so. I hope so. Thank you very much.

Andre: Okay. Like what you’ve heard today? Don’t’ miss out on our next episode. Subscribe to DevOps Radio on iTunes or visit our website at http://www.cloudbees.com. For more updates on DevOps Radio and industry buzz, follow CloudBees on Twitter, Facebook and LinkedIn.