Brian Mericle - The "Inn-side" Scoop on DevOps Adoption

Andre: You are listening to DevOps Radio, the podcast series that dives into what it takes to successfully develop, deliver and deploy software in today’s ever-changing business environment. This show is sponsored by CloudBees, the enterprise Jenkins company and continuous delivery leader.

Andre: Welcome to today’s episode of DevOps Radio. I am your host, Andre Pino. And today I’m joined by Brian Mericle, distinguished engineer at Choice Hotels International. Welcome, Brian. Thanks for joining me today.

Brian: Glad to be here.

Andre: So Brian, I understand that in September, you attended Jenkins World for the first time. I was wondering what your experience was like.

Brian: It was really, really cool. I have heard of the user groups before and this conference for this year really sort of fit into the types of things I was working on at my company. So I reached out. This year in this new role that I’m in, distinguished engineer, I really wanted to get out and start talking about the types of things that Choice Hotels is doing in this space and my focus has really been around continuous deployment. And I’ve been a Jenkins user for over – well, back in the early days before it was Jenkins. And so when I saw that they were being more formal about a conference, I really wanted to check it out. And actually, another couple things that were driving this was I feel that we are a CloudBees user. I basically picked that platform in order to help us build our continuous deployment pipeline. We had some questions that I feel that asking in person typically gets you better results. And so a combination of knowing that there would be support people on-site, knowing that there were other people in the industry that I could sort of gain insights into how they are using the platform, it just made a lot of sense to attend. It was intimate but not – it still was a good size and felt really well-organized.

Andre: That’s outstanding. Glad you had a great experience there. So Brian, why don’t you tell us a bit about your background, how did you first get into this business that led to the role that you play today?

Brian: Sure. I went to college in upstate New York and I graduated around 2000. It was kind of funny. I went in with a physics mindset and I took a couple years of physics and chemistry and calculus. I was always scared of computers. And one day in my sophomore year, I decided you know what? You’ve just got to get rid of this fear and I’ll just take one programming class. And it was Q Basic. And once I took that class and once I could see that you just type something on a computer and it does something, it was sort of I was hooked from that point on. And I ended up graduating with a bachelor of arts in computer science and just sort of jumped into the industry. My first job, I actually was interviewed by a company in Phoenix, Arizona and I was up in upstate New York. And it was like I’d really like to get out of New York. It’s sort of gloomy and it’s always sort of overcast and you get the snow and I was kind of done with all that. I went to school right on Lake Ontario so for four years I got berated with snow. And I said well let’s go to Phoenix where it’s pretty much sunny every single day. And I started my career at a small company which was kind of interesting because I got laid off three months later. So being around the college and starting in this industry and getting paid a decent salary and then all of a sudden get it all taken away from you. But it basically was one of the best things that could’ve happened to me because I found another job in a consulting company and from that point on, I just sort of never looked back. So I have done consulting work for multibillion-dollar companies, just sort of working my way up through the ranks. In around 2005, I went to a principal engineer position and from then, I took an interesting sort of turn. Around 2012, I had an opportunity to go work for O’Reilly Media as a software engineering manager. And I never thought of being in management. I’m pretty technical and I love solving problems but it was one of those things where it was well, I’m sure I will learn a lot and it will be interesting to go through that transition. So I did that. And the guy who brought me onboard ended up leaving six months later and I found myself being promoted to director of IT.

Andre: Wow.

Brian: And so that was a very interesting sort of transition because I went from developer to manager to director of IT in six months. I’m really hands-on so I understand technology and I knew that I could, from a technical perspective, help the team sort of do what they need to do from the infrastructure and engineering perspectives. And I did that for a couple years and then this project came up at Choice Hotels. I sort of was looking at where I was and not sure where I was going to be going at O’Reilly. Actually, so I worked at Choice from 2010 to 2012. The person who brought me to O’Reilly also worked at Choice Hotels. And so he brought me along to O’Reilly. Then he brought me back to Choice two years later.

Andre: That’s why you never burn your bridges, Brian.

Brian: People don’t typically follow their job, they follow their boss. If you have a good boss, it’s really hard to not want to work for them again. So it was one of those things where you want to work for people that inspire you, that bring out the best in yourself and you really click with. And it sort of happened to be with this guy, everything sort of worked out.

Andre: So you talked about moving from New York to Phoenix. Did any of those 115° days ever make you long for snow again?

Brian: No. I got married in 2008 and my wife is an Arizona native and she hates it. We actually moved to Northern California when I went to work for O’Reilly and she really enjoyed that, being 30 minutes from the beach was always nice even though you don’t really go swimming because it’s kind of cold. But she loved the beach. I will work anywhere. I don’t really want to work in snow if I can help it.

Andre: I don’t mind the snow as long as you don’t have to shovel it. So let’s fast forward to today, Brian, where your role at Choice Hotels is distinguished engineer. Now, that’s not a title you really come across very often. Can you tell us more specifically what your role entails and what it really brings to the IT group at Choice Hotels?

Brian: Sure. Let me just take one step back and when I came back to Choice, I came in as the senior director of platform engineering. It was a new avenue for Choice in that we had always been an on-premise data center user. So we went through all of the provisioning nightmares that you would hear all about I need a server, well, that will take you six weeks and because we have 100 other requests for servers – and so there was this constant slow provisioning cycle that really prevented people from getting their stuff out the door faster. But I think they were starting to look at hey, what’s this Dev Ops thing? Hey, what’s this continuous deployment thing? I think we should do something like that. And they had nobody internally that could (1) had the time to do it but had the experience and I came in with not necessarily I’ve done it a lot but I’ve designed systems that really allow for this type of transformation. My goal coming into the company was to develop a continuous deployment strategy with sort of a DevOps-centric pattern in mind. And get the company to a point where we could continuously deliver software to production to get our competitive advantage and just make sure the business was getting what they needed. That role came with a small team and I had my own budgets. Basically, for a year I worked on designing the system. At the same time, we were just starting this really big project at Choice to basically replace our central reservation system with a highly distributed and highly available microservices architecture which, if you know anything about reservation systems, the transactions volume that goes through these systems are really high. And it seems almost counterintuitive to make a centralized model into a distributed model but our CTO at the time really had this grand vision about how we could get there and how we could achieve that horizontal scale that we really needed because on-premise, our vertical scaling abilities were becoming really expensive and challenging to attain. So my goal at that point was to sort of design this, lead this team to build the tools and sort of processes to get us there. In 2015, in November, we had an internal conference, our first internal conference. It was called Mastery. And the purpose was to allow the engineers and QAs and just everybody in IT a few days of reflection of learning, of letting our peers sort of talk to us about technology that they are interested in or things that they are currently doing that people may not have visibility into. And in that conference, our CTO basically outlined a plan for – a certain progression in engineering. You come in as an engineer level I, there’s like level II through III and then you become a principal and a senior and then a principal. And then it sort of stops. There was no higher level technical position available. So your track was to either stay where you are in this principal and senior position or maybe go into management. But there was no technical position to go up to. So the CTO basically defined two new positions that were the equivalent to a senior director and vice president level but it would be an individual contributor that would be filling the shoes. And so the first role was distinguished engineer. And since I happened to already be a senior director, it wasn’t losing title or anything. But I really missed the hands-on approach. And not that I didn’t dabble in code because, especially at O’Reilly for two years, even though I was leading IT, I still was coding because we had not enough people to do all the work. But ultimately, I was sort of split. I couldn’t focus on one or the other; management 100% or coding. And it was sort of driving a divide in me. And so when I heard that they were going to have this new position, I immediately thought, that’s me. I want to be that distinguished engineer because I really want to get back to hands-on technology. I want to help drive things from actual hands, not from just a Visio diagram.

Andre: It allowed you to follow your passion.

Brian: Exactly, exactly. And so in March this year, it was approved and I moved into this new position. And in this position, I’m really looking at the future. Look at things that would help us do things faster and better. And since I had already designed our continuous delivery process, the only issue I had was I had nobody to do the work. The team that I had was sort of focused on this new project, this new version of our central reservations system and I basically had to get a contractor in to actually focus on it. And I was basically telling my boss again and again that if we don’t have a dedicated resource on this, this is going to fall to the wayside and I don’t want that to happen. Because I really am so passionate about defining a process that will get you from debt to production with no hiccups along the way and to prove the quality of that software. And so while I’m supposed to be, in my job description, looking at future stuff, trying to figure out patterns and approaches that would help us be more efficient and get things to market faster – I had nobody to focus on the implementation of this thing. And so I’ve been doing that for the last six months, focusing on delivering a pipeline using CloudBees Jenkins for our teams to deliver software to production.

Andre: So one of the things I find interesting about your career is that you’ve got a little bit of Dev in your career and you’ve got a little bit of Ops in your career. Is that correct?

Brian: Not so much ops but I have dabbled in it. So I wouldn’t say that I’m that good. I know enough to be dangerous.

Andre: Okay, great. So we’ve been talking about, you’ve talked a little bit about Choice Hotels and the reservation system. Why don’t we take a step back and why don’t you tell us about Choice Hotels International as a company, the size, the number of developers, to put a lot of this in some context and if there’s any unique challenges that you think Choice Hotels presents to you and the IT organization.

Brian: Sure. So Choice Hotels has been around for a number of years, right around 77 years right now. And they’re basically a franchise operation so an organization or a company that wants to run a hotel would buy the rights to a brand from us. So Choice owns a number of brands. This is just a few off the top of my head: Comfort Suites, Comfort Inn, Sleep Inn. So we are in a lot of the economy and midscale markets that we’ve also in the last couple years moved in more to the upscale. So our Cambria brands are our introduction into the upscale market. We have a number of really great properties that have sprung up, especially around New York, Chicago, we will actually be building a few of them in the Arizona area, Phoenix and Chandler. So basically license our brand to hoteliers and provide them with services and products to – everything from the sheets to the soaps to the point of sale system, the property system that they use, they can get all of that sort of infrastructure and support from Choice Hotels. We don’t own any hotel so basically we rely on those proprietors to really take care of the brand that they are working with. And so we basically have a lot of auditing that happens to make sure that people are following the right standards and strategies and that we have a number of hotels to convert a year. Right now we have over 6,400 hotels, approximately 500,000 hotel rooms under our wing right now. Our business unit, the CEO and all the business type people are all in Rockville, Maryland. And all of IT is in Phoenix, Arizona. So when you talk about, try to introduce an agile process into your organization, you are already typically split, even if you were in the same building between business and IT. Now you separated us for thousands of miles so it makes it even more fun. We have approximately 500 or so IT staff and I want to say total company, maybe around 1,100 or 1,200 employees.

Andre: And how many of them are developers?

Brian: So out of the 500 in IT, there’s probably I want to say 200 or so? 250?

Andre: Okay, so that’s a good size development staff.

Brian: We do a lot of work with consulting companies, some of the preferred ones that we’ve sort of vetted out and we tend to have projects that are really focused for a small period of time and then kind of go away. And so rather than try to keep a lot of staff full-time, we try to delegate some of that work out to some contracting firms.

Andre: Right. So Brian, you’ve used a couple of terms so far in our conversation: continuous delivery and continuous deployment. And I find those are probably two of the most misunderstood terms. I was wondering if, for our audience, you could describe what you mean by continuous delivery and continuous deployment?

Brian: Well, and then you also throw in continuous integration. So the way I see it is they are not necessarily all separate things. So I see continuous integration is typically the first step that a company takes. That really involves, okay, I’m writing my code, I’m committing it to a source code repository. I have some system, Jenkins just as an example, that will pick up that change and do a build and run the unit tests and either fail or succeed; notify the teams of that result and that’s sort of where it ends where you’re continuously – you’re building your code and making sure that your testing is proving that the source code is good. That’s the easiest sort of introduction into this whole process. I think many people will just do and not really think of it as a term. So I kind of categorize that as sort of phase 1 and that’s continuous integration. Then you get into okay, now that you have that artifact, that application that you are building and running tests on, wouldn’t it be great if you could now deploy that to some infrastructure and then run your functional tests against it? And so this is where continuous delivery comes in, in my mind, in that now that I have this artifact that maybe it’s a web service or another application but it needs a server to run on, we can provision a service, a server, deploy that service and then run some external tests on that service and then keep doing that in a continuous loop. And the importance of this is to get that feedback loop really quickly. So when a failure happens, you can sort of react to it really, really quickly. And then you get to continuous deployment. And there is an interesting sort of clarification between delivery and deployment. I’ve said delivery for a lot of years. And I didn’t really mean it. I meant deployment. So that you have your code being built and tested from unit test, and then you have it deployed to a non-production environment and tested from a functional perspective and probably you also have some nonfunctional tests, too, right? You could do your load testing, performance testing, resiliency testing. And then you get to a point where okay, I’ve put this code through enough. I think it’s ready for production. And so now the process is that it will take that artifact that’s been hammered and verified as being good and actually doing a deployment to production. And with that, you have a couple different options. So if you can handle outages, then you can do something like that. But typically what you would want to do is have a zero-downtime deployment. You don’t want your users to be impacted by a change if you can help it. So typically you’ll see strategies like a blue/green deployment or a red/black deployment. But basically you have an environment that’s out there, you deploy another environment that’s side-by-side and then you start slowly introducing production traffic to that new environment until you are comfortable that it’s working right. And then you basically scale up that environment and then tear down the old one. And then that way you can minimize the customer impact. Now, one thing that’s interesting is that’s easy to say, that makes sense. But what do you have to do to make that happen is a lot of work behind the scenes. There’s a lot of work on the development side. It really forces the development efforts to change and really think about okay, I know this is going to be in this environment where I need to support two versions of the software at the same time. And that’s an important thing to think about that not many people think about it first until you do that first deployment. Oh, this is not good. And I’m aware of this, a lot of people are aware of it and hopefully you’re thinking about this ahead of time. But ultimately it really changes how you have to design your software to support such a mechanism.

Andre: Absolutely. Why don’t you take us through the Dev Ops journey that you and Choice Hotels have been involved in. And why don’t you start with sort of what was the point where you decided that we need to change the way we are developing and deploying software. What drove that change or that need? And then quickly just step us through the challenges that you were faced as you went through that and how you overcame some of those challenges.

Brian: Sure. So I think like every other company, you are in a rhythm. You have your patterns, your processes, and typically they’ve all been manual up to this point. And you see lots of problems with that manual process. People make mistakes and people sometimes have a hard time accepting that. If I deploy a software one day and someone else deploys it the next day, if I don’t have the same person running my deployment plans, I could be doing the wrong thing or I could be doing this at 3:00 in the morning and skip a step. So there’s a lot of pain that you go through that eventually, you’re like, I can’t do this anymore. Business is really mad at us because we can’t keep our software up. And so I think that pain accumulates along the way and at some point there is a breaking point and you have to make a change. We do a lot of projects in IT and so I think we tried to keep up with the business needs. But sometimes you have to make trade-offs and you have to make sacrifices. And you sometimes make the wrong sacrifices. And sometimes that’s in quality, you just are trying to rush to get things done. But it all comes down to this manual process that’s really flawed, that people, despite their best intentions, do make mistakes and cause problems. And so part of my role when I first came in was to look at how do we out of this world? We’ve been thinking about this, Choice has been thinking about it for years. When I was at Choice the first time, I was really trying to push us towards this effort but what happens is, you have so many projects, so many things that people are focused on, it’s hard to carve out that time and the focus that you need to make that happen. And it wasn’t until I came in, and that was my role that I could actually make something happen. And even then, it took me a year to get through a bunch of other stuff to try and get even the plan in front of people to look at. And so I would say that the driving force behind this was lots of manual efforts that weren’t as fulfilling as we would like the last year. And then just the fact that the business really wants to drive our competitive advantage. We have new features, new things that we want to put out whether it’s for our website or for the services, that really would differentiate us in the market. And in order to do that faster, the business is not going to come to you and say hey, can you guys do me a continuous deployment? They’re never going to say that. They’re going to say I want my stuff to market faster. I want it higher quality. I want it to cost less to get there and please have that in two weeks. So you take all of that together and it makes us okay, we have to change. We have to really – in order for us to be to a point where we can be a better driving force in the market, we have to get things out faster. And what really helped was, we had this project that I talked about that basically is replacing our central reservation system, was a good opportunity to really have this be a focus on because that platform that we are building is all being deployed in the public cloud. And so one of the things that really enables you to get this continuous deployment model is if you have full control over your infrastructure and you can basically script against it and use it on demand. It’s really hard to do that in an on-premise data center without the right tools. And we have the tools, we don’t have the focus or the budget or the people to realize it. And we’re actually looking at doing that now as we started seeing our cloud bill over the last months. Obviously, I think what happens to a lot of people is this is going to cost some money. It’s not going to be free. The cloud is not free. You get brought in with the five cents an hour for your server and then you forget that you are running 2000 servers and for 24 hours a day and it gets kind of crazy. So as you see that, you’re like well, we have a data center. Why can’t we use our data center? And we’re like yeah, we’ve been saying this for a long time. So it’s one of those things where we’re talking about now that we really want to be able to take any workload that we have and have it run on-premise or in the cloud.

Andre: Yeah, I think that’s a smart strategy. There is so much workload that’s moving to the cloud and I think like you’re saying, sometimes that makes a lot of sense. Sometimes it may not make a lot of sense. So from an application standpoint, if you are developing an application to run in either environment, then you’ve got a business decision that you can make as to where it’s actually going to run.

Brian: And we thought about – so one of the things that we did was we had a really smart couple people that were really Dev Ops focus. So the people who knew programming, who had some background in operations understood how the OS works, understood Linux and so a couple of those people we had built tools out was get our applications to the cloud using their APIs. And knowing that at some point we may want to move to a different public cloud, we wanted to make sure that we are as agnostic as we can be where it makes sense. And so that would help enable us to move to a different vendor if we needed to. But what’s interesting about that decision is we sell hotel rooms. We don’t build software to automate deployments to production, to the public cloud. There are companies that do this work, that’s all they do and they’ve done it for years. They’ve gone through all of the cycles of testing and getting beat up and making their software better. I know that we were doing the right thing from a high-level perspective, I’m always feeling like we needed to really adopt some tools that help us do this rather than build it ourselves. But I think it was a good experience to go through that ourselves to learn some of the mistakes and learn some of the things that we have to learn. But I don’t think everybody has the time or money to keep doing that kind of work. Why not leverage some of the things that people are doing externally? Like learn from Netflix. Learn from other companies that have already been through this and learned a lot of stuff and have published a bunch of blogs and information about what goes wrong, what goes right.

Andre: And to what extent does container technology make the development of applications portable between cloud, on-premise and private cloud, public cloud, etc.?

Brian: Yeah, so one of the things that we did was in our microservices, we decided to deploy to Tomcat. And obviously Tomcat can run on numerous things. There are some things that are outside Tomcat that the server needs in order to run everything correctly. We didn’t jump on any of the container bandwagons but we are starting now to think this makes total sense, especially when we start talking about enabling our on-premise data center to have the same capabilities and same scriptability that our public cloud does, that we have to abstract that level of work and put it into something that will easily run in either one. And I think that makes sense that a container is that sort of vehicle for that movement. It would enable us to go to any public cloud vendor and as long as we do the right thing on-premise, we can move it there. And you have this little self-contained unit of things that will just do the work for you. And so I think it’s really important to think about that and that is our next sort of evolution step and that’s one of the things that I’m looking at doing is taking what we’ve done so far, containerizing it and then making it portable.

Andre: So we’ve touched on a lot of the automation that you’ve been involved in and used at Choice Hotels. But we all know that in these DevOps transformations, some of the cultural and people issues are some of the most difficult aspects of that type of change. What kinds of cultural and people issues did you have in your transformation and how did you overcome them?

Brian: Yeah, that’s a great question. This project I was telling you about, the replacement of our CRS, we decided to take the team of people that were working on that and put them in another building. We sort of separated out from our main building into this sort of separate building that would allow us to focus on this work and not be interrupted by other things. We took an approach of listen, we don’t want to boil the ocean. We don’t have time to get everybody in the entire company onto this platform. But because we are working on this platform that’s going to the cloud and we have these opportunities to use infrastructure the way we can, it makes sense that we focus on this one project. Let’s do this one project right and let’s do it in a way that will enable others to come onto it but there wasn’t a big emphasis on okay, IT as a whole, here’s what we are doing. We had high level bullet points, our CTO which was the sponsor of the project, would do lots of roadshows. When I came in, I sort of defined what platform engineering was going to be, what our mission statement was, what our objectives were. And I went and did a lot of roadshows to sort of let people understand what platform engineering was because it was a new department. So I did a lot of presentations, a lot of talking with our change control people, our infrastructure people. I really wanted to make sure that everyone knew what we were doing but at the same time, I was sort of told to only focus on this one project. I did a talk yesterday and the title was Always Be Selling, Always Be Closing. And it was basically a meme that had this really stock sales guy saying how can I get you into this continuous deployment today? You have to be able to communicate and be transparent about what you are trying to achieve. There are so many departments that you may not think need to know but should know. And so everything from quality assurance to system administration to operations to the executive staff to – basically there isn’t a team or a person in IT that shouldn’t know what’s happening because it really affects everybody. But sometimes you don’t tend to think that way. You think the engineer is going to be involved, yeah, the QAs will be involved and my boss will need to know. But other than that, some people don’t sort of get that message out to everybody else. And so I really tried to meet with all of those teams to discuss how is this going to change your life? What can you do for me? How can you help me achieve this goal? And I did a lot of that until I couldn’t do it anymore. The thing is, because people want to feel involved, too. They want to feel invested. And if you don’t talk to them, they’re never going to feel that way. And so you have to be proactive. You have to make those conversations happen. And you have to make them happen multiple times. I met with our change control people at least three or four times to ease them into the idea that something could go to production without their blessing. And basically what I had to do was, it’s not that you are not giving your blessed, but I am going to give you all the information through this pipeline that you wouldn’t normally get and feel comfortable with making a decision on. We have metrics that we are collecting along the way that will help us guide okay, so this team on this project has deployed to production 99% of the time successfully and maybe they can go through these gates that we have automatically and not necessarily need to go to change control. Well, this team, this project, they fail all the time so they have to go. Right? So there’s things that we can do to help automate that process but I want to make sure they felt comfortable. You’re going to get these notifications, you’re going to get these things that will tell you what’s happening. But at some point you’re going to have to trust that things are going to work okay. And the only way you really do that is with some history and over time seeing that happen. So we wanted to make sure that we had gates in there that would help us feel comfortable. And then also come up with ways to eventually bypass those gates to really get to production quicker.

Andre: So there’s no such thing as over-communicating, is that what you are saying?

Brian: Absolutely, yeah.

Andre: I think that’s really a smart approach because it gives you sort of the best of both worlds, right? It mitigates some of the risk but allows you to open the gates when you feel comfortable with it. So Brian, what advice would you give a manager like yourself who is just getting started in a transformation like this?

Brian: It really depends on who’s driving the conversation. Like I said, the businesses not typically can tell you I want continuous deployment. They are going to tell you things that they want that will hopefully elevate this idea to your mind as a solution to their concerns or problems. You need support. You need focus. You need time. Luckily, at my level, my boss is a VP of engineering. I can go to him and get his buy-in and have him help me push it down and get people to understand what’s happening. You really need a sponsor. You need somebody at the highest level that can help champion this process. I’ve not seen very many successful bottom-up approaches. You can start bottom-up. I feel like in a way, not that I was all the way at the bottom but having the idea, having the drive, having I guess the courage to say we can do better and then having someone above you accept that and really get them bought into that idea and really help drive the success. And so what I would say is regardless of where you are in the hierarchy of your company, you have a boss. You have other peers. And the first part is, don’t wait for someone else to come up with it because they may not. And if you have that drive and you have that purpose, then embrace it and go forward with it. And if you feel that it’s a burden or it’s a lot to deal with, just work with your peers and your boss to help try and mitigate that a little bit. That really has kind of helped me and it helped to just be transparent and trust that you’re not going to know everything. There’s no way you possibly know everything. And even if you design a process and you think it’s perfect, the first time you run through it, you’re going to find flaws. And so just realize that and don’t stress about it. And just work to make it better. Like I said, even through automation, there are still humans building it behind the scenes. Someone has designed that process; someone is writing the scripts that will be executed in this automation step. So yes, there will be flaws, there will be problems. So just accept it and fix it and move on.

Andre: That’s great advice. Brian, thanks so much for joining us today and good luck at Choice Hotels.

Brian: Thank you so much.

Andre: Bye-bye. Like what you’ve heard today? Don’t miss out on our next episode. Subscribe to DevOps Radio on iTunes or visit our website at cloudbees.com. For more updates on DevOps Radio and industry buzz, follow CloudBees on Twitter, Facebook and LinkedIn.