Pinger: A DevOps Rising Star is Born

Brian Dawson: All right. Hello, thank you, everybody, for joining us for a special edition of DevOps Radio live from DevOps World | Jenkins World 2019 at the Moscone Center in San Francisco. Joining us today, we have a couple of gentlemen who happen to represent our inaugural, or our very first, DevOps Rising Star Award. These guys will be on the big stage tomorrow being recognized for their accomplishments in the DevOps space and their accomplishments at Pinger. So, to take away the suspense, these DevOps Rising Stars are Adam Robertson, head of DevOps, and Kristian Windsor, a DevOps engineer and former software engineer -- we’ll talk about why we’re talking about that shortly -- at Pinger. Welcome guys.

Adam Robertson: Thank you.

Kristian Windsor: Yeah, thank you.

Brian: So I probably didn’t do you justice giving you an introduction. What I’m going to ask is for you guys to fill it in, explain your background, what your role is at Pinger, how you got there, and if you don’t mind Adam I’m going to go ahead and start with Kristian, put him on the spot first.

Adam: Absolutely.

Kristian: Yeah, I was working under the web team, I was in charge of the UI automation tests along with kind of owning the whole Jenkins build pipeline and all that stuff. This was before DevOps was even a thing. Adam was hired just last year as the first DevOps engineer and kind of started this whole initiative to move towards the cloud to automate the whole pipeline larger scale deployment, things like that. And I just kind of got recruited to DevOps and now I’m here.

Brian: And that’s all she wrote. It was the right decision as we move to Adam who recruited you to do it.

Adam: That's right.

Brian: It was the right decision.

Adam: Totally, yes.

Brian: Can you tell us a bit about yourself Adam and also how you ended up in this role of Head of DevOps at Pinger?

Adam: Sure. So I originally just – I've been in DevOps for 10 years or so and then before that I was a sys admin DBA, worked on racking servers and stuff, and then basically when I moved to Silicon Valley 10 years ago, I started writing code, and then just kind of came from there just working closely with engineering teams at various start-ups when things are small and you have to do that and then just kind of followed the DevOps curve as its happened over the last 10 years or so. I came to Pinger just having conversations with our VP of Technical Services about what his goals were for technology at the company. It's a 13-year old company, it's got some older practices, they were into data center and stuff like that, and they were looking to kind of make some changes to freshen things up. And after a few conversations with him, I decided to come on board full-time as the head of DevOps and kind of start the DevOps journey for Pinger.

Brian: Awesome, awesome. So before I dig into that a little deeper to get some parameters, how big is Pinger? How many applications and/or teams do you have?

Adam: Pinger is about 140 total. We have 80 committers right now so the engineering team is around 80. We have a European counterpart that we work pretty closely with, but about 80 total. The engineering teams we've broken up to kind of – we have, of course, two major apps in the App Store, which is going to be Sideline, which uses the PSTN network to make phone calls and give you a second number and then Textfree, which uses VOIP for a lot of the same stuff so those are the two major weights. And you know the engineering teams break down into more client teams, which web is one of them, we have an iOS team, obviously Android, stuff like that, and then we have back-end supporting teams that handle the back-end infrastructure and such. So we've got about eight engineering teams, all semi-autonomous, and then an operations team as well.

Brian: But at the end of the day what I did hear is in terms of customer capabilities and the back-end data, these eight engineering teams are largely sharing the same customer experience delivery, different applications but shared customer base?

Adam: Yeah, that's – we all basically contribute to supporting the – both applications, while they do have different and siloed customer bases they're more or less the same service and they use the same back-end so the engineering teams have to work closely together for that.

Brian: Okay. Awesome, well thank you for sharing that. So as you brought up, you came to Pinger to kind of bring DevOps, make DevOps real. Since you've started, you've overhauled the DevOps process. Can I ask first, and please share because you were there Kristian, before Adam, right?

Kristian: Yeah.

Brian: What did the process look like before you started the overhaul and in that change or delta what are some of the benefits that you've seen? And I'm going to direct this, if it's okay, to Kristian first.

Adam: Yeah, yeah.

Brian: Tell me what the difference is.

Kristian: Yeah, so the whole deployment strategy is we have sprints every two weeks we deploy. The developers, they merge develop to master, they create an RC build, they have QA mainly test this, QA then sends this artifact over to ops, it's just gone, operations then deploys it, and it's this whole major process. It takes like a day or two, you don't deploy on Fridays, lots of data tests, all these things.

Brian: Right and this is prior?

Kristian: Yeah, this is all prior.

Brian: Okay.

Kristian: And now there's no manual testing, or at least –

Adam: Minimal.

Kristian: Yeah, minimal testing and we just launched our Cloud Native application Sideline Web this past Monday so now –

Brian: Congratulations by the way.

Adam: Thanks.

Kristian: Yeah, so now operations doesn't need to deploy at all. They're just out of the picture pretty much. It's automatically deployed, developers don't need to worry about mainly merging, there are no tags of dates and sprint names; that's just all gone.

Brian: Trying to correlate multi-stream, multi-reason development.

Kristian: Yeah, it just took all the manual steps out of the deployment process really.

Brian: Awesome. Awesome and I'm going to dig in a little deeper because we are particularly interested and I'm particularly interested in how we are improving the dev experience. So as a software engineer working within the legacy process and then helping bring forth the new process what are specific benefits that you personally are feeling or seeing or are your peers?

Kristian: Well I mean just fewer things to worry about really. The whole process is a process. It's things you have to know, things that can go wrong, things that take time, and without those things, there are just fewer things that go wrong, fewer things – more time for developing features, more time –

Brian: To focus things on things that matter.

Kristian: On what actually matters, yeah. A developer is hired to develop code. When you're spending all this time on this process stuff it's not really – why?

Brian: Yeah and we did a bunch of interviews and I spent years as a developer myself and what I find is a lot of us want to do stuff that matters and stuff that gets in the way with actually developing. It's frustrating. So I'm going to ask you, Adam, you know since you have overhauled the process can you share some of the benefits? Now you and I talked about engineering benefits, business benefits, cost benefits. Maybe you can share some of that with our audience.

Adam: Sure. So it's interesting because there wasn't a DevOps program to overhaul. It was basically a very solid wall. When you said that developers would develop an artifact and throw it over the wall that's literally – like they had no clue what would happen.

Brian: Then they'd wait.

Adam: Well and also it was on ops' schedule. It could happen days later. There was no correlation so the development teams and operation teams didn't have the kind of cooperation that management wanted, which is part of the reason they started investigating DevOps principles and practices, right?

Brian: Ah okay.

Adam: So when I came in that was kind of the goal was like automate all the things and started to build really seamless pipelines that could allow engineers to focus on what they do best, writing code and writing features for the product. So initially the first three months or so I was there I was just kind of learning, taking in what are things and how do things work and recognizing not just how they work but why they are the way they are, which is a really important thing. When you go into an organization as an agent of change you can't just say, “This is stupid” or “This is wrong” because no one wants to hear that and you're not going to get buy-in from anyone talking like that. So really understanding why the processes are, how things are and how they got there and then start to kind of get everyone onboard and saying, “There are different ways we can do this and different ways we can approach it. Here's the problem set and here's a possible solution and here's the benefits” and that's something I had to sit there and work with each engineering team and work with.

Brian: And it's hard work. It has to be nurtured, right? It's not instant.

Adam: Yeah, you're dealing with people and people have pride and people are used to things and maybe someone wrote that code and they're not looking to be told it can use improvement, right?

Brian: You know and just to give feedback and we'll go back to the benefits, I love that you said that and you parsed it out that way. I think you will usually find and you guys can confirm that yeah, things are broken. Usually in most organizations, nothing's perfect, there are always things to fix, but did you find as I usually assume for the most part it's not because of bad actors. People would prefer for things to be better and is that the case?

Adam: Absolutely. I mean tech data is just a part of everyone. Everyone who goes to this conference has it and we know it and we try to do our best to minimize it but you've got deadlines, you've got things that you need to iterate, you need to deploy, and sometimes shortcuts are made and then those shortcuts became permanent structures and that's just the way things go unfortunately. And sometimes you don't have time to go back and refactor because you've got to keep moving. I've not worked in any place in my career where that did not happen.

Brian: Where that didn't exist.

Adam: Right. So – and you said like bad actors, that's a good way to do it, making sure people know that you're not here to correct them or scold them or anything remotely close. It's like let's solve problems. Let's identify what is a pain point for you, figure out a better way to operate, and let's go that way. And people generally – I haven't run into any issues with pride if you approach it that way. It's not like you wrote bad code. It's that you wrote it, for this reason, I get it, and now it's causing issues so how can we maybe do something different.

Brian: Yeah and what I often talk about myself and again really resonates with me in what you're saying is that there's a lot of value in taking the time to get people to identify their pain points and get engaged in finding a joint solution together. You may able to get started with a mandate or get sort of the quick injection of we're going to set stuff done, but in my experience, and I assume this has informed your approach, if you take the shortcut up front you're going to pay for it downstream. You're carrying along culture debt, emotional debt.

Adam: That's a really good way to put it and if you sit down with the teams and these are the actors themselves and they're part of the solution they feel ownership of the solution too and this is not just you coming in saying, “Hey, there's a much better way to cut down this tree.” We're all doing this together and I can take the experience I have from my career and kind of suggest things but a lot of times it's the conversations with the groups about hey, this is – because the engineers you're speaking to, they know the code and they know the product way better than you. They what the end goal is and so just together you get four pretty good heads in a room, 4, 5, whatever and you come up with a solution and you'll find out that a lot of people are way onboard. They're excited even because like you said no one wants to be weighed down by these manual processes, these two-week cycles that are causing them issues. People want to be happy when they go to work, especially engineers that want to go and create cool things.

Brian: I love it. I think we need to understand it and underscore people generally want to be happy at work. They want work to do. I think some of the tribalism we develop is we have to build in our mind whether it's ops and dev, dev and QA, management and individual contributor, we have to forget that at the end of the day we all came here to do good. There are very few people that didn't show up and want to deliver, right?

Adam: Absolutely.

Brian: So we talked about some of the benefits from the dev side, we talked about the current state, how you approach change, which I think is important for our listeners to note. Now okay, once you did that, you engaged the team, you investigated, you started to get aligned in terms of common solution. Where'd things go from there?

Adam: So I started proposing solutions and one of the things with proposing solutions to engineers is ideas are great but showing examples are better. So basically the first month after I went there Kristian helped and we actually made a pipeline on Blue Ocean showing a deployment to an S3 bucket and literally I started at the all-hands engineering meeting and I started this and I just sat there in the background ticking through and we deployed a full version of web up to a bucket in 3 minutes and 40 seconds. And everyone's like, “Wait, you just deployed web?” “Yeah, we just deployed web.” And they were like, “Wait.”

Brian: Wait a second.

Adam: Mind is blown, right. And they just – it took – but that was a pretty big hump where everyone was like, “Wait, this is actually possible and this isn't just like BS that he's touting. This is something that's actually cool,” and everyone kind of was wanting it and it was funny after that there were a lot of teams that were like, “When can we do it?”

Brian: I want some of that.

Adam: Exactly and that's – since then it's actually been a lot of buy-in from that. And a lot of things that help too is our VP of Engineering and our VP of Technology Services and just top-down was saying, “This is the direction we want to go,” and they had the same mentality with like this is not a mandate in any shape or form. How can we talk about this as a group and everyone have the buy-in and say that this is the right thing to do. So as we started showing benefits we had demos, we had web going, I was showing them – we actually got the enterprise version of CloudBees, we got CloudBees Core working, we got DevOptics and I'm able to show pipelines and just demonstrating these like these kinds of things aren't an opinion. I'm not a fanboy of this. It's not that. It's actual demonstrating value and you can see it. So this is not – it's a strong way to convince engineers when they can see it with their own eyes.

Brian: Yeah and not to stereotype engineers but we work in hard science as engineers, quantifiable things so to say so there is a bit of ‘show me the proof’. Was that your experience as well, Kristian?

Kristian: Yeah, I'll agree with that.

Brian: Good, good. I like people in here to agree with me. So now on that journey from socializing, proving, giving demonstrable proof points of where everybody can be, people are on board and I'm going to flash ahead. As people onboarded into I'll just call it the new DevOps approach for now, what are the benefits? What are the benefits that individuals have seen and you guys have seen as an organization?

Adam: Absolutely, so concrete examples like we decided as an organization everything we're going to do we're going to build as code. So we use Terraform and basically we launched a production cluster in an hour, full Kubernetes deployment, and it's all in code so I sat there taking our various engineers or our operations engineers and I sat there, they pulled the code, and they were sitting there building clusters. This iteration was so quick and they were just like “Wow, this is amazing.” So our infrastructure is so elastic and scalable it can be done in a short amount of time. And for traditional operations engineers to see this is mind-blowing, right? They're used the 6-week turnaround to buy servers and then just things like showing our boss my monthly bill compared to the bill that we had for a data center.

Brian: Yeah, what is that? I'd like to hear the number.

Adam: We have – I'm not –

Brian: Oh, you can't. Give me orders of magnitude. We can't give real –

Adam: Put it this way: for what one server costs I can run my production server for 9 months.

Brian: Okay, that's a great way to put it. That's not small.

Adam: Yeah and also just the scalability like we were talking about benefits, right, like we have auto-scaling with Kubernetes and AWS. If we start to experience large amounts of load we can auto-scale and this is not a manual thing; this is fully automated so we can scale out and scale back in. Our costs are fairly low compared to buying servers and also just we're able to do so extremely quickly. Engineers themselves are very much on board; they're starting to get the culture. It's been a year. A lot of them are talking about pipelines, a lot of them are writing pipelines. We actually have an automation team that writes automation tests and they are fully writing all of their own pipelines, which is a huge change in the –

Brian: Wow, that is big, empowering them, bringing them into the process.

Adam: In the beginning, they were just literally like, “When are pipelines going to be ready?” It's like excuse me?

Brian: And it was all on you.

Adam: Yeah, it was one of those things like you know teach – it's very much a teach a man to fish or in this case teach an engineer to write pipelines so everyone's on board. It's actually really exciting because the approach that we had, in the beginning, was very much a collaborative one and now the trust is built up. And so engineers know that we're just not someone else. It's like we're –

Brian: We're you, we're a team, we're together.

Adam: Exactly and DevOps is very much a service organization and engineers are our clients and it's like our job is pretty much to make their lives a whole lot easier and make them more efficient. And they know that so nowadays people are actually spending the time to educate themselves on how to write pipelines, how to be more DevOps-y. We have engineers that are now talking to our operations team about the infrastructure in which the code is being deployed, which didn't happen before like we're actually getting a mix of things. Operations actually knows more about the code and it's great.

Brian: Cross-functional conversation, distribution and sharing the knowledge.

Adam: Absolutely.

Brian: Yeah, this actually hits me, you hit on a point, and I have like 50 things in my head I want to talk to you guys about but we just don't have time. If we can talk about each one in 10 seconds we'll make it on time.

Adam: That'd be impressive.

Brian: No, but great conversation. I'm really curious but you hit upon something I wanted to ask about. So today you guys have a DevOps team, Head of DevOps, and your job is to make their job easier. As engineers, as the software engineers and the app dev teams begin to gain some of these skills and you share knowledge what's the role of the DevOps team? If we flash forward you know at Pinger or ideally in the industry alone do you see a place where DevOps becomes fully embedded in the app dev teams or how does that pan out?

Adam: Yeah, do you want to go for it?

Kristian: I don't have a solid answer for that but I will say that DevOps is the mindset. It's like a culture. I don't know if you agree with that.

Adam: Yeah, that's very fair. I mean this the classic thing too like you're going to automate yourself out of a job, and okay, there's some places but I think that would represent a very static company that's not looking to grow. The whole idea, like Kristian mentioned, is it's a principle or a set of principles rather and it's about how you operate and that never – that's always changing. New technologies are going to come out, new ways to deploy, lots of things, so being able to iterate on that I would go through my – we have enough DevOps projects until 2022 right now so I'm not worried about that.

Brian: Yeah and then there'll be the next thing after that, the next thing we need to do better.

Adam: Yeah, Kubernetes 1.16 will come out and there will be new stuff.

Brian: Right.

Adam: But I think in general it's like – I think it'll be great but things are never done and engineering teams as they embrace these DevOps principles and operations teams do as well we'll work together and things will work very quickly and we can focus on making better products, like getting out features quicker and that's where we want to be. And just like to say you're there is – I don't think anyone ever gets there. There are always ways to keep moving.

Brian: Right. Good and that actually sets up for shifting to something we also talked about earlier and that's this concept of green fields and DevOps in small green field organizations versus DevOps in what we'll tend to call brown field. There's existing work, existing code, we're driving business, but we have to rebuild the airplane in-flight and it sounds like you guys have learned some lessons about that. You have anything to share?

Adam: Yeah, my history is usually with smaller start-ups that start with 1-2, 3-4 people maybe where that green field exists, you don't have an infrastructure, you can build it from scratch, and you have a lot of options. You may not even have a product yet so you're in stealth mode so that certainly allows a lot of flexibility in the technology decisions you make. You come into a 13-year old company with a very well-established product that's been – we're also profitable so one big, key thing with that is you have to keep doing that.

Brian: Yeah, the one thing about when you cross from red into black is you have to stay in the black.

Adam: Yeah, as it turns out if I ask our CEO if I could just turn off the app for 2 days it just wouldn't go well.

Brian: Yeah, maybe not.

Adam: So yeah, the way you put it is just perfectly, a perfect metaphor. The plane has to keep flying and you have to change the wheels and the wings in-flight. How do you do that? And that takes a very careful approach but also one of the other things is just collaboration with the engineering teams too. You have to – just with all teams, the operations teams, the QA teams, the engineering teams, the management as well to make sure that everything is aligned with making sure your product is being delivered with new features and stuff. So one of the things with that that I've found is just extraordinary amounts of planning. Plan more than you think you're going to need to because once you pull that trigger and you're in flight and you're moving and you detach the wing and you're still going it's like, “No, put the wing back on.”

Brian: Right.

Adam: And so have just an extraordinary amount of planning and buy-in from lots of people. One thing I did find when I started doing this here at Pinger is there's things you find along the way that you did not plan for and so a lot of times it's, “Okay, have we thought of everything?” No, probably not. Am I wrong about a lot of things? Probably. How wrong? I don't know yet. But as long as you put in a good amount of work in the beginning and everyone kind of agrees then you can deal with it on the way. Just make sure you don't fly by the seat of your pants thinking you've got it solved because you don't.

Brian: And I assume a component of it is that work you put in upfront to build buy-in and build trust is what allows you to respond to the unknown that you know is going to come eventually, right?

Adam: Absolutely. People are far more – you know something goes wrong the mentality is, “Oh, we didn't think of that. Darn. Let's solve this problem,” versus, “I knew it. Oh god. Roll it back.” It's like it's whoever's – it's the people's attitude when they come in. If everyone knows that you put a project plan together, you have the experience with this and they trust you, when something goes wrong they're like, “Well okay, let's just figure out it,” instead of, “Oh man, okay. Roll it all back. We're going to back to the data center.”

Brian: Right. So I hear, and it'll be a good shift, is that one of the key differences with green field and brown field is you've got to rebuild the airplane in flight, you've got to protect revenue, you have existing functions that are already doing their thing, they have to carry business as usual, so you have to do more collaboration and more planning in a green field versus brown field and brown field versus green field.

Adam: Yeah and also one important thing is the business side of it. When you've got X million dollars' worth of hardware sitting in a data center that 2 years ago you just convinced your CFO we needed and then you're saying we're moving to the cloud that's an interesting conversation. You have to make sure you're prepared for that one and of course hardware ages and that's part of that, but when you just – not everyone likes to watch a couple of hundred servers sitting a data center that you said you don't need anymore. So that's a tough one. I can't really give much advice on that except think that one through before you have that conversation because it's going to be awkward.

Brian: That's great. So shifting, so communication we've already heard a theme in this, communication is a key part of your product and customer experience. How do you prioritize that within your development process and how do you prioritize that to unite your teams?

Adam: Yeah, that's a big one too. One of the biggest things is when I came on there wasn't any DevOps and so this was new and why is this getting such a priority and other engineering teams didn't know why we were able to spend money when they weren't. And so that communication was important in terms of everything we do we have to justify. How is it valuable to the company? Why are we doing it now? When people heard that we're getting another full-time and possibly – why can't I get a full-time guy and some of these questions. And one of the things that's really helped with that is communicating and there are all the lists of DevOps priorities that we have, why they're important. We give updates every month at the all-hands about what we're doing and how it's adding value. And when you get in the trenches too, when you're talking to developers, like part of it is optimizing our pipelines. We bought a bunch of new hardware for our guys so that they can start – hardware in terms of building – they're building pipelines so they can build faster and they love it and they appreciate it and they...

Brian: They get benefits.

Adam: They get benefits and they see it immediately. And so after a while, they don't mind what's happening. They understand the value in DevOps and they see it and the value for them and for the company as well. And so one of the things that I do is I've got a DevOps roadmap that I publish and I give to – the entire company sees it and I actually have when projects are slated so no engineering team is wondering when they're going to get their turn; they actually know. Every two weeks I meet with every engineering manager, we have 1-on-1's, and I do that religiously just to have a conversation about what's your team up to, what are you missing, here's what our team is doing, and so we always are in communication and they don't have to wonder and they don't feel left behind.

Brian: So the way to prioritize communication is –

Adam: Prioritize it.

Brian: By communicating.

Adam: And it's also a site game. You make them feel – you let them know that they're important. That's the important thing.

Brian: So we're at DevOps World | Jenkins World and one of our themes of the conference is superheros and DevOps superheroes so I'm going to put you guys on the spot and I'm going to start with Adam – I'm sorry, Kristian - on the spot first, apologies. Kristian, who is your DevOps superhero and why?

Kristian: Obviously Adam's my DevOps superhero because he introduced me to the whole DevOps world really, the DevOps culture, the mindset. I wasn't really aware of what DevOps was a year or two ago.

Brian: Right, awesome. And this is the space you think you'll stay in? You've found something you like.

Kristian: Absolutely. This is the direction I've been moving in but I didn't know where I was going and now I know.

Brian: Awesome, awesome. All right, so I'm going to make it – actually what I'm going to do is I'm going to let you off the hook. I'm going to get you with the other question, not our DevOps superhero. Can you share briefly with us a dev-oops moment, something that happened where you're like, “Oh *****,” and you learned from it?

Adam: Absolutely. I can think of hundreds but I actually created what I call a dev-oops page, which is where we track all of our mistakes.

Brian: Oh wow, awesome.

Adam: Yeah, and it's actually every time something goes wrong you put an entry of what happened so other people can learn from it too. But I'd say the biggest is writing some automation that would go through and clear out old tables and accidentally – this particular one was testing MQA and so it was fine to drop that table except it was pointed to production.

Brian: Oh.

Adam: So we kicked off the script and it dropped all the production databases.

Brian: Oh ouch.

Adam: And this was at a different company years ago. This app was generating about $300,00 per day in revenue and so I basically – our CEO came over and he's like, “Fix it and fix it now,” and so I slept at the office for three days fixing it and reloading the data from back-ups and such. And so that was an interesting lesson in both the pros and cons of automation.

Brian: And how you need to wield it wisely huh?

Adam: Yeah, automation giveth and it taketh away and scripts do not care and they will most definitely drop any database you tell it to.

Brian: Well awesome. I am glad we get the benefit to gain some amusement and learnings from your pain. Thank you for sharing it with us. So we're going to get ready to wrap here. Guys, thank you for your time.

Adam: Yeah, thank you. It was great.

Brian: I think there's a lot of insight and this is where we gain value from sharing our experiences, this is how we all get better, and more importantly I really do see why you guys won the DevOps Rising Star Award in having this conversation with you so congratulations and thank you for your time.

Adam: Well thank you.

Brian: I hope you enjoy the rest of the show.

Kristian: Thank you so much.

Adam: This is great. We're excited to be here.

Brian: All right.

Read More >>