Forrester’s Charlie Betz and Jeff Hammond – Digital Transformation, Cloud and DevOps

Andre Pino: Welcome to this episode of DevOps Radio. I'm joined by two Forrester analysts, Charlie and Jeff. Welcome, gentlemen.

Charlie Betz: Nice to be here.

Jeff Hammond: Good to be here.

Andre: Charlie, what's your area of focus and research?

Charlie: Well, I have recently been named to be the global DevOps lead for Forrester, serving infrastructure and operations professionals, so I work closely with my colleagues serving application delivery and development professionals. I am here to serve the traditional INO and bring them the word. [Laughs] Previously, I was focusing more on IT service management frameworks and I've also done a little bit of work on cloud with Lauren Nelson, another Forrester analyst, but looking forward to the New Year and moving into this new area.

Andre: Great, thanks. Jeff?

Jeff: Well, if he's the ops in DevOps I think you can kind of guess what I do.

I've been with Forrester now for about 12 years and I've been in the application development space for about 25. Started as a developer and then built development tools and then have been with Forrester, although as of this year I'm actually not on our application development delivery team anymore. I was moved onto our CIO team, and one of the reasons for that is as we see more and more companies depending on software to be a critical part of their business and underpin the digital transformations that they're embarking on, the CIO needs to understand just how important transforming their development shop is to being successful with their digital transformations. DevOps is a real part of that, not just from a tooling or a process perspective but also from a cultural perspective, from an organizational perspective and from a measurement perspective, and so part of my job at Forrester is to help make sure that our CIO-level readers understand just how important getting this right is.

Andre: And are you finding that those CIO-level individuals that you talk to recognize the importance of software to their businesses?

Jeff: I certainly think so. One of the things that we've seen – I did an analysis of our inquiries that we've gotten on DevOps in 2017 – good time to do it since it's the start of the year – and one of the things that we see in this inquiries is what role that client has in the organization, what role they've coded into. And as you would expect, there's a lot of folks that are from application development, there are a lot of folks from infrastructure and operations, but almost one out of every three of those inquiries either comes from a CIO reader or from an EA reader, an enterprise architect.

And so when you see that it's pretty obvious that DevOps thinking or interest in DevOps isn't just coming from the devs and the ops; it's coming from leadership, it's coming from architecture, and I think one of the reasons for that is that one of the places that we see our large client start to hit a wall with respect to DevOps is when some of these larger issues arise that require more than just the app dev and the operations folks to get their act in gear. Let me give you an example. I spent some time with a very large telecom client of ours, and they got to a point where they could not go faster without making some pretty significant structural changes to their application architecture. They needed to put in feature flags and version flags, they needed to begin to modularize these massive applications they had had because it didn't matter at that point how many more people they threw at it, how much more tooling they bought, how many more – how harder – much harder they worked the process; the architecture was holding them back. Classic thing that you see when you start to scale and move beyond the initial stages.

Andre: So even though they want to do updates in small batches, the fact that the application was a huge monolith just still continued to stand in their way.

Jeff: Yeah.

Charlie: Yeah. And Jeff's absolutely right. You see too much transactional friction in the handoffs, which is why you've got this big push towards product-oriented teams. What's our term for it again? You pointed that out to me. The…

Jeff: The idea teams?

Charlie: No, the product-centric teams, the cross-functional teams.

Jeff: Oh, cross-functional, co-located teams?

Charlie: Yeah, we put a label on it I can't remember. But at any rate, you know, when you have for years organized around centers of excellence – you know, the network team, the database team, the Java team, the QA team, the QA is separate from dev, and the whole idea was that we were gonna try to emulate some kind of an assembly line and hand the work on in big batches. Well, you can skinny the work down, you can get it to smaller and smaller batches but you still have those transactional – that transactional friction between those handoffs.

Andre: Sure.

Charlie: And after a while, to your point, that just does not scale, and so you have to start thinking about, well, maybe it's more effective if we have a more – a higher variety of people on the delivery team. So you start to have the developers and the testers and the data specialists all on the same group and, of course, then you start to get into the microservices conversation and the full stack teams.

Jeff: Yeah. I recently, in the past couple years, been spending a lot of time looking at mobile application development and delivery, and one of the reasons I've come back into the DevOps space is because in some ways we found that the mobile teams had to become really good at DevOps because of the velocity demanded in the mobile space.

Charlie: Exactly.

Jeff: When I would talk with a mobile client, one of the things I would say was the first thing you'd have to do is you have to organize your teams to deliver into production a minimum of six to eight times a year, and that's not even, you know, thinking about feature updates and – because you have no control over the – you know, when Apple or Google updates its platforms it's not an option to say, "Well, we'll just hold the operating system constant like we used to with Windows 95 Service Pack 3, right?"

Charlie: Right, right.

Jeff: And so those sorts of cross-functional collocated teams with rapid delivery processes just became the way that people built mobile apps, which then led us to, you know, cloud-native architectures behind those things with mobile backends as a service, so for a while you saw the most common workload for public cloud infrastructure. The number two workload in our data was as a mobile backend.

Charlie: Yeah.

Andre: And so in some ways I feel like what we're doing is saying here are the things that we have learned in this new area, in this cloud-native world and this mobile world. Let's bring that back into our mainstream development and apply the DevOps principles that have really become battle-tested there to our mainstream application development activities.

Charlie: Exactly.

Andre: So Charlie, you talk to a lot of enterprise clients. What's your view on the current state of DevOps? What's happening generally within organizations?

Charlie: Well, I think we have rough numbers industry-wide, what, 42 percent are – you know, will identify to some degree with saying we're moving in this direction, and then there's still some people kind of in the middle. I would say that the laggers and resistors are really – they're diminishing in number, although what do we mean by DevOps? And you can start to ask people that and then you have to get in the question, well, what really are you implementing if you say you're doing it? That said though, the heat map, which we've put together and updated the heat map for 2018, the hottest area is probably telecommunications and/or financial services. We know that the big telecoms are under a lot of pressure.

They do not – you know, you can always walk down the street from the – and go to the other telecom carriers; a lot of competition. So they've got to actually keep updating those products. And financial services, for whatever reason, has also really started to up their game with increasing their release velocity. Now they've got regulations, so I think there's more research to be done as to how that's exactly playing out, but the heat map is consistent across a number of domains, that financial services is big. And then finally, manufacturing is a, you know, third place. Not necessarily a close third but it is interesting to see the interest there.

Andre: So it's interesting that you guys come at DevOps from different directions, right? Charlie, you from more of the ops side and Jeff you more from the devs side. Jeff, what are you seeing from a developer perspective with respect to DevOps? Is it empowered – empowering developers?

Jeff: It's interesting because I find that developers seldom need to be convinced of the value of the tactics and the processes because they want to change things, they want to go faster, they want to make new things, they are paid to change things, and anything that helps them do that faster is great. I find that they often chafe at their management, at the larger IT organization that, you know, prevents them from going fast with a lot of rules or with the focus on standardization or on control, and one of the interesting side effects of that has been that we've begun to see a little bit of a schism in the way that development organizes as a result. We're increasingly seeing situations where developers are getting embedded inside the business, inside a digital organization –

Andre: Yes.

Jeff: – or inside a product group and are no longer in IT, which suits developers fine because then they can work directly with the product manager or the business sponsor that is pushing for the change and they're allies. I think, you know, that can go too far, and so one of the things that we tend to recommend is that's great for frontend development, for user experience, but, you know, keep your microservices, keep your systems of record inside that centralized organization with a little bit more control and governance around it, but in both of those cases those teams are looking for solutions that are going to help them go faster, and so you see them adopting solutions including DevOps tooling that allows them to do that, whether it's something like a platform like Heroku, which makes it very easy to push, or a platform like the Cloud Foundry that makes it easy to push or, you know, finding their own open source tools that they can then begin to instrument and use to automate their processes and hook those up to their deployment platforms of choice, whether it's Amazon or Cloud Foundry or Azure or Pivotal. They are taking matters into their own hands and they're looking for tools that allow them to do that.

Charlie: Yeah, and if I could just tag onto that, so in preparing for this talk here that we're going to give, I dug into our biggest survey that Forrester does. It's called "Priorities and Journey," and it's a close to a 20,000-size panel. It's just an immense survey if you're a survey geek.

Andre: Sure.

Charlie: You know, this is not a few hundred people.

Charlie: Worldwide, we saw clear and, you know, really statistically significant evidence businesses everywhere are creating – saying that they have a priority of creating IT teams that report into the business. So I mean you can talk about and bemoan rogue IT, shadow IT all you want; it's a thing and it's clearly there in the numbers and it's increasing year over year. It's one of the actual indicators – new indicators we've put in the heat map as suggestive of terms

Jeff: I hate that term rogue IT.

Charlie: I know.

Jeff: It almost makes it sound like it's bad.

Charlie: I know, I know, it's terrible, but it's what –

Jeff: You know, if you look at like –

Charlie: – people are thinking. [Laughs]

Jeff: Well, that's what IT is thinking. [Inaudible due to laughter]

Jeff: But, you know, I'll give you an example. I was working with a client of ours, which is a very large agricultural equipment manufacturer. You know, they make, like, $500,000 combines and that sort of thing, talking to that side. So, you know, we've had development embedded inside our product development for years. We know how to build software, embedded software – not a problem, okay? We've had IT systems for years. We know how to build software that runs the business.

Andre: Yeah.

Jeff: But now we're trying to build connected products and we want to have that farmer that it's in the cab or maybe is in their office with a table or a computer and see how their products are operating. And you know what? We actually also want to be able to update that stuff multiple times a year, unlike the embedded software. How do we do that? Where does it need to live? How does it need to be organized? It's not in IT, it's not in engineering; it's in a digital organization or an operational organization.

Andre: Exactly, exactly.

Jeff: I can think of an interview that I had with a very large hotel chain. They were like, "Building the mobile app was one thing, but when we wanted to implement mobile door unlock, we had to go so deep into our operations organization 'cause they owned the relationship with the franchisees and had to convince them to put the hardware in to allow the unlock capability."

Andre: Yeah.

Jeff: In that world, those teams of devs and operators need to be as close to the business as possible to understand their concerns.

Andre: A similar situation exists in the automotive industry.

Jeff: Absolutely.

Andre: That supply chain, it's very deep.

Charlie: Yeah. And the thing is that it's a two-way street because the OT folks for years have just said, "Well, my systems are air-gapped, don't bother me," but that's not true anymore. And the one deficit that they had in OT was networking, of who actually understands the networking. It's your corporate IT people who actually have the networking background. So there has to be some out there, right? So…

Andre: So let me ask a question. So with all that sort of priority changes occurring within organizations and developers moving into business units, what's that mean to offshoring? What's happening in that world?

Jeff: Well it's really interesting. In our research we have seen a bit of a shift in how companies want to buy those external services, especially when it comes to development, and it's way from kind of a wholesale project outsourcing model towards something which is much more of an augmentation model, and I don't mean augmentation in terms of a body shop but augmentation in terms of specialized resources that come in that help to round out a team and that also transfer skills over time. So whether it's a pivotal labs method or if it's, you know, a digital agency that is providing data science talent and experience mobile development or some IoT development chops, we find that these organizations are using those service providers but also working with folks that can teach them how to change their delivery culture as part of that effort, and tools becomes part of that but also just experience and coaching becomes an extremely important part of that as well.

Charlie: Yeah. You can clearly see in the data the interest in DevOps and some increases in release velocity in countries like India. I was actually surprised, so…

Andre: So let's talk a minute about the intersection of Agile and DevOps.

Jeff: Okay.

Charlie: Okay. You first.

Jeff: Well, I'm gonna get up on a soapbox here. Maybe I should stand on my chair. But I think part of the disservice that those of us on the supply side of the industry have done is we've introduced so many terms and concepts that I constantly talk to our clients and I spend as much time trying to help them understand the terminology as the intent. So where does Agile stop and DevOps start or what's the difference between continuous delivery and DevOps or how do I think about versus ALM or, you know, if I'm scaling to Agile do I think about Safe or Nexus or any of these sorts of things? And it's like timeout, timeout, you know?

I fundamentally see two ways that our clients react to this implementation. One of them I think is good and one of them I think is not so good. A lot of our clients faced with that confusion of terms and capabilities and how they all fit together adopt a very dogmatic attitude; we're going to choose something and we're going to standardize it and we're gonna drive those standards through. The more holistic approach, the better approach from my perspective is to start focusing on individual tactics, processes, things, and the observable results from those tactics and get a little bit less hung up on am I a Safe shop or am I a Scrum shop or an XP shop or am I focusing on DevOps?

But things like are we using feature flags, do feature flags work, how do they work, do we use red/green deployment, does that help us get out faster – it does? – Let's continue to use it, let's double down. If not, maybe we'll move to the next thing. A focus on the results of the activities and the flexibility to adapt those to the needs of the team are what we tend to see in the most successful organizations that I would call Agile in spirit as opposed to Agile by process. So that's, you know, I think a challenge that our clients face.

Charlie: Yeah, absolutely. I agree with everything Jeff is saying. I had a couple more observations. Agile predated digital transformation tool and made sense for independent software vendors who were releasing software to be stamped on CDs that were gonna go into the – into boxes at the shelves of CompUSA or Best Buy. I mean you actually go and once upon a –

Sooner or later I'll be telling my grandchildren, "Yes, we used to go into the store and buy software in a box." And Agile made sense back then. I mean, you know, some of the core principles, you know, transcend, you know, the – or extend back to that point in time. But with digital transformation and the rise of cloud and the global internet, software delivered as a service is very different, and it's so easy to just kind of rattle of the cloud, you know, acts as a service, but it was fundamentally different as compared to going and buying software and sticking it in, you know, to your PC and loading it. So – go ahead.

Jeff: I'll give you an example of that.

Charlie: Yeah.

Jeff: When I was pressing software on CDs and shipping them in boxes.

Charlie: Exactly. You were rational.

Jeff: Rational.

Charlie: Yeah.

Jeff: We could not ship our product any more than twice a year, not because we couldn't do it but because our clients couldn't handle that level of loss.

Charlie: Couldn't absorb it, absolutely.

Jeff: And usually they were two to three releases behind. So what was the point of going any faster than twice a year?

Charlie: Yeah.

Jeff: If you look at where Amazon is, you know, it's what, in 2014 they were at 50 million releases a year? The Netflix guys – there's a great quote from their chief technologist – there are 33 million different versions of Netflix. I feel like I have five of them in my home.

That's the difference between where we were then –

Charlie: Yep.

Jeff: – at the time of the Agile manifesto –

Charlie: Yeah, yep.

Jeff: – and where we are now with DevOps. It's incredibly granular and it's –

Charlie: Exactly.

Jeff: – powerful in a way that we just didn't have back then.

Charlie: And it's very much an R&D discussion. We were talking about this earlier today. You know, the idea that we have unknown unknowns and we're gonna use fast feedback because we need to test our hypotheses against what's actually gonna work, and that makes a ton of sense for actually the core intellectual problem of developing a software. But when you say I'm gonna develop it, it's gonna be great, and now I need to put it in an operational status as a service that will deliver somebody a moment of truth, this is a somewhat different conversation. And guess what? There are some repeatable things in between X and Y that can be automated and should be automated.

I mean I remember the bad old days of 50-page run books and, you know, here's the 50-page install document and we're gonna have a human follow it and they need to follow it precisely the same way every time. Let's see how all that works. And so you started to see the automation. That then, of course, is test-driven development and we are now quickly going down the rabbit hole of the whole history of IT here but, you know, there are significant differences and yet there's so much commonality and there's so much of the one that led to the other with a big dose of Lean thinking, you know, applied via things like Gene Kim's interest in the goal and the work of Eli Goldratt; a lot of influences have converged to this moment in time.

Andre: Speaking of intersecting or converging influences, what's the impact that cloud is having on DevOps and development teams?

Charlie: Well let me take that one first.

Andre: Okay.

Charlie: So I'll – here's the thing: I talk to folks, you know, Agile advocates, especially if I may say, you know, some younger Agile advocates, and they'll say, "Waterfall; why was that ever a thing? They must've been stupid back then." I'm like, "No, these are some of the smartest people in the history of the human race that developed computers and, you know, were doing – working with computers in the '60s and '70s and '80s." Why would they have ever developed something as seemingly dumb as Waterfall? And I'll tell you why – physical systems engineering.

I came in on the tail end of that era and we weren't just writing code. I was installing PeopleSoft ERP systems for consulting, and you had to size those things precisely for the user base. You needed to understand the performance characteristics of the transactional load. And guess what? If you ordered the wrong size HP boxes or the wrong size EMC storage array, that meant that some vice president somewhere had put their name on a $10 million check and you didn't want to iterate on that. It was not good for your career; and so Waterfall.

Once cloud services became more fungible, flexible, virtual, that's when Agile exploded. The Agile concepts were around in prototypical fashion all the way through the whole history, but they only really took hold when people started not having to engineer and design the physical systems specific to the application.

Jeff: Well, the software, too. When I was in as well, when we went to CAPS you got three compiles a day.

Charlie: Yeah.

Jeff: So it's really hard to think about doing Agile when you only get three compiles a day.

Charlie: Right.

Jeff: You desk check your code.

Charlie: Yeah, yeah.

Jeff: You know, you do those Waterfall-types of things.

Charlie: Yeah.

Jeff: I think one of the big things that we've seen with cloud though recently with devs perspective is the slow change in thinking from, you know, managing your servers as pets to managing your servers and your services –

Charlie: Right.

Jeff: – and now your containers as cattle and –

Charlie: As cattle or ants.

Jeff: Yeah. And from a dev, you know, that is so liberating –

Charlie: Yes.

Jeff: – because, you know, we don't write perfect code. I'll be the first one to say it.

Charlie: Right.

Jeff: If you are able to begin to start working in a way where the pressure upon having to be perfect is reduced and you can start to focus on being fast and you know that, you know, you don't have to – you've got a little bit of a memory leak then you just kill the thing and you move on or, you know, you work across multiple individual nodes, it becomes a lot easier. This is one of the reasons I think we've seen the rapid growth in something like Node.js and, you know, you're an EA by trade so I'm sure you remembered the first time devs started throwing node runtimes out there and you were like, single-threaded small servers? Why the heck would you even want to do that?

You can't scale up with that kind of stuff.

Andre: Yeah.

Jeff: And yet you have architectural patterns that enable that sort of thing and we get to the point where this works. So, you know, that punctuated equilibrium, that shift in thinking and architecture has just become so powerful that it has essentially enabled us to cast off the shackles of the $10 million sizing. We don't even size anymore.

Charlie: No.

Jeff: It's gonna be what it's gonna be.

Charlie: Yeah, it's gonna be –

Jeff: And we'll optimize after the fact.

Charlie: Exactly.

Jeff: And so from a dev perspective that's just been liberating.

Charlie: Yeah, yeah. It's been liberating for a lot of us, although I will say I like fleet vehicles versus collectible cars, 'cause I spent time on the cow campus at the University of Minnesota and large animals actually have veterinarians and they don't just shoot them.

This was never a good metaphor. They never just shoot the cow when the cow gets sick.

Yeah, anyways – but, yeah.

Andre: So related to cloud is the whole container movement and trend, and especially now Kubernetes seems to really be taking off. What's your perspective? What are your clients coming to you and asking about that environment?

Jeff: You want me to take this one first?

Charlie: Go for it. I'm kind of…

Jeff: Okay. Well it's interesting, from a developer's perspective I have to say I struggled with containers a little bit at first and the reason for that was, well, how is this any different than VMs? I mean 20 years ago we were using VMware when it came out and putting it in our test labs and, you know, creating images and throwing them down and then we were doing AMIs and we were putting that out there and things were working. You know, it's like this is just the next turn of the VM crank, the image crank. And that's not the case anymore –

Charlie: No.

Jeff: – because we are starting to see the beginnings of programming models emerge –

Charlie: Exactly.

Jeff: – around containers that let us do more interesting things. And so as we start to call those patterns and those architectures cloud-native it begins to become much more interesting from a developer perspective. The idea that if I package to a container I can run pretty much anywhere I want to on-prem, off-prem. You know, I heard somebody describe, you know, essentially containers are Docker as the modern day POSIX from a developer compatibility perspective. I don't know if it holds or not but that level of empowerment I would say, well, it's kind of like the modern days Windows.

I can run it anywhere that there's hardware and be successful with it and do it with the minimum of investment of my part. But as we start to put these additional programming capabilities around those containers and we start to see service meshes of containers and we start to see really robust telemetry from these containers, it's gonna be a perfect complement to the sort of microservice architectures that we're trying to build from a developer perspective, mapping those into containers, mapping them to smaller-scale run times, and it just feels like it's gonna be a much better model than the traditional VM image. So that's the dev perspective.

Charlie: Yeah. No, I think that whole Kubernetes is the new POSIX, I think I actually got that from...

I thought it was brilliant.

Jeff: It's in the...now.

Charlie: Yeah, it's a – oh man, it's a thing. That was the most outrageous thing I'd heard all year and it just made my day. So from an infrastructure side I think we're still very much trying – you know, it's another – so first of all, you know, from kind of a pragmatic point of view, it is another layer. On the other hand, you can actually re-platform – I saw somebody had re-platformed I think virtual desktops into containers. You know, so there's even, you know, some things that we just kind of consider as a done deal, it's old – you know?

So there's even some implications there. The thing I'm looking at is I'm looking at things like manageability, observability, and you know, trying – how do we troubleshoot this stuff, how do we actually successfully operate it? One of the things that scares me the most – and I'm gonna call out to Charity Majors as I got this – I call this Majors Hypothesis – I don't know if she would identify it as her hypothesis, but I want to be fair and give credit where I think it's due – is that as we automate and we make these systems more and more resilient and robust and anti-fragile, choose your term of choice, what's left in terms of the actual problems we're having? It's all of the ugliest, nastiest, hardest problems. You know, the finding the needle in a planet-sized haystack is the metaphor.

And so what is this going to mean in terms of well how do we manage this stuff? How do we trace transactions? And so in addition to, you know, Kubernetes and that landscape, I'm also looking at the transaction-to-service mesh frameworks, the Envoys, Zipkins, trying to make sense of that stuff beyond just a superficial level. There's some major transitions going on here, absolutely.

Jeff: Yeah, and so much of that stuff just kind of goes naturally on top of Kubernetes.

Charlie: Yeah, it does.

Jeff: It's a starting point.

Charlie: Yeah, exactly.

Jeff: …or any of that sort of thing.

Charlie: Yes.

Jeff: So it does feel like it's hitting de facto standard in terms of the number of –

Charlie: Yeah.

Jeff: The providers that devs care about building stuff on top of supporting.

Charlie: Exactly.

Andre: Well, Charlie, Jeff, thank you very much for a great conversation today.

Jeff: It's a pleasure. It's always nice. We had, you know, no fights between dev and ops here. This is a good thing.

Charlie: We've gotta work harder.

We probably could've found something.

Andre: That was awesome.