Episode 101: Charlie Betz on Building Better Business Outcomes with Software Maturity

Brian Dawson: Hello, thank you for joining us for another episode DevOps Radio, I'm Brian Dawson and I'll be your host today. I'll be joined by a gentleman that I've had a number of enlightening conversations with, Charlies Betz, a lead DevOps analyst at Forrester.

Hello, Charlie. How you doing?

Charlie Betz: Brian, how you doing?

Brian Dawson: I am doing well, I am doing well. And as you know I always enjoy a chance to talk about things with you. The market overall as well as things outside of the market like social issues. Today however, though, I'd like to take some time to learn a bit more about what you're doing at Forrester, how you ended up there, and then I'd like to dig into kind of the state of software delivery maturity and talk about a white paper that we worked on called The Software Delivery Maturity in Power Developers with End-to-End Automation Orchestration and Collaboration Report.

Before we dig in, Charlies, can you start by giving our listeners and overview of what you're doing at Forrester? What is your role today? And then maybe you can go back and tell us a bit about your career background and how you got there?

Charlie Betz: Absolutely. And thanks for asking. So today I am a principal analyst and I cover DevOps, enterprise service management, and more broadly, the transition to next generation IT operating model especially having to do with product centricity, the trend to product teams and platform teams. And all that that implies for how the modern digital professional needs to think about their daily activities.

Brian Dawson: Well, and real quick before I take you into the career background, is there – well, a couple of questions, one, what are you working on at this moment? Like what's the exciting research on your plate today?

Charlie Betz: The exciting research really is about the new operating model. I mean we've been aware that the traditional IT operating model, which is based on deep specialization, people working in specialist functions like programming or systems administration, database administration, testing and Q & A. And so, you develop that really deep domain identity and then you're brought together on a temporary basis, you could call it matrix management with projects and processes.

And this all getting mixed up and rearranged and refactored with the trend towards product thinking and in product management you tend to have deeper and more permanent relationships with a variety of peers who have different skills. And you all come together to deliver an outcome for the customer or the consumer.

And this trend towards product centric thinking, I just recently released a paper The Product Centric Operating Model is Upon Us. And it really is overtaking a lot of traditional IT organization as they pivot into product centricity but boy, Brian, is it raising a whole lot of questions around it. Finance and governance and shared services and platform teams and what happens when you still need some specialization and the questions just, you know, they don't let up.

Brian Dawson: Well, are organizations – so on that, and I can't help but dig in a little bit. It almost seems like we're swinging back especially as we talk about enterprise DevOps or software delivery where there's been a hyper focus, you know, obviously, we're still talking about shifting left, in a sense, but there's been hyper focus on full stack, right? On a single person or single set of people having decross domain expertise.

I believe that it sounds like you're finding that the enterprise, you know and the reality of an enterprise where you have complex, multiteam software releases, the project to product orientation is a better way to approach it.

However, in my experience that also, people have struggled to take legacy organizational structure, the finance, the HR management, how do we incentivize, etcetera, etcetera, and reorientate around these cross functional teams. One is that is my assessment correct in your mind but then also, are we finding that we're at a place where organizations really can embrace and absorb this new orientation of cross-functional teams?

Charlie Betz: Well, I think that there is some missing pieces, Brian, and I'm glad you mentioned from Project to Product ‘cause it's a great book by Mik Kersten. And he really nailed it and he nailed it at just the right topic at just the right time and it's having a global transformational effect.

And the reality is, is that the full stack engineer is a myth. Plain and simple. You know, I mean, you know, and the risks of the full stack engineer were well illustrated in The Phoenix Project, which I was one of the initial reviewers on way back in the day, in the character of Brent. And those of you who have read The Phoenix Project know who Brent is and those who haven't read The Phoenix Project you need to go read The Phoenix Project because it's one of the pivotal texts in the history of DevOps.

And Brent is that full stack engineer. It's probably overstated to say they don't exist but when they do exist and you start to become comprehensively dependent on them, they become a constraint. And they will always be rare, purple squirrels in HR parlance.

And so, you can't base a team on full stack engineers. Not unless you have, you know, immense amount of cash and tremendous hiring leverage to bring them in. You have to assume that teams are going to need to specialize to some degree and the interesting thing is, is that product teams also specialize they just specialize along a different dimension.

They're no longer specializing in their expertise or their skills, they're no longer specializing as Java programmers. They're specializing in search or in credit authorization, or in shopping cart management. They're specializing in some particular thing that needs to be delivered to do a given job. And in that sense, they still become specialists.

Now the challenge really is, is how do we then support layers of these product teams? And you have some teams that are going to be closer to the customer, end customer experience and you're going to have some teams that are going to be further away from the customer experience, you can call them platform teams. And these platform teams then support your more customer adjacent teams and you wind up with a stack structure that looks similar, maybe different, but also similar to what's come before.

And I think in some ways, you know as we reinvent IT that we wind up with a mix of platform and product teams. Everybody uses product principles, everybody is more customer focused, they're continually improving but at the end of the day, you've got people who are specializing still in network and compute and storage.

And maybe what they're doing is they're curating the cloud offerings there but you still need somebody to determine well, which cloud offerings are we going to use and which are the desired configurations that have been proven to be secure and performant in my environment? These are still hard questions that have to be answered by the organization.

And so, I think that this is where, you know, we see, you know evolution, we see revolution, and we see, you know, in some ways that there is nothing new under the sound.

Brian Dawson: Right. Right. And, you know, and I'll also add a plug as we talk about texts before I shift gears a bit is look at Manuel Pais' Team Topology, listeners, and I probably reference it in the past, but it does hit on these sort of different organizational models. How you put together DevOps teams and speaks to platform ops in a sense.

Now Charlie, before we transition ‘cause you really set me up well to get into this discussion on software delivery maturity but before we got there, you know I kind of wanna be you when I grow up, right, an analyst that people, that gets to research, gain a deep understanding of things, pair it with their experience, and have people listen.

Tell me, how'd you end up were you're at today? How can I become Charlie?

Charlie Betz: [Chuckling] Well, it was kind of a checkered career here up in the upper Midwest and I certainly did set out to be an IT professional. I had early, you know I, early on, went into the nonprofit sector and had a bit of an aptitude for computers. I, you know, there are those in the audience who will recognize the name Oregon Trail, and the Oregon Trail was a product of the Minnesota Educational Computing Consortium, which also did things like put computers in every high school classroom around the state.

Brian Dawson: Wow.

Charlie Betz: And so, I'm an alumnus of that, that whole effort, you know, going back more years than I care to admit. And without going all the back, too far back into time like oh my God, he's going back to high school.

[Laughter]

But the bottom line is, is you know, when I got into the nonprofits I'd already, you know I'd programmed an HP basic, you know, and so, it was not hard to pick up some database programming and the nonprofits needed it.

And then I went on and got a degree in software engineering and after some work at the University of Minnesota went into Anderson Consulting back in the day, which then became Accenture whilst I was there.

And then, a tour through Target and Best Buy, ultimately wound up at Wells Fargo. And I was doing a variety of things that were kind of obscure but were all adjacent to the problem of what I'd call the IT control plain. You know, I mean I went into Target and I got tasked with building something called a metadata repository. And most of the audience, at this point, is probably what the heck is a metadata repository?

Charlie Betz: Yeah, they were a thing before CMDB's. Now and half the audience is still saying what the heck is a CMDB? But basically, it's all around this question of how do we manage the IT delivery capability effectively and efficiently? Because a lot of it was done using e-mails and spreadsheets and had no automation.

Now we didn't have, you know, enterprise source control, we had pockets of it. We had, you know, a ticketing system at best. We did not have the degree of automation that we see, you know, in organizations that have a full continuous delivery tool chain and a full enterprise service management and full-blown, you know, well mature monitoring and log management. All of that was a glimmer in folks' eye and we were just trying to figure it all out.

Charlie Betz: And, you know, metadata was one of the ways that, you know one of the kind of early discussions that kind of then led into other things And it, ultimately, was why I wound up talking to Gene Kim in the 2005, 2007 timeframe was because of an interest in well, how do we make IT perform better? It was the Keystone Cops; it was the gang that couldn't shoot straight.

Brian Dawson: Yes, yes.

Charlie Betz: It had a horrible corporate reputation. You were in IT in the year 2006, you know, you were low on the totem pole.

Charlie Betz: You know, you were in an organization that could not keep its promises and you were not well regarded. Things have improved, they really have.

And so, Wells Fargo, ultimately, I was the lead architect and a VP for the business of IT. I took a leading role in the Wells Wachovia merger in terms of application decommissioning, application portfolio management. That's just another aspect, another facet of this problem of the IT control plain. Again, how do we manage the digital capability at scale when the budget is literally $5 billion, with a B, a year?

I mean the spend that I started to get into it was just, just a massive, massive, fascinating business problem. I had an all-access pass. Then I took a lateral, I went off and became an analyst at a little boutique advisory firm, EMA, still got a lot of respect for them, they're good folks there.

And then, another surprising this pass through AT&T for reasons I still don't quite understand. And then, I took some, I did some solo work and then, then the job at Forrester opened up and it was just a great fit, so now here I am.

Brian Dawson: Well, I am, and refresh, how long have you been with Forrester?

Charlie Betz: Four years, I just completed my fourth year, I'm in my fifth year now. It will be five years in February.

Brian Dawson: Yeah, I believe, I believe that I got a chance while at CloudBees to spend some time with you within like your first, like one of your first few trips. And so, just to say it's been always loved the insight and information you're able to give. I've had, you know, it's been a pleasure to work with you so I'm glad that you made this choice.

Now, we won't get into but I'm just going to plant a seed, I'm not going to have time to get into your, your, your, your social activities. And really kind of what your overall orientation is that I think, you know, brought into your nonprofit work and some of the work that you've now done here. Sorry to tease you, audience, but Charlie, hopefully --

Charlie Betz: Another day.

Brian Dawson: - we will get a chance to dig in and talk a bit more about that. I'd just Charlie potentially, so you all people who know him, they get a chance to talk to him, he may be able to compete with the Dos Equis man for being the most interesting person in the world.

Charlie Betz: I don’t know if that's a good thing or not.

Brian Dawson: Let's next, Charlie, go to this report that we recently got a chance to work on together. A collaboration between Forrester and CloudBees on a report that as I kind of screwed up earlier called Software Delivery Maturity: Empower Developers with End-to-End Automation, Orchestration, and Collaboration.

I'm hoping that you can give an overview of what the overall premise of the report was.

Charlie Betz: Yeah, absolutely. I mean the premise was, you know, basically, that, as I kind of alluded to earlier, as we continue to scale digital systems up, the problem of delivering those systems, and in particular, software delivery there's still unrealized potential there.

And we hear over and over again people, they do a proof of concept, they maybe do a pilot, they like the results but as they try to really scale up DevOps and digital and agile, they get stuck. And yet, there is real value in being able to scale these things up and we illustrated some of that value and doing some of the correlations and the analysis that went into this report.

It was super exciting to, you know, see the overall conclusions that are supportive of consistence of forums and approaches to collaboration, the benefits of unifying tools. But then I think a very nuanced message on unification of DevOps platforms and especially the business benefits of high software delivery and maturity.

And I'm really just pleased, you know, as punch that we were able to again validate. Some insights that came out I think were, you know, the pioneering work there, you know, of course, credit due to Dr. Nicole Forsgren and Jez Humble, Gene Kim and the accelerate state of DevOps research. And I'd done my own survey last year, state of MTO that touched on some similar topics. We did a lot of correlation analysis there.

And here again, in this survey, in the CloudBees Software Delivery Maturity work, again we see that high software delivery has tangible business benefit is absolutely correlated and we can, you know we just see such a consistency of evidence there. o that was super exciting to partner on this. And again, seeing the data, you know, replicable research here.

Brian Dawson: Well, so let's dig in for a moment into some of those key findings that you went over. The first you mentioned is common workflows. And this concept of the need or the value in creating common workflows across your software delivery organization. Can you comment on what you found in terms of where people in maturity and why people should be looking to improve? Why people need to establish these common workflows?

Charlie Betz: Well, I think it is very much about – I'm just going to pick on one specific aspect that's kind of near and dear to my heart and that's the problem, the coordination problem. Because the coordination problem is really where, you know, agile as it scales, is encountering some weaknesses. And this is, you know, work that I've seen and, you know, reported by other thought leaders, I've seen it in my own research.

You know, it's great if you've got one highly independent product team, yeah, sure, they can move at speed and there's a lot of focus on keeping the product teams loosely coupled so that they aren't stepping on each other. But I have to emphasize that is a north star and ultimately, it's not, it's very dangerous to confuse having a north star with saying that we never have the problem of dependencies and coordination.

You are going to have the problem of dependencies and coordination. You can do everything you can to minimize them with automation and APIs and blah, blah, blah, you're still going to have the problem of coordination as you scale up.

And this is where the story was not great for the high performers and truly dismal for the low performers.

Charlie Betz: So fewer than 40 percent of the high maturity respondents reported strong coordination and high visibility across their S DLC. So less than half, you know, well less, less than 40 percent.

Brian Dawson: And these aren't – to interrupt on that real quick – these are people that we identified are meeting business expectations.

Charlie Betz: Yeah. No, they were, I mean they answered positively across a series of 17 questions about their software delivery maturity and their software delivery organization. And they were the top 25 percent on these maturity questions. Questions like development teams are able to push code all the way to production, we manage inversion, all deployable assets using repositories and registries, we routinely pass security and regulatory compliance tests and audits. You know, agree on a scale of one to five.

Charlie Betz: And then, we took the top 25 percent across these 17 questions and fewer than 40 percent of them reported strong coordination. The lowest maturity respondents were down in the single digits.

Brian Dawson: Wow. Okay. So there's still a lot of work. I guess you answered my question about where the industry stands. Establishing these common workflows are arguably critical and we still have a lot of work to do or look at it as half full, we still have a lot of opportunity to improve and optimize our process.

Charlie Betz: Yes.

Brian Dawson: So let's double-click on the unifying tools. Man, you know in the report you make a statement that there's a key finding is that unifying tools enables better visibility and collaborate across teams and process. To an extent it helps support or service the common workflows. Can you tell us a bit about the why and where pl stand in terms of unifying tools?

Charlie Betz: Yeah, it's an interesting and like I said, kind of a nuanced set of insights that we developed here. Because on the one hand, visibility, having the information that you need is critically important. And visibility, I think, tracks with more platform integration, you know, just as simple mechanics here of not having to cobble together reports across multiple sources. And yet, high maturity organizations were more likely to say that they were using more of a best of breed strategy. And while the low maturity was more likely to use a suite.

And so, I think that really what we need to focus on is this suite spot, so to speak --

Brian Dawson: Da-da-da-do.

Charlie Betz: -- of having suites that then also support integrated reporting and coordination. And then, certainly, I also would inject that personally, I'm very concerned about the security exposure of pure best of breed strategies.

And so, its definitely a moving target right now in terms of what the numbers are telling us. You know we're saying and we're seeing the people, they want they coordination, they want the visibility, they want the unification of reporting.

But then also, there still is a desire to use the tools that we become familiar with, the tools that may go very deep into particular focused areas. And it's an old conversation, Brian, I mean best of breed versus integrated suite, you know, this is not the first rodeo, right. You know, IT industry has wrestled with these questions for a long time now. And so, it's not surprising that they're arising in the continuous delivery tool chain itself.

Brian Dawson: Yeah, and, you know, I suspect, a couple of comments here and I'll get your thoughts. As a person, you know that has traveled the evolution of version control systems, for instance, right, this moves towards centralization, decentralization, and back. But I think it's fair to say that in the early stages of kind of the tooling and really process and practices of CICD that we'd have loosely federated groups of tools and people would cobble together solutions.

But I'm starting to see signals, including hearing from you that IT leaders are looking to mitigate or temper the loss of velocity and cost with building bespoke tool chains, and it's kind of the focus on platform of ops, some centralization.

But, on the other end we'll talk about platform ______. Like we're seeing a lot of vendors build platforms and some vendors and not to make this a vendor specific conversation, I mean it even happens in Open Source, but focus on, like you say, a suite. Having a single tool chain with built in sets of tools so you really arguably only have one tool to deploy. However, now all of this is going to be just to leave it open for comment for you, Charlie, as you probably heard me say, the one thing that is constant in enterprise technology is change.

Charlie Betz: Right.

Brian Dawson: So it sounds to me and I, you know I start to worry, I would tend to warn people that yes, you don't want some of everything under the sun. You don’t want to have manage licensing, integration, etcetera, for everything. You need some level of standardization but that it's unrealistic to believe that you're going to be able to standardize on a single tool chain for perpetuity.

I'm curious, you know, am I wrong? Do you agree, disagree? What have you uncovered in that area?

Charlie Betz: Well, I think that there always is a trend towards commoditization and I've been studying, I mean one of the things I've been studying in the last couple weeks is Wardley Mapping. And the classis Wardley map, the horizontal axis is an axis of commoditization a move from higher variation and single purpose bespoke, custom built, you know proofs of concept through productization. And ultimately, you know mass commoditization.

And I think that this is a pattern in industry that just continues to repeat. You get the commoditized offering, you know you get the – you know for example, public cloud. And it enables new innovation that starts to come in at a higher level and then that gets commoditized. And you're just never done.

But at a certain level, I mean this problem of, for example, what should the continuous deliver tool chain look like? That's been solved for a long time now. It wasn't obvious, I don't think that 10 years ago it was particularly obvious that, for example, we would pull source control and package management apart.

When I was actually writing code, we checked binaries into the source repository.

Brian Dawson: That seems wild. I mean I guess we did too early in my career but it seems wild.

Charlie Betz: Yeah, I know, it seems crazy. And then everybody said well, you know, it doesn't feel right, it doesn't look right, it doesn't smell right, you know, we need two different repositories. But, you know, and I'm a big fan of like alternate timelines. You know, folks, in an alternate timeline that didn't happen. You know? [Laughing]

And in this particular timeline we did long ago. It's source control, build management, package management, deployment, and then product you know. And those are the basic elements of the architecture. But other architects, other architectures could have existed. We've now stabilized on this and so, why do we want to, you know, necessarily, insist on best of breed, you know, when these are becoming well understood functions?

Brian Dawson: And this is where, at risk of injecting my opinion before I pull you elsewhere is I think, right, this is where we do tell, like you say, the suite spot, right, quote, da-da-da-do, pun not intended there. Is, you know, there's absolutely kind of the delivery pipeline and process that we can commoditize on. Yet the landscapes through which you deploy that do tend to vary.

So having a little bit of shock absorption, flexibility to allow my back-end team to leverage the best of breed solutions for their particular domain, allow my mobile team to leverage tools best for their domain, but at some point, connect at a common layer. So they had the common processes. Or an organization that's 50 people that works in a given way versus an organization that's 1,000 or 10,000.

Charlie Betz: Yep. Right.

Brian Dawson: Well let me, ‘cause as I was worried about I can easily, Charlie, just burn time that we don't have digging in. So I want to actually pull a forward a little bit and talk about what's sort of the third key finding. And that has to do with the fact that you found that higher software delivery maturity unlocks continuous business benefits i.e., if we can deliver software well then, we can better serve the business.

And you know, and that's just to say I'll go thinking about a bit of sort of social concern is serving the business may mean driving ARR, it may mean driving a better business outcome or having higher social impact. But in short, if we can deliver software well, we can deliver more value and have better outcomes, right?

I guess I'd say how did the maturity of the software organizations that you studied impact an enterprise's overall ability to implement digital transformation? And to what extent were you able to identify positive business outcomes? Are there any that you want to highlight?

Charlie Betz: Oh, we definitely saw them. I mean we saw them across the board and this is published in the report, customer loyalty. You know the high performers surpassed on customer loyalty, innovation, user adoption, market share, greater flexibility to adapt to changing market conditions. They were high maturity organizations were more likely to say that they responded to the pandemic better than their peers.

One case study that I heard that supported that, you know, there's a major US bank that had worked very hard on pivoting towards an agile operating model. And very specifically that enabled them to stand up and very quickly deliver on the paycheck protection program that the federal government put in place about a year ago. And if they hadn't had an agile stance, they would not have been able to do that.

And so, we're seeing it in the aggregate data, we're hearing anecdotes that support it, and again, this is compatible with, you know it's consistent with other research that has taken place using different methods, different organizations, different times, and places.

So we continue to see that this is not just a flash in a pan, you know that there is at least correlation, and yes, correlation is not causation. [Laughing]

Brian Dawson: Right. Spoken as an analyst.

Charlie Betz: You know, have to say that, but nevertheless, you know, we see the benefits. And I think that there are a lot of big organizations that are moving down this road and these are smart people. And they've seen the data, they're convinced, and that's why this is a large scale, that's why large-scale transformation is happening.

Brian Dawson: Well, there's an interesting thing here when we talk about, you know, and digital transformation, of course, you know, it has a lot of breadth to it.

Charlie Betz: Yeah.

Brian Dawson: Land depth arguably. But dig into where the study showed measure business outcomes exceeding expectations. We found that as you exceeded expectation significantly in user adoption. Now interestingly enough between medium, high performing, medium performing, and low performing organizations there wasn't a huge variance in innovation.

But what I found interesting is you know, not a huge variance in innovation but still better customer loyalty, better user adoption, which led to the point where there's the widest amount of variance, market share. So the high performing organizations were 36 percent of them were exceeding market share expectations while only 14 percent of low performing organizations were exceeding expectations for market share.

Charlie Betz: Right.

Brian Dawson: And I mean there's a lot, I mean I guess I have to remember you're the guest, I want to play with what I think is there and I know I sprung this on you out of the blue. But is there anything that you derive from the fact that the different maturity organizations generally tend to be able to innovate the same?

Charlie Betz: Well, I think part of the problem is in even measuring and conceptualizing innovation. And you gotta remember, I mean, okay, these are the base, the sample base was IT decision makers. And how many organizations and first of all, you know, innovation kind of implies that you've got a strategy of market leading, a product centric market leading, you know a strategy to lead the market with product improvements.

And there are other market strategies. You can succeed very well with customer intimacy or operational effectiveness. And so, I don’t know just how the concept of innovation shows up for somebody when they're taking a survey like this. It could be that hey, we're doing great, we're making money, our customers like us but it's not necessarily because we're innovating all the time.

So I just, you know, I just in general, not of caution there.

Brian Dawson: Yeah, and I’ll take the caution but then still, overlay what I think might go on here. And there's a scale factor that I may hit on but I think it may speak to the fact that we believe we can develop innovative features, innovative capabilities, we have an innovative product. But at the end of the day, if we look at the software delivery management space, the value stream management space it's this realization we'll harken back to what IT was, how you characterized it earlier. That committing code and generating a binary that has innovation in it is only part of software delivery maturity.

Only part of the process and I'd argue only part what true digital transformation provides you. It's not just that we can develop more cool stuff but we can sustainably, we can deliver that cool stuff to market. We can sustainably develop that cool stuff and innovate. And that ultimately, if we can do that in a repeatable, reliable, rapid manner we are more likely to gain market share.

So I know I may be reading a lot into that but you know, that's sort of my take on it. I get that mm-hmm, yeah, you may be over reading that a bit, Brian.

Charlie Betz: No, no, no, I'm agreeing.

Brian Dawson: Okay, okay. But yes, I think that there's that qualitative stuff that's really hard to capture.

So shifting off of this for a minute, you also found that there were low maturity organizations. And you surfaced some challenges that they had, some roadblocks that they needed to overcome. Can you highlight some of those?

Charlie Betz: Yeah, absolutely. Well, I mean, GRC was a huge challenge for them and you know, governance, risk, and compliance. I mean they clearly were much less likely to characterize themselves as very mature or they were more likely to characterize themselves as somewhat mature.

And you know, of course, in today's environment, yeah, you can't compromise on maturity. So you know, a GRC, governance and compliance, I mean it translates to security. So that's, I think, one of the clear ways in which software delivery maturity has real consequences.

I mean we've talked about the business outcomes, you know, that low maturity. I mean we did also look at, for example, how is it going with your DevOps transformation per se? And the low maturity organizations would say that they were – six percent would say that their DevOps transformation is exceeding business expectations. As opposed to 35 percent of high maturity respondents saying that the DevOps transformation is exceeding business expectations.

So you've got transformation handicaps, you've definitely got security and governance handicaps, and certainly, we also saw, as I mentioned earlier, transparency, coordination. Information was a problem across the board. We had fewer than 3 in 10 high maturity respondents who said they had all the information they needed in a single report to understand software release status. Fewer than 3 in 10 but it was two percent for low maturity, you know.

Charlie Betz: I mean, really, you know, these are organizations that just have some just fundamental lack of, you know, fundamental problems with transparency in situational awareness even understanding where they are and what they're doing.

Brian Dawson: Yeah. Which is funny. It's interesting, I'd say it probably translates down to individual contributors, the various practitioners that are involved in the supply chain. It's hard to do your best job if you don't have context. Similar to how we called out in the key findings, right, it's not just control plain visibility. It's, as we look at the various stakeholders in the software delivery process, back to your point around organizing moving from project to product, is you have to have visibility across function so that you can coordinate all your deliverable.

Well, let's flip a bit and go to key recommendations. And I know we can't get through them all, you covered some challenges, what are some recommendations that you have for these lower to medium maturity organizations. And really, any organization in terms of how they could achieve higher maturity in their software delivery?

Charlie Betz: Well, table stakes is automation and this won't be surprising to anybody on the call who's been tracking DevOps for a while. I mean DevOps does not reduce down to automation that's a fallacy. We understand DevOps is also about culture and operating model. But automation is one of the three legs of the stool, if you will, and this includes automated testing.

That's probably the biggest gap we see in a practical level. People want to do DevOps; they don't want to invest in retrofitting an automated test ____ across their regression testing. And when you ask them well, how do you do regression testing and you get a lot of hemming and hawing anyway.

And then I'm a big fan of package management. I think package management is super important. It's a critical control point. It's where your outsourced, your commercially acquired, your Open-Source stuff comes together with the stuff you're building and give you your whole sort of bill of materials.

And so, you know, it's a conversation that in some ways – it's the DevSecOps conversation. And you definitely then also want to along those same lines you don't want to forget a quality assurance like static analysis, software composition analysis.

And then, of course, you know deployment, I mean what came before continuous delivery and continuous deployment? What came before was 30-page Word documents written by the developer in the final 30 hours of project on a crunch, you know just before the developer is going to go off on vacation. And then tossed over the wall to some poor operations dude or dudette who had never seen the thing before in their life.

You know, and fine, you know. No, automate that, you know, show that you have an automation script or an automation template resource, you know, recipe, what have you that actually works. And can lay these bits down and get them all standing up and signing in harmony. And don’t give me a 15-page Word document with some random screenshots. And yet, that was and I think it's important to remind folks that was the state of the art 15 years ago.

Charlie Betz: And we still see people doing it that way. It's automation, yeah, you know, it's --

Brian Dawson: So digging in on your automation – sorry to interrupt and talk about ‘cause we talked about platform. And one of the recommendations is to unify and I'll blend them together, continuous delivery and release tools into one platform. And then provide access an enterprise service portal. Can you dig a little bit into those recommendations?

Charlie Betz: Yeah. So I mean I mentioned I cover enterprise service management and basically, the question is, is so you're a new developer, a whole new team that's just been onboarded how do you even find the CDRA tool? You know you've heard rumor that it exists, you know. And this is not as silly a question as it sounds because not everybody is some organization or some company of 70 people where these things are obvious and you see the person at the coffee machine every day. Sometimes you wind up in an organization of many thousands of people.

And so, that's why service portals and service catalogs are important. And if anybody thinks that that's hopelessly old school, well, go look at Spotify Backstage, which is an Open-Source developer enablement portal that Spotify is now promoting. And what's the first primary architectural component in it? A service catalog. So if the cool kids at Spotify think that a service catalog is what the developers need, you know, maybe we should all revisit some of our preconceived notions around service management.

I think that actually shared services are super important. You know we've been bogged down in the conversation around change management that's a topic for another day. But there's definitely some good practices that have emerged in that whole space. And your CDRA how do people find the CloudBees admin so you can get onboarded?

Well, you know there's more than one organization where you would first go to Service Now or BMC or maybe Ivanti and you would look up continuous delivery and bang, there it is. There's the CloudBee service and this is how you get onboarded. And maybe it's just a quick conversation with a budget code and then you're fully automated from there on out. Yeah, we don't want to have to have a ticket and too much wait time, totally agree. But how do you find it, you know?

Brian Dawson: Right. And then, it sounds like and part of kind of a couple of the unify your tools but keep platform open as in my service catalog I need some flexibility to bring together services that address my need. I need to find them quickly.

But I also need to be able to deploy them quickly so I shouldn't have to – it seems like the report is implying and I'm saying this clumsily, Charlie, here that look, releasing software and the CI/CD component of it are not things that should be separate. It's one and the same as part of the software delivery process.

Charlie Betz: Yep, I agree and there are some nuances here that are kind of implicit in the product architecture of products like CloudBees, you know the higher release choreography that operates more human to human level versus release automation that's got to get down into the weeds with the bits and the bytes and the platform specific. And there's a basic distinction between platform independent versus platform specific that has permeated IT management for a long time now. I mean incident management a classic example of a platform independent problem. You're marshaling human beings to go deal with the thing but your incident management tool doesn't care if it's on Linux or the mainframe or Windows. Right?

Charlie Betz: So the same thing with release management, you know you release choreography and orchestration across those platform but your CDRA tool better be able to speak Linux.

Brian Dawson: Rght, you need that level of flexibility. Well, I know I have a couple more things I kind of want to get out of you here, Charlie, and then, you know send you off to go discover some more great information and help make the community better.

One thing that hopefully, you can comment on quickly that I found really interesting and I know we discussed this is when asked for which of your following target platforms is your software development organization developing software? It was interesting to find that mainframe, you know, serverless came in at 38 percent, containers came in at 43 percent.

But it was interesting to see that 40 percent were still developing for mainframe and 26 percent were still deploying to or delivering on bare metal servers. Curious if you have any comment on that? Any thoughts on what that indicates in terms of maturing or practices?

Charlie Betz: Well, I mean the bottom line is that this is the transformation from bare metal to virtual to cloud, people underestimate how many trillions of dollars of computers were sold. This is macroeconomics societal capital investment. It does not evaporate, right? It's going to take time.

We worry about icecaps and glaciers melting but it's these things take place on geologic timescales. You can still be confused about where the future is going to go but the reality is, is that there are some slow processes here. And yet, inexorable, which is why you pay attention to them.

Brian Dawson: Yeah. All right, awesome. Thank you for all that information that you shared. Now, before we let you go, I want to get into some of the standard questions that we ask on DevOps Radio of all of our guests. However, the first one that we often ask is about a dev-oops moment. That's not DevOps but dev-o-o-p-s like dev, oh, smack, I screwed up. I made a mistake.

However, for you because you're in a unique position of having such sort of a wide purview of what's going on in the industry and what's going on with DevOps in modern software delivery instead of asking you about a personal dev-oops movement I would like to know do you have any thoughts on sort of industry dev-oops, industry pain points?

Charlie Betz: Yeah, absolutely, Brian, and it's funny you should mention it ‘cause the image came to me a few years ago and there's an old joke and I think it's you mostly hear it in the south. And it's the observation made when you see a dog chasing a car and the question always is well, what might you ask the dog? And the question always is well, what are you going to do if you catch it? What will the dog do if he or she caught the car?

And that, to me, is the kind of macro, you could call it dev-oops is that DevOps and more broadly agile, caught the car. You know all of this bad IT practice that we've been kind of railing against and you know, get back to the metaphor, kind of running down the road barking at this car. And then finally, the car stopped and we could run up and bite the bumper or something, bite the tire but is that really the constructive thing?

And so, all of a sudden, you're in the driver seat, you know. And at one point, I even had a little slide of a dog chasing after a car and then in the next frame, the dog sitting, driving the car. Were we really ready to start driving the car because all of a sudden there was a whole bunch of stuff that we didn't necessarily think about? Governance and compliance and finance and CapEx versus OpEx and of course, you know, governance and compliance, actually, it's a little bit of an overstatement to say we weren't thinking about that. I mean some smart people were thinking about it at least in terms of very tactical, code level operational controls.

But what about governance, you know the governance conversation at the higher level of assurance and risk management and the overall risk appetite of the organization? All these things have to be considered when you're looking at making such radical changes because guess what, the previous old, stodgy way of doing things, you know, with waterfall and project management offices, project management body of knowledge, strict segregation of duties between dev and operations, all of the processes, procedures, and controls its all organizational scar tissue.

And what that means is that at some point there was an active organizational wound. It's an old stand, I think it's attributed to G. K. Chesterton, "If you propose to tear down a gate, be sure you know why it was put there in the first place." And I'm not sure that as a community that DevOps advocates can in all cases articulate why the gate was put there in the first place. And I think that's the challenge we have as we find ourselves driving the car, paws, and all, with our tongue hanging out, you know, right, it makes sense of this new experience.

Brian Dawson: Yeah, that is – and I'm starting to think of some responses as I was processing that dev-oops. And then you kind of hit it with the adage before you tear down the gate, make sure you know why the gate was there. And what I'm hearing is where we've made some missteps and/or risk making missteps, we're already running into this where we had what I would argue was a very practitioner led movement.

Hey, dev and ops, let's collaborate, we're solving the same problem, let's work together. But, but now this is no longer a grass roots movement. This is CIO, CTO down, it's a business movement so we've kind of caught the car.

Charlie Betz: Exactly.

Brian Dawson: And I see it as there's been kind of that left-hand side DevOps for want of a better word, right. Let's automate, automate. Let's all go full stack. Now we have to truly, repeatably deliver complex business critical applications at scale into a production environment and all of a sudden, we've sort of went oops, we didn't involved security, oops, right? And we're now just starting to tackle that.

Now I didn't get a chance to see that presentation, I would love to check it out. That actually, you know, we went a little off our normal path here, Charlie, in asking you an industry dev-oops moment. And I think that is a great one. I think before we move off of that, any lessons, right? So usually with a dev-oops, you know when we make a mistake there's a sort of a key takeaway point. Do you have any key takeaway in terms of that dev-oops?

Charlie Betz: Well, it definitely gets into and something that – I mean systems thinking has been part of the DevOps conversation for a long time. You know it's, I think, embedded in the original three ways. You have to look at the problem as an integrated system and that includes finance and governance, risk, and compliance, and organizational strategy, all of those things in some level get pulled in.

And that, I think, is remains and it's easy to say, you know, systems thinking, systems thinking, a lot harder to do. And we wind up maybe looking again at practices like architecture. I mean that's an area where I'm sympathetic to the DevOps advocates who've had bad experiences with enterprise architects. I have seen, I have worked in organizations with architects, I can't call them anything but ivory tower architects who did not really add a lot of value.

But I've also worked with very, very good architects and I've worked with architects who understand the business and who understand the systems that they're dealing with and had strong, strong practitioner backgrounds. These were people who could really write code at one point.

And now, they're managing systems of systems and they're not to be faulted for the fact they're not writing code anymore because what would they write it in? They're managing, you know, multiple technical stacks and their responsibilities are now more add a portfolio optimization level but they still need to understand how this entire mesh of stuff works together as an integrated system.

So a little plug for the EA.

Brian Dawson: Yeah. Well, EAs, in my experience, are becoming or not increasingly more important but increasingly more recognized and acknowledged as important as this DevOps shift matures. And it's interesting, I'll just call out that your dev-oops ties very much back to the things that we need to look out for as we shift to platform ops, we shift to platform teams as we look to drive holistic software delivery automation. It is having to take a systems thinking approach. It is having to be able to think iteratively and in an agile manner but not forgetting about planning for what happens when you catch the car. So I think that's very relevant.

Well, another thing I'm really interested in hearing from you, Charlie, because honestly, you are so well read and so well versed in our space and beyond. What is a resource, a book, a podcast, a person to follow, a blog, what is a resource that you absolutely recommend or resources that you absolutely recommend our listeners go out, get, read, consume?

Charlie Betz: Well, I have a couple obvious ones and I would be surprised, actually, if at least one of these hasn't been mentioned before. I'm a huge fan of Dan Reinertsen's Principles of Product Development Flow. That book is so widely cited in the agile community if we had a citations index in our specialist's domain, he would be right at the top, I'm sure.

And I'm also really liking John Smart's Sooner, Safer, Happier. That I thought was a cut above a lot of material that's out there. Very pragmatic, lots of great cases, lots of great insights. But I'll leave you or I'll close, I mean I'm giving you three instead of one because I wanted especially, I'm reading a book right now copyright, ii think 1971 and I think that the research dates back into the '60s by a guy named Jay Galbraith. Not J. K. Galbraith, that's John Kenneth Galbraith, that's an economist, a different guy.

This is Jay Galbraith who's an expert on organizational development. And the book I'm reading, he's well known for something called the Star Model but I'm reading some of his earliest work, in particular, a book called Designing Complex Organizations.

And Brian, what blows me away about this book is it's just incontrovertible evidence that there's nothing new under the sun with agile. They had all of the basic dynamics and solutions figured out decades ago. I mean there was a renaissance there was a huge push around questioning organizational management during the '60s, you know a lot of radical thinking going on at the time.

And that translated into people looking at the old tailoress models of the '20s and '30s and really moving past them in way that, you know, if you read it in the right spirit, you understand that they're talking about the same stuff that agile movement was talking about 30 years later. Even down to daily standups and cross functional teams and the realization that high variability work couldn't be proceduralized.

And just and so much, you know I hear people talk today and it seems like there's this perpetual, I would say dancing on poor old Fred Taylor's grave, you know. But the thing is, is that people have been criticizing him for 80, 90 years now and to some extent, you know, we've just gotta kind of say, look, you know, the new model is pretty well established and a lot of this stuff was thought about.
And so, you know, if you're interested in some of the, I think, intellectual, I wouldn't say roots because I don't think any of these signatories or the Agile Manifesto read Jay Galbraith, maybe some of them did but I don't think anybody necessarily did. But at least the, its antecedents, you know, the antecedents go way back.

And I think that's so important because it just seems like we just assume that there is this benighted era of purely procedural, very command and control. And everything was all command and control and nobody, you know there was not even a hint of agility or product discovery until the Agile Manifesto was written in 2001 and then the clouds parted and the sun burst down from the heavens.

That's not how it was. I mean these dynamics that we're dealing with have long, long, long, long history. But I will say if I think about what is actually different and what isn't covered in the classis reevaluation of OrgDov that we saw in the '60s and '70s and it's not just Galbraith it's also people like Mensberg. What is different and what is new, I think is resiliency engineering and its relative aspects.

Because even though they knew in1965 that product design was not the same as running the assembly line and that high variability work in the R&D department required different management forms and a lot more autonomy and all this stuff. That assembly line was not producing emergent behavior. When you walked out of the plant floor at night that assembly line, if it was making Buick Skylarks did not all of a sudden start making toasters, right?

Charlie Betz: Or making first a toaster and then a refrigerator and then a car and then a locomotive, you know. And in digital systems are failing and cascading ways. I mean if you had some part of it, some subset of the assembly line blew up, well, you'd go fix it. But in general, you didn't see cascading failures that could take out the whole manufacturing plant.

Charlie Betz: That is what is genuinely different with digital systems is this resiliency question, this problem of dynamics like cascading failures, just the sheer understandability of these systems operating at the limits of human cognition with what we can understand an even cope with when it comes time to fix them. That's the truly new conversation. Not agile.

And with all respect to my agile friends and colleagues in the agile community and they've done great work over the years, but we have to look honestly sometimes and ask well, what really is new here? What's really different? And that, to me, the more I study, the more I dig back, the more that that's where I think the real differences are.

Brian Dawson: And then, and I will tell on myself, I'm going to ask you to restate the author and the writing because I was on Oxford dictionary googling some of the terminology you were using while you were talking.

Charlie Betz: Sorry.

Brian Dawson: Oh no, no, no need to apologize it was actually rich and I'm serious and is opportunity for me to learn, which is why I love this.

But what is the text and the author again?

Charlie Betz: The last text was Jay Galbraith and again, not to be confused with John Kenneth or J.K. Galbraith but Jay Galbraith. And the book was Designing Complex Organizations. And it's an older book, you gotta buy it used, it's pretty obscure. And I think not terribly cheap even if you do find it used. It's just evidence of the antecedents that I'm talking about.

Brian Dawson: Yeah, yeah. And I can see the antecedents there but I mean, you know, the value in recognizing that I will just quickly state another conversation I had recently is that you sort of hit upon is the complex systems on which we operate today's modern software development and delivery. And when you talk about resilience engineering the importance that we acknowledge that and the importance that we ponder and think about how we're going to manage it, if not mitigate it, right.

But anyway, there's a bunch threads that I could pull there, Charlie. You've spent a bunch of time with me today, I don't have the right to continue to bombard you with questions so I'll only get one last one in here and that's we got a couple of minutes for you to just share final thought. Final thoughts around software delivery automation, around this modern software delivery movement. What would you like to leave our audience with?

Charlie Betz: Well, so it may seem like an odd closing choice but what I'm really interested in right now and what I'd love to hear from people about is the problem of shared services. So shared services are, I think, critically important to the modern DevOps team. You have a limit of in terms of your team members, ideally, your team is seven to nine people. And what is it that you need in the environment to rely on to shift some of the cognitive load away from you so that you can focus on discovering new value and delighting your customer?

And shared services are highly unpopular with many folks in the agile and DevOps world because they have a well-deserved reputation for taking too long and high-handed bureaucratic behavior. But it's the classic, you know you can't live them, can't live without them thing. And bear in mind that Amazon is a shared service, you know, and it works and people like it.

And so, how do we move shared services, generally, to be more Amazon like even in cases where there's expertise, human expertise involved? You know, how do we solve that problem of when you have a limited set of experts and they have limited time and you have more demand than they can accommodate?

And yeah, there's a lot of passion out there, there's a lot of, I think, fairly dogmatic responses that we should never need specialists like that. But in point of fact, in large, in organizations that scale up at the enterprise level this problem never goes away. I have yet to see it go away. It's solved for in different ways but in general, you know you still have this problem.

And so, how do we move to applying more of DevOps mindset of product thinking to this problem so that we have the platforms and services we need to continue iterating in this fast DevOps way?

Brian Dawson: Awesome. Great points. And it's funny that in my mind it also still ties back to, again, okay, we chased the car, what happens when we catch it?

Charlie Betz: Yeah, exactly.

Brian Dawson: Charlie, thank you for your time, as always, I just enjoy having conversations with you. I appreciate you sharing these insights around modern software delivery, modern software delivery automation, resiliency, and a number of other things with our audience.

Charlie Betz: Yep, my pleasure. Always great to talk to you, Brian.

Brian Dawson: Thank you all for listening to another episode of DevOps Radio. And make sure that you subscribe and stay tune for our next episode with another phenomenal guest.