Episode 99: Pete Hodgson on Feature Flagging and Scaling within Startups

Brian Dawson: Hello, this is Brian Dawson with DevOps Radio joining you with another awesome guest. We have Pete Hodgson, independent software delivery consultant, and overall – well, corny – overall practitioner extraordinaire. I'm looking forward to talking with Pete about continuous delivery practices, software delivery, and engineering best practices, feature flag management in Manchester, the Bay Area, and anything else that comes to mind. Pete, hello. How are you doing?

Pete Hodgson: I'm doing great. Thanks for having me.

Brian Dawson: Awesome. Awesome. Glad to have you, and excited for our conversation. Just to kick it off and get started, I'm sure I did not do you justice even with extraordinaire. I probably did not do you justice in terms of helping people understand your background. So can you start by giving me and our listeners a brief overview of what you do today, and then also what brought you to what you do today?

Pete Hodgson: Sure. I guess, so I'll start with what I do today, and I actually kind of struggle to describe it a little bit myself. So I'm an independent consultant and most of the work I do is with startups. And I provide strategic consulting somewhere in the intersection of architecture, organizational design, and engineering practices. So I've been playing around – like I said, I've been struggling to describe what I do. I've been experimenting with this cheesy terminology of sociotechnical architecture, which sounds incredibly pompous. So I probably – I don't know. Basically, what I think is my most valuable thing I do is work in the intersection of people and technology. I'm kind of – I have an engineering background and we'll go into that, and I understand technology, but I also understand how much it always ends up being about people, and organizational design, and communication systems and stuff like that.

So this whole thing that people have been talking about recently about sociotechnical systems, I think like @jessitron – I can't remember her second name, Jessica Kerr, and Charity Majors and other people have been talking about this a lot, I think recently, but helping organizations – generally what I end up doing is helping startups, that are going through some kind of inflection point in terms of scaling, figure out how to change the way they work or the way they architect things to support that.

Brian Dawson: So before we go into your background, I want to jump in and dig on that a bit, and then in my narcissism, I want to make a plug for myself. I really appreciate that I – so the intersection of technology and humans also ends up being something that not only I am personally very interested, inspired, and intrigued by, but I also see an emergence, if not reemergence, of that in our space. I very much appreciate that. And sure, maybe it sounds a little pompous, but if it's accurate, it's accurate. Right? I've come up with I'm a technologist who enjoys – and this is true. It's authentic, but it sounds a little again, pompous, right?

I see myself as a technologist, and what I enjoy and what I am good at is optimizing software development and delivery, whether that's optimizing code, that's driving reuse in open source, or that's helping inspire and enable people. But at the end of the day, that's where I get energy. Now I am, I was going to say I'm curious. As an independent consultant dealing with companies that are in an inflection point, can you give me a quick idea, what does that look like? When does someone call you? Why do they call you? And overall, give us an understanding of what engagement like that looks like.

Pete Hodgson: So the shape of the engagements are really varied, but I think the common thing is this feeling of the stuff we were doing that got us here isn't helping us the way it used to. Like the way we used to work isn't working anymore. The way we planned architecture – let's say we'll take continuous delivery as an example. The way we build and deploy our software grew when we were 20 engineers. Now we're 200 engineers, and it turns out it doesn't work anymore. When you say it like that, it's kind of "Well, duh", not surprising. We're 10x the scale, but a lot of times, it's like a boiled frog kind of thing where it's like, you're suddenly at 200 engineers. You're like, "Why is it taking us a week to get this thing out the door every week? We used to be able to do it in a day. What happened?" Generally, the answer isn't "Oh, we're doing it wrong." It's more like the practices we're applying now were for a different problem, and now the context is different, but we haven't adjusted that practice.

And particularly, with startups that are scaling fast, the inflection points come pretty quickly, and everyone's super busy and so often, there isn't as much time to give attention to that kind of, the meta kind of stuff of how we do things, and also particularly, with younger startups, sometimes there's just not that many people who have done it before. A lot of the people at the startup, it's like the biggest startup they've ever worked at because they were there – when they joined at 5 people, it's the biggest company they'd worked at, and now it's 100 people, and it's the biggest company they'd worked at. So I sometimes joke that basically what I'm selling people is my decrepitude. I'm old enough to have been around the block a few times and seen a few things. Sometimes it's literally the value of just saying, "That thing that you're trying to do, that actually has a name. If I tell you the name for it, you can Google it. There's a bunch of stuff you can read." Sometimes it's just that, helping younger teams get past that second order ignorance, mapping the terrain.

Brian Dawson: Interesting. If it's your decrepitude, I'm just going to let you know, I won't take it personal, but I'm sussing out that I have some years on you. So, but –

Pete Hodgson: That's good. That means I've still got – it's the gift that keeps on giving. Every year, the industry gets younger, and I get older, so it's great.

Brian Dawson: Yeah, I'll spin it. I'm going to embrace my ++ decrepitude. So I'm like decrepitude ++, so I think I hopefully have some value to offer.

Pete Hodgson: Even just calling it something ++ is a reference that a lot of engineers are going to go right past their heads.

Brian Dawson: Yeah, and then I'll get – I talk about it often here. I'm going to beg not to go down the path of pointers, memory management, and registers that will really show my decrepitude. I want to pull on the thread a little bit more before I let you talk about your background. This may be hard to answer because of the varied engagements, but is there a – what does success look like in an example of engagement?

Pete Hodgson: That is a great question. It is something that I have – I think about a lot, because fundamentally, I want to help people succeed. Success, for me, is being able to fade into the background, and for the – have left the organization in some way levelled up. So I did an engagement a couple of years ago helping a startup figure out their continuous delivery system for their mobile apps. In that case, success for me was there being a core set of technical engineers and engineering managers who a) we'd fixed up – figured out what we wanted the CD pipeline to look like and done some implementation work there, but more importantly, they had the vision for how to keep this thing going in a positive direction, and they didn't really need me anymore.

In another organization, I helped define a set of architectural principles and dispersed those throughout the engineering organization, so everyone was moving with what I call a lined autonomy. So individual teams had a lot of autonomy, but they were all moving in the same direction because we had this shared vision of where we're trying to get to. In that case, it got to the point that we didn't need to talk about it anymore, or certainly didn't need me to talk about it anymore because people were referencing these architectural principles in conversations, and there were Slack emojis through the different principles and all this kind of stuff.

For me, success is I can leave. I can fade into the background, and people miss me because of my sparkling personality but they don't feel they can't survive without me, and maybe if I come back in a year, things will be different, but they won't have fallen back to where they were before. They'll have evolved, and I'll be like, "Oh, that's a good idea. I should try doing that somewhere else."

Brian Dawson: Nice, and so I hear a couple of things with success, and I'm sure it's wide, and you probably stated it better than I will, but there's a level of sustainability. There's been sustainable progress, even if it's localized, if it's in a given area, and there's a bit of you have achieved incremental irrelevancy in that context. You've become less relevant to them because you've made sustainable improvement. Awesome. So there's a bunch I want to jump into in your background, Pete, but let me give you the floor again, since I took it away. You're now – and first, brave man. You're an independent consultant, going through, engaging with companies that are in a tough time.

And I'd say there's an extra dimension of difficulty, to a certain extent even in decrepitude, if you're an early-stage startups or mid-growth startups, with a lot of young people in them. That's brave. I haven't been brave enough to go out and take that leap. I guess what I would ask, and start where you want, is whether it's top to bottom or bottom to top, what brought you here? What told you this is something you should and could go do, in terms of being an independent consultant? I'm sure there was some skillset acquisition along the way that brought you there.

Pete Hodgson: Yeah. So the long story – it's not that long a story – I've been in the industry, been a professional software engineer for 20 years at this point, I think, something like that. I started my career in startups in the Bay Area. I moved to the Bay Area.

Brian Dawson: Into a big city, you were telling me earlier.

Pete Hodgson: [laughter] Yeah, in the U.K. where I am from, when we say city, we mean a large city like a large population. I think it's actually changed now, but technically you can't be a city in the U.K. unless you have a cathedral. So it's like a major urban center. In the U.S., that is not the case. I did not know that when I was 19 and looking for an internship in California. I went – came to California to Silicon Valley pursuing my dreams of streets paved with gold and all that kind of stuff. Got an opportunity for an internship in the city of Sebastopol, and looked it up on MapQuest or something. I actually think I might have actually gone to a bookstore and found a physical map because I [crosstalk]

Brian Dawson: You really are dating – you are justifying your decrepitude.

Pete Hodgson: Yeah, and yes, so city of Sebastopol. So I get picked up. I get this job, get picked up by my new boss at San Francisco airport, drive over the Golden Gate bridge. It was amazing. It was a beautiful, sunny day. The cars were giant. The freeway was really big. [laughter] And we arrive in the city of Sebastopol, population 7,000 and I was not ready for that at all. I ended up loving it. The North Bay, for people who haven't been there, North Bay area is just an amazing part of the world, wonderful people, really nice to not be surrounded by technologists. I was working at a startup there, but it was definitely not – I was actually next door, literally next door to the O'Reilly headquarters, which was pretty cool.

Brian Dawson: Oh, wow.

Pete Hodgson: Yeah, they built their headquarters in this apple orchard in Sebastopol.

Brian Dawson: I never knew that. Okay, but when you're 19, and you're seeing the stars, going to San Francisco, and you end up in Sebastopol, there's some dissonance there.

Pete Hodgson: Yeah, but it was amazing. It was amazing. I love that place. So anyway, I worked in startups, bounced around the Bay area, was in San Francisco for a while, love that place too. And then I got this opportunity to work at this consulting company called ThoughtWorks. As a young technologist, I had – I think like many young technologists of that era, I had fallen in love with – fallen in love isn't maybe the right word, but I had just developed this amazing respect for these kind of –

Brian Dawson: Were enamored with, yeah, yeah, yeah.

Pete Hodgson: Like Martin Fowler, Dan North, all these incredibly smart technologists who worked at this consulting company called ThoughtWorks. I had the typical imposter syndrome. "There's no way I could possibly work there." Figured it out, got a job there. Worked at ThoughtWorks for six years, which was just an amazing experience. You asked how did I end up doing this type of consulting?

Brian Dawson: Yeah, yeah.

Pete Hodgson: Absolutely. I would give 90 percent of the credit to my –

Brian Dawson: To your time at ThoughtWorks.

Pete Hodgson: – my tour of duty at ThoughtWorks. Yeah, such a cool place.

Brian Dawson: And in my classic fashion to interrupt your journey, I figured that was a key pivot point.

Pete Hodgson: Oh, for sure.

Brian Dawson: How long did it take you to get over the imposter syndrome there?

Pete Hodgson: Well, I was younger, so I probably had less of it. I think probably like a year. I think probably about a year of – and it's that way with many people when you're in a job and you're in a comfortable rut, but I had gotten to this point with a startup I was working at where I was like – there was an acquisition and the thing I was working on was still kind of technically a live product, but it clearly wasn't getting too much love, and no one really – I had reached the peak of the learning part of that job. I realized it was time. I needed to do something and so I –

Brian Dawson: Yeah, well. We did talk about before – I won't pretend we didn't. I am for – not the best descriptor – a bit of a ThoughtWorks fanboy. To borrow your language, I very much respect the output. Really, as an organization too, and what they've done with their consultancy business, etcetera, but when we talk Martin Fowler, Sam Newman, Jez Humble, David Farley, Chad, and frankly yourself. It comes across as a magic place, so I did just want to call that out. It looks like that really was your bridge.

Now was there something that you saw that said, "I can go out and help people be better and I'm going to take that leap", or is there anything in between glorified typist or consultant at ThoughtWorks to software delivery consultant, to Brian calling you practitioner extraordinaire? What was the pivot?

Pete Hodgson: I mean I think working for ThoughtWorks was definitely an absolutely transformational experience for me, I think for a few different reasons. One was just in general working for a consulting company is a hyper-accelerated learning curve because you're just going – that transition, that thing that we do when we move jobs or move roles where you go through this firehose of learning, and then things peter off, and then at some point, you're not really learning more. It's super condensed because you're doing that every time you go to a new project, and you're maybe going to a new project every six months or something like that, depending – depends on the client. So just super condensed learning and also a lot of different scenarios. I worked at ginormous banks, and I worked at small San Francisco startups. I worked in a bunch of different industries, travel, finance, and so it's just super accelerated experiences plus the caliber of people there is insane.

It's just – it is – my perception from the outside before joining was there's all these smart people who write books and give conference talks. You get there, and you realize "Oh my gosh. That's the tip of the iceberg. There's all these other amazingly smart people and there's just super interesting conversations happening all the time". It was just such a cool experience and just learned a bunch of stuff and realized – the other thing is I realized that although the absolute beating heart of ThoughtWorks is technology, it's consulting, and it turns out there's this cheesy consulting phrase: "It's always a people problem." It is. At the end of the day, the solution, I think, is always people stuff. So that kind of going through that realization that if you really want to have impact, you've got to look at more than just the code, that was definitely something I learned at ThoughtWorks, was educated, beat into me at ThoughtWorks, I guess.

Brian Dawson: That's awesome. For the audience, we're on a podcast. They can't see you. Probably heard it in your voice, Pete, but your face – there's a moment where you're talking about how awesome the people were at ThoughtWorks, and your sincerity showed in your expression that people don't get to see. That actually sounds really, really empowering. I can perfectly imagine how being put in different situations, arguable potentially varied and high-pressure situations, short-term not only gives you a purview, but I asked about imposter syndrome. It gives you an amount of leveling and confidence in your ability to grasp things in where you sit.

Well, I'm glad you did. I'm glad you ended up here because it does bring me to a point where I get to ask about some of your work, and in particular, as I signaled at the top, we get to dig into feature flags a bit. So actually, kudos to you, which I'd call out, you wrote an article on feature flags in 2017, which happens to be still in one of the top search results in this space. In fact, I believe I arrived at it through a Google search sometime last year. In fact, I've read a few of your articles and probably driven some of my framing around it. What is your take? I'll direct a little bit. I am curious to understand what the life cycle of feature flags have been over the past couple of years, but first, let's just pull back a bit. What and why a feature flags, if you have a high level on that?

Pete Hodgson: Sure.

Brian Dawson: First, feature flags are just an if statement. There's just a pragma in code. They're something developers only really care about.

Pete Hodgson: That's right. We can move onto other topics now.

Brian Dawson: Good talking to you, Pete. No.

Pete Hodgson: Yeah, I would preface this by saying although that article does rank very highly on Google, I think a lot of that is thanks to Martin Fowler hosting – it's on the Martin Fowler website, and basically, a lot of the Google juice flows down into that article. So I've got to be humble and give credit to him, and also give credit to him being very gracious in terms of sharing that Google juice. So I wrote that article out of frustration because I was at the fourth client where they were using this word feature flagging, and they weren't using it the right way, or they were using – they were doing it wrong, and there wasn't a good definition on the internet. There were bits and pieces of old blog posts from Etsy in 2014 or 2012 or something, but there wasn't some really good content. Eventually, after being frustrated by this enough time, I think Martin Fowler said to me – I was complaining on some internal mailing list, and he said, "You should just write something", so I did.

When I wrote that article, feature flagging was a very well-established thing in certain parts of the technology world. I think it's become much more well understood now, but like I said, the Flickr, Etsy, that kind of – those organizations, those kinds of pioneering organizations had been doing something like feature flagging for a really long time. I'll take a step back and talk about what is feature flagging. It is fundamentally, it's the ability to choose at runtime between multiple code paths. At runtime, we can say, "Should we do A or B? Should we do X or Y?" This fundamental capability is then used for a bunch of different use cases. A really common use case would be what I call an ASCL release toggle where I'm working on some new feature but it's not ready for prime time yet, but I still want to – I don't want to have a long-lived branch. Maybe I want to push it into a shared environment so I can show it to a product manager or a product owner or something like that.

So I will essentially hide that feature, that half-finished feature behind a feature flag, so behind an if else statement. Then I can land that feature, land that half-finished code as latent code inside of our main branch, or our mainline branch, or trunk, or master, or whatever we're calling it, without – that latent code can deploy even all the way through to production. It's not – I can hide it from users. If I've got a fancy feature flagging system, I can turn it on for just for a product manager, or just in a certain environment. If I have a very fancy feature flagging system, I can turn it on for one percent of my users once the feature is complete. There's all these – underneath that foundational capability of choosing a code path at runtime, there's a bunch of use cases and capabilities that you can start to build on top of that foundational thing.

Brian Dawson: So do you – and I'll string these together – is there any particular you think force or forces in the shift in development approaches that have fueled the interesting growth or adoption of feature flags? And –

Pete Hodgson: Yeah, I think – sorry, go ahead.

Brian Dawson: No, go ahead. I'll – I'm a trademark compound question guy, but I'll stop, and I'll leave it with one.

Pete Hodgson: I think there's two things, and it comes down to what I see generally is there's forces that bring feature flagging into an organization. Either it's engineers wanting to do that kind of release management stuff, or it's product managers wanting to do things like AB testing. I think that both of those are becoming more necessary as broader trends in the industry roll out. So continuous delivery is really hard to do without feature flags. You can do it but it's – you probably don't want to do it. You almost inevitably, if you're focusing on continuous delivery, then you're going to need some kind of feature flagging capability. Why are people doing continuous delivery? Because the world is moving faster, and people's lunch is being eaten by other people who are doing continuous delivery.

So you look at the DORA report and accelerate on all the great stuff that Nicole and Jez and others have been writing about. There's a clear competitive advantage to continuous delivery. We could get into that in more detail. So, basically, you need feature flagging, or feature flagging makes it easier to do that stuff that gives you a competitive advantage. I think that’s one thing. Then on the AB testing side, product management practices, there's been a trend, I think, towards a more scientific, data-driven approach to product management, probably pushed a lot by trendy startups who talk about how much they do it. I guess it's not startups. It's like unicorn, mega, the FANGS. They're all doing all this data-driven experimentation. Again, if you want to do experimentation, and very aggressively follow that path, you need something like feature flags to run those experiments.

Brian Dawson: Okay, I thought it was just because people needed a way to be able to commit before their code was done when their managers were nagging them. It's interesting when you talk about, even before you get to CD, if you get to what are healthy CI practices, what I ran into a lot, and I'm sure you did probably more than I did, is what I'll call traditional as opposed to digital native organizations just having insane branching and merging schemes with release trains. Granted, while I still think you're going to find that it's difficult for people to commit to trunk-based development, there's somewhere in that continuum where like you say, you really can't gain the benefits of compression of reduction of your branches which can be a nightmare without having some level of flexibility. In leaning to the future, we talked a bit about this earlier. I'm curious to hear your thoughts. I kind of joke, and I get enough of a reaction that I'll continue to do it, that feature flags as a dev practice in an enterprise end up being the smuggler's tunnel to production.

Pete Hodgson: Yep.

Brian Dawson: They've lived as this thing that is very code-based, don't have a lot of visibility to the downstream stakeholders in the delivery process, and so in a sense, there's benefits I'll get to, or maybe you can touch on. I want to see what you think. There's benefits to QA, what I'll call being feature flag aware in pursuit of continuous delivery.

Pete Hodgson: I would say it's a necessity. I would say it's a necessity. If you want to use them in any serious way, yeah, you need to really think about testing as part of that.

Brian Dawson: Okay, and then we'll get downstream, and I'll hand it back to you. This is what we started to talk about earlier on the smuggler's tunnel to production. Think of traditional enterprise where you still have what I'll just call a big bang release. It may be advanced, but at the end of the day, you're still hanging over deployable assets, and somebody owns getting them out. And again, absent feature flags, somewhere I think the purpose of bringing them into testing, I feel like part of the maturation of feature flags, which coincides with truly driving continuous delivery and enterprise is with traditional structure, they also have to be surfaced to your release management team, and to your Ops team. I'm just going to seed it with that. I don't know what your thoughts are on that, and if that aligns at all with my question. What do you see the future of feature flags leading to?

Pete Hodgson: Yeah, I would agree with that, and I think that's definitely part of the journey that I see a lot of organizations go on as they adopt feature flags more broadly, as this realization that, at some point, we've got to stop doing the smuggler's tunnel and actually build a subway –

Brian Dawson: Bridge.

Pete Hodgson: How can I keep this metaphor going? Yeah, at some point, you've got to start putting warning lights on the trains that are going through that subway.

Brian Dawson: There you go.

Pete Hodgson: Because it is a path to production. This is also true, for example, with configuration management. There's – I think it's a really interesting metaphor that smuggler's tunnel. I see this in just more broadly in more organizations, this vacillation back and forth between this it's really hard to get through our formal processes, so we build all these back doors and tunnel. Google Tag manager is a really good example of that, and other products like it that just let us make small changes to production without anyone testing it or any security people looking at it. No big deal.

So there's this kind of waves of people figure out a way to bypass all those checks. Then something bad happens, or someone finds out, and then it all gets locked down. Then they find another route through. So yes, I think feature flagging is one of those things, and I think the right way to do it is to treat is as a – treat a feature flagging change the same as a code change, basically.

Brian Dawson: Yeah. And I'd almost say, for want of a better word, so first, I loved – I won't drill down too much, but this cycle and oscillation and I will just extend that to say, I think true software delivery maturation, and we talk about velocity, just like we talk about the movement towards DevSecOps, it's not avoiding checks. It's accepting and embracing the checks and figuring out how to accelerate and optimize through them in a way that's sensible.

Pete Hodgson: Yep, and I’m pretty sure that I stole this from Jez or someone smart like that, Jez Humble, but one of the key insights I think in the Continuous Delivery book is that if you make everything go through the same pipeline, every change go through the same delivery pipeline, then you can sync a ton of effort into optimizing and smoothing that pipeline, so there's this very common thing with organizations like "Gosh, getting changes to production is hard. Let's make a side door to get into production in case there's an emergency, but we'll probably use it [crosstalk]"

Brian Dawson: ECR, as I always say, the antipattern of ECR processes, but yes.

Pete Hodgson: Hot fixes or P1s or P0s or whatever you want to call it. So there's this reaction. There's this reasonable on the face of it, intuitive thing of like, "Oh, it's hard to get change into production. In an emergency, we need to do that. So let's make a break glass procedure." More often, what we should be doing, or organizations should be doing, is making it easier to get stuff into production in general. If you do that, then you a) it means in an emergency, you can make changes fast; but b) you can make changes fast in general. You can reduce that lead time, and like the Accelerate book talks about, that's just a competitive advantage to do that; c) it means that you've got the ability to respond to anything that happens, not just the stuff that happened in the past.

So you make all these break glass procedures for like "Here's how we can update the configuration" because there was that time when we made a configuration change and the system went down, like "Oops, we should figure out a way to stop that from happening" or "We should develop some procedure to stop blah blah blah blah." Instead of focusing on meantime to failure, you instead focus on how can we make recovery of the system in general faster, when we don't know what's going to fail? That leads you to just optimizing the pipeline and flowing all the changes through the pipeline.

Brian Dawson: Yeah, love it.

Pete Hodgson: A specific example of feature flagging is – and I encourage organizations to do this more is if you treat a large number of your feature flags as code and actually put them in your codebase, then you can make a feature flag change literally the same as a code change, and it flows through your environments the same. It gets QAed the same. It gets reviewed by security the same. You can roll it back the same. All that stuff happens for free if you use that pipeline that you've spent so much time building.

Brian Dawson: Well said. It sets me up for my other question. I firstly, want to say thank you for mentioning Jez and the Continuous Delivery book. We went back to the decrepitude. I'm starting to become more and more surprised and disappointed with people that are coming into the space and actually haven't read that book. That absolutely speaks back to a core principle in the book.

Pete Hodgson: Huh. That's interesting.

Brian Dawson: This gets to – I want to ask you about culture, but also going back to the human component of technology and software delivery. From a developer perspective, do you find that feature flagging, progressive delivery and feature flagging, and progressive delivery, tagged with this idea, let's optimize a single pipeline provides benefit to the development, the people that are driving the change, and through code, getting value out to users, customers, or the business? Is there any social, emotional, psychological benefit that you see when you help people implement these concepts that you've talked about?

Pete Hodgson: That's an interesting question. I think in general, there's a little bit of a – just the thing in general of deploying half-finished code to production, when you say it that way, that sounds dumb. [laughter] It's generally like, probably the first reaction most people have to that.

Brian Dawson: I tell you, test in production always gave me a visceral reaction. Similar concept.

Pete Hodgson: Yeah. I've been playing around recently with the idea of differentiating between testing in production versus testers in production. I think that's a distinction we could [crosstalk].

Brian Dawson: I like it.

Pete Hodgson: I think as people as adopting – the psychological journey, I think people go through is "That sounds terrible". Then they try it and they're like, "Oh, it's not as bad as I thought." I think what it ends up being is by enabling, by reducing cycle time and, more importantly, reducing batch size so we're making lots of small changes in production, at first, that feels inefficient to a lot of engineers. Like, "Why would I do this as free commits when I can just do one big commit?" or " Why should I do three different branches and merges and PRs and all that? All that's so much overhead. If I just go off into the back room here, and bang away at this thing for two weeks, then we can make it more efficient."

What I think people really learn pretty quickly when you shift to that mentality of small batch sizes and lots of little changes is each change is way less scary, and the search space for why is this thing broken is like orders of magnitude smaller when it's a ten-line code change versus a thousand-line code change. There's all these benefits to reducing batch size. I think that's what people get to is what teams – when I've been on really high-performing teams, it almost gets to the point of coding in production, dark lang style where you're making a change, flowing it out to production within – sometimes it can get crazy fast like within half an hour or something. And you just – it's just so much easier to do that. It's just so much less friction if you work that way.

It's one of these things that you have to experience, and like branching strategies and trunk-based development is definitely one of these things that I talk particularly with a lot of younger engineers, or engineers that are newer to the industry, which depresses me that this is the case, but they've never worked in – with a trunk-based system. They've always worked with pull requests.

Pete Hodgson: Yeah, so I go to an early-stage startup, and I say – and they're doing this crazy Git Flow-based branching system. You were talking about enterprises doing it. I see a lot of four-person teams with a Develop branch, and a Mainline branch, and a something else branch. PRs all over the place. And I say, "Have you ever thought about just writing code and just checking it in directly to that Mainline?" They literally look at you like, "I think I misheard what you said, because it sounded like you were saying 'Commit directly to Mainline', but you can't do that. That's not possible."

Brian Dawson: Wow.

Pete Hodgson: Yeah, it's really weird. I have to really walk engineers gently up to that idea. Once you've lived it done well, you realize, "Oh my gosh. This is actually a much nicer way of working, with pros and cons, but in general, a much nicer of working", but when you've never done it before, it doesn't sound – it's not intuitively a good idea.

Brian Dawson: What is really surprising to me, and please don't lose your thought. The reason why you hear me underneath go, "Wow", first I hadn't thought about how Git Flow, as amazing as the PR revolution is, what I missed is I'm thinking about people my age that are used to first starting – you start form ClearCase or SourceSafe or RCF, and you have a particular view on things. I also hadn't thought about how DCVS in a lot of ways, both in a distributed version control, it kind of both supported a level of trunk-based development, but also encouraged code forking.

Pete Hodgson: Yeah, it makes branches easy. It turns out, the problem was never branching. It's merging that's hard. It doesn't make merging easier. It makes merging a little bit easier, but it doesn't actually solve that problem. It just makes it easier to get into the situation where you need to do the merging.

Brian Dawson: If you're a little dense like me when you're adopting – going to fork, it actually makes merging a whole lot more dangerous, but that's once you get it. Hence, the power of GitHub. Hence the power of SAS-based Git, and the magic of the PR. I'm not knocking it.

Pete Hodgson: I would knock it. I'd be very happy to knock the PR, actually.

Brian Dawson: [laughter] Okay, okay. I've celebrated it before, but yeah, interesting insight. I'm sorry. I'd interrupt. We had talked about the change, and then I guess, just to set you up, thank you for that insight. That does get to assuming there's a level of mindset change, and cultural shift that organizations have to take to adopt feature flags at scale. I ask you the question, what should teams and organizations be doing to orientate around feature flags or really feature-first development which tries to progressive delivery, hypothesis-driven development? But at your starting point, considering these blockers you noted, what recommendations do you make for organizations to move in that direction?

Pete Hodgson: I mean general advice that I think applies here is to start small. Generally, I would advise a team that's not really doing feature flagging that much at all, to just start with literally an if else statement. Start with release toggles, because they're the easiest to manage, and the easiest for engineers to see the value in and reason about. For one of the features that you're working on, put it behind a toggle, and don't create a long-lived branch. Make sure that the first one you do is an easy one. Then make the second one a useful one. And take it from there. I encourage teams to not start with some fancy all-encompassing system. I would say don't try and build something that supports both the product manager use case like experimentation and the engineer use case like release toggles or things like that.

And I'd say, start small, start simple, but as soon as you start seeing adoption, do not continue to roll your own feature flagging framework. Stop. Don't do that. You will accidentally do it, and in a year's time, you'll be like, "Oh wow, we have two engineers that are working full-time on this thing. What competitive advantage do we get from that again?" Like, hm. It's half as useful as a thing we could buy or use open source, but we're sinking a lot of energy into it. How did we get here? One of my pet peeves is feature flagging is such a fun – a feature flagging system is such a fun thing to implement. And it's definitely an iceberg in terms of amount of functionality you actually need lurking under the surface. But it's – I worked at a startup where the feature flagging system we used was literally built by an intern, and it was – we had the silly _____ fallacy of "We can't throw that away. That was at least two weeks of that intern's summer project." But yeah, it's just one of these things where for very few companies, is it a differentiating competitive advantage how you do your feature flagging. You should just find an open-source tool or use one of the SAS products to do that.

Brian Dawson: Yeah, that's an awesome – two points I just want to dig in on there is one, what I've started calling feature flag management – or what people have, not me. I didn't – is that at some point when you get past it as an early-stage dev practice, I like your cycling and iteration. Realize when you're going to have to incorporate feature flag management. That is almost where that product owner, product manager use case and dev use case start to converge. There is something you didn't say that I'm curious about that maybe sits somewhere in there _____. When you don't choose to create, buy, leverage, refactor a feature flag management system, and you start to develop flag debt, is that something that you tend to see?

Pete Hodgson: Yeah.

Brian Dawson: Yeah, okay.

Pete Hodgson: Absolutely, the number one thing that every organization that adopts feature flags gets is a flag debt around retiring flags. It's definitely – it's the absolutely most common thing I hear in terms of the drawbacks of feature flagging. And so there's a bunch of things we can get into how you deal with that, but it's the kind of thing where having an open source tool or a something that makes it easier for you to keep those flags in check, and identify which ones should be retired and removed and all that kind of stuff. There's some pretty common practices, not pretty common. There's some pretty well, some pretty clear, good practices for flag management that just aren't talked about that much unfortunately yet, that you can do to keep them in check. Even if you're really focused on doing it, it's a drawback for sure. It is an ongoing cost of feature flags. I would say the benefits generally outweigh those costs but dealing with that cruft and those old flags is a continual drag for sure.

Brian Dawson: Could be a pain. There could be a feature flag. We think about our Phoenix project, Brent, to make a reference. "Hey, I implemented all these flags and then I got poached by a FANG, and I'm out." What do we do with those?

Pete Hodgson: "What does this thing do?" There's this classic scare story that the nightmare story. There's this company called Night Capital, who lost like a quadrillion of a gazillion dollars in like ten milliseconds. I'm exaggerating numbers obviously. Just almost went out of business by misusing an old feature flag and recycling it because apparently, bits are extremely rare and you can't reuse them, or you want to reuse them.

Brian Dawson: That's fine.

Pete Hodgson: I should say, it's very easy in hindsight to say that kind of stuff, and to make fun of the decisions. I'm sure that the people that made those decisions were doing their best work, and it all made sense at the time. I've – don't like _____ ragging on other people.

Brian Dawson: No, right. It's easy to look back in hindsight, but I think it speaks to what we're talking about. Because it's funny, the Night Capital story came up on a podcast recently. But it wasn't associated down to feature flags and I _____ bring it up in this podcast, it's important, especially if you take an architectural view on it, and you have features that are distributed across the codebase. Different modules, controlled from a simple location. If they're not well-managed, you can go, "Hey, look at this. Let me flip this thing. I'll turn it off." But you don't have an idea of what the distributed, for want of a better word, impact of that toggle is. I think that that's a great point.

Now, I'm going to pull you into another work of yours that I'd just like you to share with me, and listen to me, or help me understand. You recently wrote a book for that place that was in the apple orchard for O'Reilly on continuous delivery in the wild. I'll just ask, we won't spend a ton of time on it, but I'd like to understand what it is. I am going to plan to go read it but give me the Cliff Notes.

Pete Hodgson: Sure, so the inspiration for this was you see a lot of trendy startups or well-known companies doing conference talks about stuff, and it's generally, they talk about the stuff they are doing, or they did, in a very positive light. You don't often see conference talks where it's like – I wish this was a thing. I wish this was an entire conference actually. I think someone tweeted about this a while ago. A conference of two years later, like the thing that we were talking about in 2020, Velocity2018, here's the terrible things that happened. Like in general, it was good, but here's the terrible things. So generally, conference talks are very – put a positive spin on things and tend to be focused on these kinds of unicorn-y companies.

I wanted to find out – I wanted to get real talk, not conference talk. So I wanted to go and find some actual regular companies, regular, non-unicorn startups, mid-market companies, more enterprise-y companies and find out how they were doing continuous delivery in practice. So companies who are finding success and value in continuous delivery but aren't going and talking at conferences about it in general. So split to our feature flagging company. We're generous enough to sponsor me researching this, so I went and did these interviews with a bunch of different companies, and tried to write up what I found, basically, around that kind of continuous delivery practices, looking for things that were similar across those companies, things that were different in interesting ways, and wrote it up.

Brian Dawson: I'm sold, and I will give it a read, but to help me cheat a bit, what are one or two of your most interesting or favorite use cases that you came across there?

Pete Hodgson: I'll tell you. Something that I thought was really interesting was the commonalities. The things are the same and the things are differences. So a very common threat across all these companies was a focus on reducing batch size. So all these companies that were finding success with continuous delivery had independently come to this same understanding that if we can shrink down the size of those changes that are going into production, that's a good thing. They were orienting their implementation of continuous delivery and how they were doing it based around this kind of principle of how do we reduce batch size? That was this interesting commonality.

I think what was also an interesting difference is how the architecture of the system made the approaches different. So monolithic systems versus fine-grained SOA, microservices. You can aim for that same value, that same underlying value of reducing batch size, but the way that you do it is actually quite different, which is not that surprising, I guess, if you state it that way, but it was interesting to me to realize that the practices that make sense for microservices don't make sense for monolith and vice versa. That doesn't mean that they're good or bad practices. They're just good in certain contexts or bad in certain contexts.

Brian Dawson: That's – so first, to some people's annoyance, I kind of live on this principle that context matters. I say, personally, to some people's annoyance because some people are like, "I don't want the context, dude." But I'm going to apply that here to what I think I'm hearing that we often – it's great to hear the happy path. It's great to hear the inspirational story, but part of this is context matters.

Pete Hodgson: Yeah, absolutely. This is my fundamental frustration with how much people listen to what Google, and Facebook, and Netflix are doing. Unless you work at that scale, the stuff they're doing won't be valuable in the same way. The principles behind what they're doing is where the value is. So if you can go to a conference talk and listen to someone from Google talking about – listening to someone from Facebook talking about how they do continuous delivery, aping that process at your 20-person startup, or your five-billion enterprise organization isn't going to work. It's not going to be valuable in the same way. Understanding the principles behind the approach that Chuck Rossi set up there is – that's where the value is. I think that's the art of really thoughtful think leaders is not talking about what we did, but talking about what were the principles behind it? Why was that valuable? That's where the nuggets are.

Brian Dawson: That's powerful. All right, I can't keep you captive like I do with all of my – someday someone's going to call the police because I bring guests on and then I just hold them. I want to get to where I'm really going to challenge you to open up, and it's actually aligning from principles. Let's be real about continuous delivery in the wild. Pete in the wild. What is your DevOops, not DevOps, Dev-O-O-P-S moment? A software development challenge, or otherwise? A time in your career where if we get the juicy stuff, you screwed up, but that screwup gave you an opportunity to learn. If you can help our audience and myself learn from that, without having to go through the embarrassing moment. Is there something that you can think of to share with us?

Pete Hodgson: Yeah, so an example –

Brian Dawson: I'm sure it's only happened once, probably.

Pete Hodgson: Yeah, there's only one. I mean it's not hard to choose, because I've only ever screwed up once in my life. It's kind of the opposite problem. I have a paradox of choice here. One that comes to mind is, so I was – when I was consulting with a startup and we were tasked with building a new product for them, like a standalone product that was semi-integrated into their main product. Part of the reason we were there, was – it was a _____ team, and we were there partly to help them quickly build this product in a high-quality way, but also to demonstrate some of those Agile engineering practices that this startup wanted to percolate into the engineering org.

That's kind of common pattern for quite a few ThoughtWorks engagements. Part of what we needed to do was deploy this thing into production, and we – and the existing setup at that startup for getting stuff into production was very not DevOps-y. It was very over the wall, and there was this ops team that had – they had Puppet scripts, but these Puppet scripts were totally out of date, and were this mix of manually applied things. It was a bit of a hot mess. And we told these infrastructure teams, like "Here's how we're going to do it." And they were like, "It seems really tough. We're – this seems hard, and we don't have time to do it." We're like, "No, no. We know what we're doing. We're pretty good at this stuff." There was definitely some arrogance on my part. We pushed and pushed and pushed for them to change how they managed their production environment to be super DevOps, and we were going to ship DEBI in packages through a pipeline because DACA wasn't a thing yet.

We were going to – it was all going to be 100 percent infrastructure scope, blah blah blah. We couldn't persuade them to do it in prod, and so the approach that I came up with, and I'm pretty sure this was my fault was, "Let's build our own staging environment, and we'll show them how it's done." And then when we're ready to go to Prod, we'll just say, "Look, check out how nice this is."

Brian Dawson: It works. Oh, I think I see where this is going. I think I see where this is going.

Pete Hodgson: And we kept on having these conversations with the infrastructure folks. The conversations became more and more fractious because we were whacking them over the head. We were also basically doing their job for them. We were like, implicitly saying, "You're not very good at your job, so we're just going to replace you with a shell script" as the old t-shirt used to say. So unsurprisingly, we lost almost all the trust and credibility that we had initially had with these nice people, these nice human beings that were trying to do their job to the best of their ability, and it got to the point that we were ready to go live with this thing that had been – we'd been – we'd set up this awesome, delivery pipeline that flowed our changes directly into this shiny staging environment that we'd built every – within five minutes of deploying, committing to Mainline or whatever.

Then, we're like, "Okay, cool. Now we need to change to Prod." Everything came to this screeching halt, and we had to spend like I want to say a month or two months figuring out how to get this into the actual production environment. We'd done the exact thing I had probably wisely told to other clients. "It's not done until it's in production." I'd somehow managed to convince myself that that didn't apply to building infrastructure environments [crosstalk]

Brian Dawson: I'm smiling over here. I'm cracking up over here.

Pete Hodgson: Because we burnt all this credibility, the folks that actually owned the actual environment, who were going to be running the thing, who were going to be wearing the pagers, they were – yeah, they should have been our partners because we had been just dumb, and not nice, and not good human beings. I'm using the royal we here. It was mainly me. I had got frustrated and irritated at how hard it was to make change, and so they'd locked up and then change got harder, not easier. My lesson from that is there's a surface level lesson of just focus on getting something – start by getting anything into prod. I wrote a blog post about this years later called, "Hello Production." Talking about this idea of your Hello World should go all the way through to prod.

So that's the surface level advice, but the value underneath that, the what's the general lesson we can learn? Treat human beings – be aware of people, and understand at the end of the day, you've got to look at the human beings in the picture and figure out how to get the system to work. The system involves the – understanding how production works was not – was partly about understanding the Puppet scripts and what was deployed where, and why was there Salt and Ancible and Puppet all running together? Then there's also "Well, why does this person here feel like they have to do this? Why do they not talk?" The sociotechnical system was the thing that I didn't have the patience to understand, and it totally ruined lots of people's day for a month or so.

Brian Dawson: That is awesome. I was going to say like parents don't have a favorite kid. A lot of the DevOops are awesome. I can't rightly call this my favorite, but I thank you for that. Not only the dimension it covers, not only how vulnerable it was, yeah, there's just a lot there in it. So I do really appreciate you sharing that. There are a lot of lessons I think I'll reflect back. No A-hole rules, which is kind of important. Building credibility is an incremental process. Think about Agile partnership development, especially coming into an org. You don't become partners in a waterfall manner. It's not a big bang thing. Yeah, love it. So to get a couple of more things out of you, gems of wisdom, Pete. What is a book, podcast, other resource, person to follow, Twitter thread? I don't know what the cool kids are doing these days. TikTok –

Pete Hodgson: Clubhouse channel or something, or a room or something. I don't know.

Brian Dawson: Exactly. I'm so bad with Clubhouse. The only time I listen to it, and I am going to sit down. I realize – someone said, "It's a conference on your phone. It's really live podcasts." But I'm now at the age where I just accidentally open it all the time when I'm on my phone. That's when I –

Pete Hodgson: [laughter] "How do I turn this thing off?"

Brian Dawson: Right, "Damn thing." But is there a resource that you recommend to our audience?

Pete Hodgson: I think that I am just absolutely in love with at the moment is this concept called Wardley mapping. Have you heard of Wardley mapping?

Brian Dawson: Recently. I'm not dug in. Explain.

Pete Hodgson: I can't explain it. I've been trying to figure out how to explain it, because I've basically had a variant of this conversation about like ten times in the last month where I've been like, "Dude, there's this thing called Wardley mapping. It's amazing. I can't tell you how cool it is because I can't figure out how to explain it." It's basically this way of thinking about the moving parts of your value chain. So if you think about the product that you're building is built on top of this pyramid of other things underneath. So the Twitter app is built on top of a bunch of frameworks and languages and databases and stuff like that. You map that out, and then you place all of those things in a – on a spectrum from innovative to commodity. So for example, if I'm Twitter, maybe I'm using some super fancy custom written – actually, this is a really good example. So sorry, this is going to go longer than maybe you had wanted it to.

Brian Dawson: That's fine. We already – we'll probably keep it. We always have post-Pete. I want to hear it.

Pete Hodgson: So Twitter maybe have a custom database that they use for storing. I would imagine they're not storing their Tweets in Oracle or whatever. So that's innovation, but they're also building – that custom database is running in a data center. Maybe they built their data center, so that's like a product, but it's not super innovative. It's like pretty standard data center. There's not one other kind of approach to building data centers. Those data centers have commodity infrastructure. They're just building them on boring old racks of servers.

So you lay out your – this kind of ecosystem, the value chain, and then you look at where these things are in terms of innovation levels, and you ask yourself, "Should they be there?" "Is this thing that we've custom built, is it actually a competitive advantage for us? Is there a commodity thing that we could be using instead?" That's just the very surface level of the concept. There's also all this strategy and actual grown-up business strategy you can apply here, and you can say, "I know that in general, there's a trend for innovation to move to commodity." So data centers is an example where they used to be – back when I was first in the industry, if you were starting a startup, you would literally put computers into data centers. No one does that anymore until they're like at huge scale.

So the thing that was like custom has become a commodity that you just buy from someone else. There's all these really interesting things when you start thinking about the thing that we're doing now that's innovative, what happens to that map and the shape of that map as it moves to more of a commodity? Where's the competitive space that we can – where's the risk of the thing that we're doing now suddenly becoming something someone else can do? Or where's the opportunity for us to hop onto that commodity and build innovation on top of it? This – Simon Wardley is the person who came up with this. He has a short book, reasonably short book that's free on – free online. I read the first three chapters of it and I was metaphorically highlighting every other thing. It's just full of super, super interesting insights. The actual mapping, that kind of model of laying things out that way, is a really surprisingly general-purpose way of communicating with other people, the shape of an ecosystem of things. It's so cool. It's super cool. I'm super excited about it.

Brian Dawson: I'm excited. We could start a whole 'nother segment based on my questions about it, but we won't. I'm going to drill in on one thing you said though about the general applicability and ways of communicating things in our talk about it's always a people problem. The discussion about your DevOops moment. Communication is key. I think a lot of times, not only in our particular space, but in the human space, failure to properly communicate or efficiently communicate is key. Want to underline that. Before I pin you down for some other work, I do want to get a chance to – any closing thoughts? Any final words that you want to leave our listeners with?

Pete Hodgson: No, I think I would – I'll give myself a plug because I love having conversations with people. I think if there's anything that we talked about where people want to tell me how wrong I was or add their own opinion, the best way to do that is hit me up on Twitter @ph1. I just love talking about all this kind of stuff. I could talk about it all day and all night. If anyone wants to talk about it, they should hit me up, and we can talk about it.

Brian Dawson: Great. I appreciate that openness. Likewise. If there's anything you heard here that Pete was wrong about, reach out to Pete on Twitter. If – I don't want to hear what I was wrong about. If there's anything that you love that I say, reach out to me on Twitter.

Pete Hodgson: [laughter]

Brian Dawson: The fragility of my ego – no. I'm joking. Hey, Pete, I also love talking about the stuff. I loved the conversation. One of the things I want to pin you to while you're live here is as the practitioner chair of DevOps World, which is conducted by the sponsor of this podcast, and the sponsor of my paycheck, CloudBees, will you commit to ensuring that you get a talk submitted or that I can get you in to speak at DevOps World?

Pete Hodgson: Wow. That is a seriously judo maneuver. Yes, I would love to. Do we need to commit this onto the blockchain somewhere so it's like a _______, a smart contact?

Brian Dawson: I'm generating an NFT on it now, a whole smart contract.

Pete Hodgson: Yes, right. [laughter]

Brian Dawson: Now, I'll reach back out. We'll make sure it's a live code for you, but really, I think your background, your experience, some of the stuff we covered here, but some of the stuff we didn't cover. I want people to hear from you without me involved, interrupting the conversation. So I'd love to get you in on DevOps World. If I don't see a submission come in, expect that I'm going to hunt you down on Twitter.

Pete Hodgson: Yeah, do it. No, I've actually fallen out of the habit of doing as much conference talking as I used to. I never did a huge amount, but I actually really enjoy doing it. I think many people would say the same thing. The best way to learn is to teach. Trying to figure out how to crystallize something down into a good talk really makes you – like I was saying earlier, it helps you to figure out what's the general principle behind the thing I'm talking? Because just telling a story is fun, and people like listening to it, but the actual extracting generally applicable ideas is – that's – I have aha moments myself when I'm figuring out how to write stuff down. I'll put it into a conference talk.

Brian Dawson: Right now, based on the talk we have, I already have five abstracts ready. I've got to be careful. I'm not volunteering to write them for you, but I can imagine about – continuous delivery in the wild, for example. Really doing unconference is a different thing, but really, it's the unconference. It's the unhappy path. Let's have a conversation about that.

Pete Hodgson: Although, the funny things is all these companies were very happy with their – they weren't – none of them would say that they were innovators, although some of them actually were and didn't realize, which is interesting, but they were all just happily kicking butt. I love – I find a lot of teams like that where they're just happily kicking butt. They're not tweeting about it. They're not conference talking about it. They're just quietly kicking butt, which I think is – for many people is a very nice way of moving through the world.

Brian Dawson: Awesome. [outro music] Well, Pete, really enjoyed the conversation with you. I'm sure our listeners will as well. Then, I look forward to seeing you again at DevOps World. Thank you for your time.

Pete Hodgson: Thank you so much for having me. It was super fun.

Brian Dawson: Thank you, all. I hope you enjoyed the episode. Please make sure to download our next episode next week where you'll get time with more exciting and informative guests. Again, thank you. This is Brian Dawson with DevOps Radio.

[End of Audio]