The Release Train Has Crashed Into the Station

Painting with a brush that's admittedly pretty broad, we could divide software development process at a larger scale into two buckets:

• Centrally planned and highly interwoven.

• Loosely coupled and relatively autonomous.

This dichotomy has existed since a time that well predates my own career, and, I would argue, stretches back decades.  But I don't believe that it's going to continue to exist for much longer.  Modern software practices are conspiring to render the centrally planned model obsolete.

The latest incarnation of heavy central planning goes by a variety of names, which are generally variants on the idea of "agile, but for the enterprise." But there is, perhaps, no term that encapsulates this better than the "release train."  If you've ever worked in an enterprise that has done an agile transformation, you're probably deeply familiar with this term, having heard it from the lips of wave after wave of process consultants.  But, if not, here's a quick definition:

An approach to aligning the vision, planning, and interdependencies of many teams by providing cross-team synchronization based on a common cadence.  A release train focuses on fast, flexible flow at the level of a larger product.

If it sounds, well, "centrally planned and highly interwoven," that's because it is.  The release train targets the enterprise-level program, which usually employs a few hundred people.  It offers a way to herd all of those cats into operating as one gigantic, machine-like system.  It's a system not unlike the various trains offering public transit in a big city. But as companies aspire to ship software like Facebook, it means that they need to reduce the time to market from months to days to hours to minutes.  And no matter how efficiently your trains operate, there is going to be a hard limit on how closely you can deploy them at high speeds.  In other words, process methodology marketing efforts notwithstanding, the central planning model---the release train concept---is dying.

A few years ago, I was consulting with an enterprise when I happened to stand in for a pretty surreal program meeting.  It was the quarterly release train planning meeting.  (I believe the term "release train" is tightly associated with SAFe, the Scaled Agile Framework, but this company had created a hybrid model of that already-complex approach.) I was actually attending as an observer with no horse in the race.  My consultative charter at the time was working with teams on certain technical practices.  The process side didn't have much bearing on my work, but I attended because the entire program, with hundreds of people, attended. At this massive meeting, something like 10 teams all mapped out, in exhaustive detail, what they would accomplish each week for the next quarter.  The idea was that they were so interdependent, they needed the three-dimensional equivalent of a Gantt chart to keep track of all of the prerequisite tasks and whatnot.  So each team "committed" to tasks with incredible granularity and every one of them said they did so with 100% certainty for all tasks.

At this point, a grizzled veteran of the process consulting world stood up.  He was a short-timer, having given two years of his life to this agile transformation, and he was shipping out in a week or two.  With an audience of a few hundred people, he had a Jerry McGuire, "nothing to lose" kind of meltdown that was, I imagine, the culmination of two years of cries for pragmatism falling on deaf ears. So he said, "Really?  Really?! You think that 10 teams and all of these people can all commit, with 100% certainty, to every single thing they're going to accomplish over the next 13 weeks?  Doesn't that strike you as...insane?!" He finished and utter silence followed in his wake.  About 30 seconds later, without acknowledging this outburst at all, construction of the three-dimensional Gantt chart resumed.

"Scaling agile" is only the latest incarnation of attempts to centrally plan all details of software development.  This was probably the default approach many decades ago.  It works for assembly lines and building construction, so why not software?  Let's call these planners "architects" and proceed as if we were building a skyscraper. This gave rise, in ironic fashion, to the so-called waterfall model.  A man named Winston Royce drew a picture that looked like a waterfall.  He then said, more or less, "this is how to fail at software development."  The picture lasted to this day.  His warning did not. Not surprisingly, at least to Royce, this didn't really work.  So along came other methodologies and attempts at mitigation.  Maybe people just needed to plan... harder.  Or maybe they needed a different planning methodology. Eventually, the manifesto for agile software development came along and upended the industry.  From then on, central planning models became these interesting hybrids.  Enterprise process consultants would come along with advice like,  "Oh, don't worry, enterprise organization---we'll help you get agile.  We'll just do it in a safe, comfortable way that feels pretty familiar and still involves all of the planning and overhead you know and love." Early in my career, the Rational Unified Process scratched this itch.  Today, it's the release train and all of the other attendant methodologies that preserve the underlying, fundamental paradigm: massive central planning.

There's an old saw that makes its rounds a lot in the consulting world called Conway's Law.  It basically says that organizations will build systems that resemble their org charts.  This isn't just idle, anecdotal wisdom, either.  MIT and Harvard Business School researchers have produced supporting evidence. So taking Conway's Law as something that's at least decently likely to hold up, what do you think the software produced by teams of teams in a "release train" is going to look like?  What do you think that program with the complex, three-dimensional Gantt chart web built?  Those intensely cross-coupled teams, marching to a relatively bureaucratic, highly-coordinated development process are going to produce intensely cross-coupled software, long on defensive coding and highly ceremonial interaction patterns.  They're going to build monolithic messes that barely wheeze across the finish line each quarter, if they even make it. If you want to cut the shipping time down from months to days, hours, or minutes, how do you think that's going to go?  Can a "train" with hundreds of people aboard chug from "value stream planning" to "value delivery" in minutes?  If you're saying yes, then you probably know of some secret locomotive technology that lets trains travel at about Mach 5.

I've consulted with a lot of enterprises over the years.  And, whenever I'm there, they're inevitably looking for help to improve how they build software.  Usually, they want to "go agile," but they're really after a path back to the sanity of small-scale software development.  A handful of people building a web app is pretty easy to manage and reason about.  But hundreds or thousands of people building a nine-figure, entire enterprise internal system overhaul?  Not so much. So it's very popular for these companies and the leaders in them to look to the startup world for wisdom.  It's so popular that the acclaimed book, the Lean Startup, even includes a chapter specifically about how enterprises can capture a little startup magic.  But, while that entire book is well worth the read, I'll distill it down to one idea for you to take away. Stop building centrally planned software, and figure out how to break your teams and your program into decoupled, highly autonomous, relatively small teams. Once you do this, the rest of the dominoes start to fall.  Small teams can act autonomously.  They can experiment in production and have a product focus.  And they can ship to production as frequently as their tooling and maturity allow, without worrying about other teams' schedules. If we're talking about vehicles, small, nimble, and highly autonomous teams are the equivalent of people with personal rocket-powered jetpacks.  They have incredible power to go fast, though controlling that power can be a challenged.  But if you try to implement the same thing with an entire train filled with people, you're just going to crash the train right into the next station.