The Future of Agile Software Development: Releases to Features

I'm going to talk in this post about what I see as the future of the agile movement.  I realize that this is somewhat of a loaded claim since agile now means all sorts of things to all sorts of people.  Depending on who you ask, it's any one of the following:

• A methodology

• Project management techniques

• A philosophy

• A massive consulting industry

Which of these is it?  The answer is "yes." But regardless of the current meandering state of affairs, lessons from the history of the agile movement provide lessons for the future of software development.  And as you look for where to invest next, both in terms of skills and money, those lessons for the future will help you be competitive.

The Agile Origin Story

Anyone with some years in the industry knows, at least in the vaguest terms, where agile came from.  In the early 2000s, a group of successful software consultants got together.  All of them were exploring so-called "lightweight" methods for developing software.  And so they thought it might make sense to get together to hash out some common themes. The rest, as they say, is history. They created a manifesto and published it.  Some of the methodologies in question would kind of peter out.  But others became iconic things we hear about today, such as Scrum and eXtreme Programming (XP).

The Rise of Big Agile

The origin of agile had a David-and-Goliath kind of feel.  On the one hand, you had the Goliath of major software development firms, practicing so-called "waterfall" methodologies.  (Or perhaps similar concepts like the Rational Unified Process.)  In this world, you planned software for months and then spent years building it, pretending that requirements would never change once you "gathered" them and then froze them for eternity. That didn't really work. So along came David with his agile sensibilities.  He won an improbable battle and got the world to see the light.  People realized that a better way existed. But that was 15 years ago. Now, our metaphorical agile David has made like the great baseball sluggers of the late 1990s.  He's lifted a lot of weights, taken some questionable supplements, and seen his head grow by four hat sizes somehow.  David has become Goliath. Agile is no longer an upstart.  It's the de facto right answer for how to write software.  And it's all the things I listed at the beginning of the post, including a billion-dollar process industry.

Agile Helps Enterprises Be Less Bad at Software

I remember when I first heard of agile methods.  The year was 2005, and I was earning my controller's degree in computer science via night school.  I signed up for a class called Software Engineering I (and then, later, Software Engineering II).  In this class, we studied these upstart methodologies, including XP. Agile was a breath of fresh air to me during the 2000s.  And that was because I worked for medium-to-large sized companies who didn't know how to develop software in shorter increments than six months or two years.  In this world, project managers in khakis and ties waved Gantt charts and reigned supreme.  Software construction proceeded a lot like building construction:

• Spend four months planning a year-long project and doing a lot of loafing around.

• Start building and get behind schedule immediately.

• Assume you'd make up time later and get really angry at anyone who changed requirements.

• Hit the year mark with no end in sight and start working 50 hour weeks.

• Wheeze past the finish line by marking 800 outstanding priority-one bugs as priority four.

• Mail CDs out and start furiously working on patches.

Agile came along and said, "Hey. That's kinda, ya know, dumb.  How about since that's never in history worked out once, we try something different."  And with this approach, agile helped a lot of lumbering companies get better at software.

The Quantum-Relativity Agile Problem

So the history of agile is that it helps large, lumbering enterprises get better at shipping software.  It does this by showing them how to plan reasonably and in bite-sized chunks.  On the technical practices side, it shows them how to stop building gigantic layered monoliths. To sum it up, agile methodologies help enterprises bring their software down to a scale that approaches reasonable. But what about non-enterprises?  What about relative newcomer models to the scene: startups, bootstrapped SaaS, boutique agencies and the like?  These newcomers have never lived in a world where old-school executives pretend building a CRUD app is like constructing a skyscraper, and they've never lumbered anywhere.  From day one, they've frantically shipped and tweaked software constantly. What does agile say about this?  Well, the agile-as-philosophy proponents might find creative ways to put it under the big umbrella.  But the truth is that it doesn't say much. In physics, there are actually two disjoint flavors of physics: relativistic and quantum.  Relativistic describes how planet orbits and gravity and "the big stuff" works, while quantum describes how electrons and atoms work.  But physics lacks a unified theory -- each set of internally consistent rules breaks down when applied to the other branch. This is what has happened to agile -- it explains only the large.

A Unified Agile Theory

To put a finer point on it, agile centers around two concerns.  First, you have collaborative process and ceremonies.  Set a delivery cadence that forces you to get things into the hands of "the business early and often," have certain meetings, estimate and prioritize work in certain ways.  Secondly, you have technical practices designed to minimize total cost of ownership and keep the change curve flat.  Pair program, write unit tests, continuously integrate, keep the code shippable. At first blush, all of this sounds mostly compatible with the frenetic world of bootstrappers and funded startups.  Professional agilistas would assert this emphatically.  But is it?  If you don't have "retrospectives" every week because you don't have time or you create a technically disgusting prototype made out of PHP, Flash, and baling wire, are you "doing agile?"  Not really. The agile world has never really reconciled the very large and the very small in software. If it had, DevOps wouldn't be called "DevOps."  It would be called agile.  Again, agilistas would argue that DevOps is 100% totally agile-compliant, but it is its own new movement.  And it says that you should break down the specialization silos among building, deploying, and monitoring software in production. DevOps is the start of a unified software field theory.

The Future of Agile

So what does the future of agile look like?  (And the future of DevOps?) Let me answer that question with a question.  Does Facebook do agile?  The answer, both on that Quora question and in general, always seems to be "kinda, sorta, in some ways, depending on the particular Facebook team."  This also applies to Google and other similar companies. These companies bridge the gap between scrappy startup and enterprise because they've logged time as both.  But these companies rose to their large state without lumbering, and they did so on the backs of insanely sophisticated software delivery operations. They don't ship CDs, of course, but they don't even ship releases.  In a lot of cases, they don't even ship features, opting to ship experiments or micro-features.  They use this incredible control and granularity to remain competitive and provide top-flight user experiences. The history of agile involved shrinking software shipment cycles from years to weeks.  It involved raising the bar of software quality to allow consistent delivery of new releases and features.  But that's agile only in the large.  The future of agile involves a grand unified theory.  And to get to that, we can't really shrink time between deliveries much more.  We must instead shrink the size, scope, and risk of the functionality that we deliver in each increment.