FOSS and Agile: Two Different Reactions to Dysfunctional Traditional Software Delivery

In my year and a half as CPO of CloudBees, guiding its evolution from Enterprise Jenkins to Software Delivery Management and establishing a product-centric culture, I’ve had a lot of opportunity to observe the nuances of different contemporary and emergent approaches to software delivery. I’ve come to a thesis that the Free and Open Source Software (FOSS) movement, and the closely linked agile and DevOps movements, represent two fundamentally different reactions to the deep dysfunction of how software was traditionally delivered in corporate environments.

The road here in the last 18 months or so hasn’t always been easy, and I’ve come to attribute that in large part to the conflicting assumptions and worldviews of adherents of these different cultures within our own organization, and then even more fine-grained subcultures within the two major cultures. I’d like to explore this a little here. I am also very curious to hear from others on their views of this and related topics.

Before I begin, I want to say that while my own background and inclinations involve more of the product and ops parts of agile and DevOps vs FOSS, I strongly believe both are powerful, critical movements that have together made the emerging world of software changing every aspect of life possible. There are problems best solved by one and others best solved by the other. I am proud to have counted as good friends and colleagues some of the pioneers of FOSS for decades and greatly respect what they have done for the software industry and the world at large.

My basic theory goes something like this...

Back in the bad old world of 1970s-1990s corporate software development - whether in house IT, commercial enterprise software or consumer software - the lives of developers and others on software teams were terrible and the outcomes for customers and users weren’t so hot either. Development organizations were forced to commit to scope, schedule and quality years in advance. When the software was finally delivered, it always missed on at least one of these dimensions - sometimes all three - and anyway, by the time it was finally delivered, the users’ needs had changed. Oh, and don’t get me started on how lovely it was to be ops receiving software thrown over the wall by dev with inadequate runbooks, logging, monitoring and testing.

The corporate view of proprietary IP, plus the additional political conflicts between different groups in the same company, meant that developers wasted time re-implementing the same functionality that others within and outside their organizations had already implemented. Budding student and hobbyist developers had no access to the basic software tools (most notably a Unix OS) they needed to work outside of a corporate context. There were many more problems, but these are some of the key ones. And to turn around a popular saying*, the past is still here, it’s just not evenly malingering. Many software organizations today still operate like this.

In response to this, ever greater numbers of developers started working on FOSS projects as side projects. In FOSS projects, they could collaborate with other developers on common building blocks developers themselves needed to solve developer problems while avoiding the unhelpful oversight or direction from management and non-developers. Collaborative cultures amongst FOSS community members emerged that motivated developers and made them happy to contribute in ways they were not usually inspired to do in their day jobs. Gradually, these developers made the corporations they worked for more comfortable with dependencies on FOSS software in their commercial projects and products, helping their teams see the benefits in terms of time to market and the efficiency of not re-inventing every wheel internally by building custom proprietary software.

The most successful FOSS projects are those that solve the most fundamental and broad problems that nearly every developer encounters. Operating systems. Data processing and management. Source code repositories. Continuous integration. For such problems, developers are the user persona themselves, and developers collaborating with one another produce optimum solutions. I would argue that other FOSS projects that attempt to serve a broader range of personas, like CRM, have been clear failures relative to their commercial market leader counterparts. (This topic brings up bad memories for me of having to reverse engineer and write my own documentation of SugarCRM’s schema in 2005 just to implement it for my company at the time.)

Meanwhile, back inside those bad old dysfunctional corporate software organizations, some other developers got together and started formulating new ideas of how those organizations could produce better software... and they created the Agile Manifesto in 2001. A few years later, the ideas of DevOps started to come together, with the first DevOpsDays conference in 2009. Agile introduced the revolutionary ideas of an iterative approach, with working software being incremented throughout a project, and encouraging close daily collaboration between developers and users and other functions. DevOps brought ops into the fold and introduced technical practices like continuous delivery that supported the feedback loops that strengthened Agile practice.

It turns out that the combined practices of agile and DevOps have had a transformative impact on the efficiency, quality and effectiveness of software delivery for corporate internal IT application development, commercial enterprise software and SaaS, and consumer software and services. Conversely, I argue that FOSS has not been able to be successful in those areas because developers are not representative of most, or even sometimes any, of their users, nor of the operations folk who keep many of these services running.

Both of these sound like awesome movements driven by great people with honorable motivations. Where’s the conflict?

Well, it’s pretty simple. In FOSS, frankly, developers make fast progress and stay happy because they don’t have to waste time collaborating with us dumb non-developers. Also, they can attack very discrete problems via small cells without having to coordinate explicitly with other cells. In agile and DevOps, collaboration amongst all roles - development, operations, product management, design, documentation, marketing, sales, success/support and users/customers themselves - is a central value and practice. And for the creation of sophisticated complex software, many agile squads need to contribute to a single end product.

Again, I think both work when applied to the right problems. But, when you try to get elite FOSS developers to work in a product organization that must now solve the problems of many personas, things can get a little hairy. Suddenly PMs are hovering. UX is questioned. Proprietary vs OSS and monetization decisions get questioned.

Here at CloudBees, an interesting wrinkle to all this we have faced is that some of the technologies that solve problems best solved in FOSS, like continuous integration (CI), have had their adoption driven at an enterprise level because of agile and DevOps transformations. So FOSS practices made Jenkins and now Jenkins X great - yet they are very different than the agile and DevOps practices that use of those tools in an enterprise are intended to enable. Yes, CI is used in FOSS projects and commercial software alike... but the context is entirely different.

And as we at CloudBees build toward the bigger vision of a new enterprise software category called Software Delivery Management (SDM) we’ve had to move from a classic FOSS development model to agile and DevOps ourselves... so these inherent differences have become more evident. And in the process, we’ve naturally come to understand our enterprise customers’ cultural challenges more deeply.

There’s a lot more to riff on around all this... differences between subcultures of different types of FOSS communities, nuance of politics of different functions in enterprise software organizations having dominance, application to SaaS vs self-managed software delivery, but I’ll stop here for now. What do you think?

* You’ve probably heard many people, including me, say “The future is already here, it’s just not evenly distributed.” Many conflicting attributions to that lovely positive sentiment.