Organizational Cartography: what kind of development shop are you?

If you've ever worked on a significant software development project you've probably looked at some part of your process and said 'we have to change this!' More often than not the topic is the software production process: that problematic build-test-release cycle. We can all identify places where our process is slow or broken; so why do some initiatives succeed while others fail miserably? If you're going to make a change which affects six people: stop reading - just go do it! Sixty? You'll have to get buy-in from colleagues. Six hundred? You're going to have to talk to your Director of Engineering. Navigating those waters benefits from drawing a map: you should know who your change will impact, how it will benefit them, and where the low-hanging fruit are. Over my four years with CloudBees I've successfully (and sometimes not so successfully!) sailed the process-improvement trade route; in this article I'm going to give you a heuristic in order to profile your own team and develop a navigation map.Are you building on a monolithic model, or a modular model? Consider this the first branch decision in evaluating the layout of your organization: is your product monolithic or modular ? There are fundamental differences between these design models, and the challenges of development will be different depending on which model applies to your team. Monolithic development teams are generally large, with dozens or hundred of developers working on the same final binary kit. Take a look at your build process: do you build the same binary as all of your office-mates? Do they build from the top-level .sln file or type 'make all' just like you? Does the build you do on the desktop take two hours? If you trace the execution of code do you see few API calls but a ton of interlaced functions? If so, you're building a monolithic application (something like Microsoft Office, PTC ProEngineer, or Cisco IOS). Modular development teams are characterized by large collections of small groups (2-8 people), each providing a separate deliverable which interlocks with other team deliverables. Does your team support your piece of the codebase to the rest of the development staff? Do you have your own design docs which are published out? Does your code compile in five minutes? Do you make a library or depend on a specific version of other internal elements? Do you find out about problems in your code only after release engineering gets through with it? Do you use a build process which is different from release engineering? You're in a modular shop (think Amazon.com, SonyEricsson, eBay). Let's also remember that the most common answer is 'maybe': the picture is complicated when you're a hybrid, but nearly every shop is. (Teams swing back and forth, too) Look for the modular model - small teams, API-as-contract - with a couple 'core' libraries or function groups that are huge. This is usually because some part of the product has to be monolithic - backend data processing, the kernel or standard library, a graphics engine or runtime - and everything else is middleware or leaf applications. Microsoft Windows Mobile and Symbian OS are examples of products like this. Here's why it matters: Monolithic developers have more of the same pain as the release team; these developers are more likely to have to link (or even compile) the whole app to test a code change . A developer may not build every language variant or platform (and usually can't!) but even a single platform 'full build' before checkin is expensive, and often means going home for the day. In this case, you've got a build speed problem. Release teams in this scenario are generally doing the 'official' builds, in blessed sane environments, and it's only a developer mistake which breaks their build. Speed is still the issue; managing builds is generally making sure every platform and variant is compiled. Modular developers chuckle at their monolithic colleagues - a component, which is often the extent of a developer's responsibility before checkin, may compile in sixty seconds - but release teams in modular shops sob. Why? Release engineers are integrating hundreds or thousands of components, each of which was built locally before checkin and generally not tested in the full system, and get to find all the integration problems at once. This leads to tons of 'try, fix, retry' loops as they discover problems, notify developers, and wait for new builds, builds which often take hours (due in no small part to the overhead of all those APIs and modular boundaries) instead of seconds. Your developers need a way to asynchronously test their changes in the production environment (a build management problem) and your release team needs help managing component dependencies, versions, and updates (also a build management problem). The release team could also use some speedup, especially when the product is forked and becomes two (or ten) different variants. So if you want to know where the low-hanging fruit is: look at your product architecture. Monolithic: developer speed. Modular: developer management, release team speed. Do the boundaries of your product elements line up to the boundaries of your organization? Now that you know whether the architectural model is monolithic or modular, you know how to identify what changes are worth making. Don't stop there! You'll need to scope more than just the benefit of a change: you should know how many people (and in what roles!) the change will impact. One easy way to evaluate this is by looking at the boundaries of code responsibility. Strong boundaries are characterized by tight grouping of responsibility and clear knowledge limits. Can you identify what part of the product code an engineer works on based on what group she's in? Is the SCM system aware of which users are approved to work on certain functional areas? The more distinct the divisions between functional areas of code the more likely it is that developers work on narrow areas of responsibility. This is common in modular product shops: groups are independent, and so are their code bases. Ambiguous boundaries tend to be ad-hoc, more to do with areas of developer preference than intentional roles or assignments. Are there guru or lead programmers in your teams who answer questions for everybody? Do development tasks (and bugs) get fluidly reassigned when someone is on vacation or busy on a special project? Are there lots of developers to ask about any specific technical question? Monolithic projects are more condusive to a large, multipurpose team (but you'll still find striations along functional units, I guarantee it). Here's why it matters: The reason you should watch this closely has to do with decision-makers: can you get buy-in for your idea from the engineering director? Do you need to? In a team with clear boundaries in its components you might get away with just convincing the component developers. A more amorphous team has to deal with individual developer preferences: if Bob works on component A and B he probably doesn't want to see A built with 'make -j' and B built with MSBuild. Most importantly, keep an eye on standardization: if you want to change a process used by everyone the impact is going to be broad; any change will require lots of support! Know whether you need buy-in from the whole team (and in this case, your champion better be that director, and he'll want to understand a comprehensive ROI ) or just to a specific group (these decisions can often be made in the lunchroom, on technical merits alone). By scoping the size of the impacted group you'll know how high in the organization you need to sell your idea, and who might be opposed to your proposed solution. Do you ship code (and its associated development tools) or do you ship a binary product? We've just looked at scoping impact on development teams within your organization. For some types of product organizations, though, you're going to have to look at more than just the internal engineering efforts: who uses the product of your labors? This is a key question, and is at the heart of the question of ownership and impact analysis. Finished products draw a distinct line between the needs of development and the needs of consumers. These types of applications tend to be independent from their development model. Does the user of your product (whether it's sold retail or is a purely-internal tool) get a fully packaged and self-contained product? Is it used stand-alone? Intuit (for example) builds a binary product - Quickbooks - and ships the executable to its customers. Intuit is thus free to use any build system they want, and can change the infrastructure used to support development as much as they'd like. Middleware, SDKs, Toolkits and Platforms are more like the components of a sailboat - it's a finished product, but will be used (and often recompiled or modified) as part of a larger system. Do you have sample applications built on your product? Do you ship source code? Do you make sure your development tools are redistributable? Qualcomm ships source and tools as a board support package with their MSM product; switching from 'make' to 'SCons' would require all their customers to switch to (and support!) SCons as well. Here's why it matters: If you ship a tool environment outside your walls - or to other groups in your company - you'll need to look at the impact on your customers before making a change. The visibility and impact of your change may extend outside of development and up to Product Management, Marketing, Support, even Sales. Don't be deterred; a change which benefits your team will undoubtedly offer benefits for your customers, but the scope of buy-in you'll need gets much more broad. Consider asking the vendors of your tools if you can ship their products (some will be happy to have you do so!) If you consume a platform product, look into what the OEM is using (it's a good bet they use a different process than what they supply to you!) Either way, be sure you know who is affected before proposing a tooling change. Is your infrastructure centralized and standardized, or do you have autonomy? Remember we said it's important to scope how many people and in what roles a proposed change will impact? Another way to evaluate this is to look at the level of common tooling and process is in your organization. This is a tell-tale which can focus your efforts on the guys with the juice to approve (or to quash) a proposed change. Do you know who administers your SCM system? Bugtracking system? Compiler environment? When a new build machine is delivered, does IT image it for you? In a centralized shop, developers get little control over infrastructure systems, and those systems can sometimes have lots of impact on development practices. Ever heard of a 'blue' shop? This is a slang way of referring to development organizations which are consumers of all things IBM; there's usually a central engineering group which sets policy (you WILL use ClearCase) or provides all the core infrastructure (Here's your Rose, your Purify, your Team Concert). Did your lead engineers pick the build tool when the project stated? Does your '/usr/local/tools' directory have Perl, Python, Ruby, and TCL interpreters? Some organizations - JPL is a classic example - let every internal team pick their own tools and generally stick that team with supporting itself. This gives NASA scientists the autonomy to use whatever tools are optimum for their current project, but it also means the project lead is the single most important decision-maker for your change request. Most shops - just like the monolithic/componentized decision - are a little of both: Electronic Arts provides some central tools, but allows individual studios a significant range of leeway in picking their own tools. This may be the most fertile ground for supporting process change: get a pilot group to adopt a new approach, show some success, then use that success as a case study to gain additional adopters. A snowball effect will lead to an enterprise-wide change once central engineering sees the growth of adoption. Here's why it matters: You've got to know who to sell to! Central Engineering (if it exists) has to be engaged in a monolithic model - everyone has to use this stuff! Standardization on Microsoft Team System, for example, pretty much means you're going to use MSBuild or devenv to run your builds. Want to use SCons? You're pretty much out of luck. It's not always easier on the other side, either: component developers are much more flexible - remember you're just building your little component! - but Release Engineering is going to hate you if you give them the first JAM project in a sea of Visual Studio. Look at who holds the budget, who is impacted, and who has the need for change before mapping out a project plan and you'll build success in from day one. Whew! Feeling like Magellan yet? It's worth it, though; by understanding the shape of the organization you're in you'll gain enormous insight to what changes to your software production methodology will help (and make you a star!) and which ones would be crucial mis-steps. Now try it out! Ask your co-workers about the history of the tools you're using now, and see how your organization maps.