A single command should have the capability of building the system. Automation of the build should include automating the integration, and preferably running a script after the build ends, which deploys the software into a production-like environment.
The build script should not only compile binaries, but also generate documentation, website pages, statistics and distribution media (such as Red Hat RPM or Windows MSI files).
The Reality – Score B (Satisfactory)
Today’s build systems are capable of building complex software components automatically, generating documentation, distribution media etc., and also “deploying”, but only in the sense of placing the files on a designated network location. Does this pass as an “automatic build” according to CI best practices? Maybe, but only for an application running on one machine. As soon as additional machines are involved, the paradigm starts to break down.
Take a simple client-server architecture, with a server and two flavors of a client, one running on Linux and one on Windows. One way to automate this build is to use a tool like Jenkins to build all three components (server, client A and client B), and place them somewhere on the network. Then, someone has to manually take each of the three tiers and install it on the relevant server – but this breaks the requirement for automation.
A more sophisticated method is to use three instances of a build server like Jenkins to build each of the components, and save each of them to the correct machine in your testing or production environment. But this introduces more challenges:
Now you have three different build configurations to maintain. Given that branching is common, this could quickly multiply to dozens of builds, even in this small setup. There is a cost to managing dozens of build configurations. But beyond that cost, because of the complexity and the typically large number of artifacts (some of which are shared between the builds and some of which are unique), there will inevitably be errors that lead to broken builds – again requiring manual intervention.
Deployment is not as trivial as placing files in a directory. For example, if your server component is based on the .NET framework, it will need to be pre-installed on the machine before the server itself is installed. And what if the target machine has a previous version of .NET? Or if the previous version of the server has not been uninstalled? And how can you make sure that the target machine for each of the clients actually has the right version of Windows or Linux? Some of this can be automated with tools like Chef, but very often, manual labor still creeps in.
The build script is expected to run all tests automatically. What if you want to test each of the clients (Windows and Linux) against the server? Presumably the same test script will run on each client. If you deploy both clients and run the tests automatically, chances are that each client will make changes on the server, and these changes can create possible conflicts for the other client, which expects a certain initial state. What is needed in this case is to run a test on the first client, restore server to initial state, then manually run a test on the second client. This is not supported by today’s build tools, and so it will typically be done manually.
We’ve seen numerous “points of failure” in which even a simple 3-part build will not be fully automated. Now imagine a web application with 3 web servers, a load balancer, an app server and a cluster of 3 database servers – in this slightly larger setup, all these issues are made exponentially worse. With today’s build tools it will be almost impossible to automate this 8-part build, with full integration, deployment and testing, except with a heavy investment in scripting and manual maintenance, that most organizations will not wish to undertake.
What Can We Do to Improve?
A typical way to deal with this complexity is to “do less stuff”: run less automatic testing, install less components on CI and test environments, and handle production deployments manually, to make sure all components play well together. But all of these compromises take us further away from the original vision of CI.
To eliminate the manual work that creeps into the build process, automated build systems need to learn a few new tricks:
Build systems need to be able to manage both variations of the same build process, and shared components between different build processes. Otherwise you quickly end up with dozens or hundreds of build configurations, many of which are close duplicates or are inter-dependent.
Build systems need to be aware of deployment semantics – what are the characteristics of the machines we are deploying to, and what does each component need in order to run properly.
Build systems need to orchestrate deployments and testing in a more sophisticated manner – to enable scheduling and a sensible sequence of deployment and testing across multiple software components.
Does your build system do all of that?