Jenkins CI: Distributed Builds

Jenkins can handle 100+ computers (called "build agents") and distribute the workload to them automatically. You can have different build agents on different OS and CPU architecture, so that you can create a heterogeneous build farm that supports all the environments that you need to run builds/tests on.

Build agents can be added and removed on the fly, and with a proper set up, your job configurations need not be modified to take advantage of the additional build agents.

Intelligent Job Dispatching
How to choose the build agent to perform a build is a very tricky question. For one thing, updating a workspace is faster than checking out a fresh copy, so builds shouldn't hop among different build agents too much. On the other hand, build agents can come and go, and available build agents should be utilized as much as possible.

Jenkins employs a variant of the consistent hash algorithm to achieve these conflicting goals, and it just works --- you need not configure anything!

Zero-setup Build Agent Installation
Build agents install themselves. You just need to specify the machine name and access credential (user name and password or private key) to Jenkins, and Jenkins will remotely perform installation and updates on its own. This substantially simplifies the deployment of a large build farm.

It is also possible to manually install a build agent, for environments where the auto installation is problematic.

Flexible Deployment Topology
Communication between the master and build agents is just a single bi-directional byte stream. This channel can be also established from either direction. There's no need for direct TCP/IP reachability between the master and build agents, nor a shared file system. This simplicity and flexibility makes it easy to tunnel connections across firewalls or route them in ways that satisfy your network policy. Master inside intranet and build agents on EC2? No problem!

Build Agent Health Metrics Monitoring
As the number of build agents increase, doing a good house keeping work on them gets more tedious. Jenkins will proactively monitor "vital signs" of build agents, and if any of them develop a problem, such as a low disk space, then it'll automatically cut off the build agent so that it won't cause build failures. Administrators can go in, resolve problems, and bring the build agents back online.

This helps you maintain the overall health of the entire build farm, even when individual build agents are unreliable. Monitoring is extensible, so custom health metrics can be added through plugins.

Automatic JDK/Ant/Maven Installation
As your cluster grows in size, deploying the tools necessary for builds to all the build agents becomes an overhead. It is particularly so if your cluster contains different operating systems and architectures. To make this type of environment more manageable, Jenkins can automatically install tools everywhere on demand. It picks up the right bundle depending on the environment and helps ensure that you have the exact same version of tools everywhere.