Accelerating Ninja with CloudBees Accelerator

CloudBees Accelerator is an acceleration platform that optimally parallelizes software tasks across clusters of physical or cloud CPUs. This gives software-driven organizations the ability to speed up any number of concurrent activities so they can deliver better software faster. CloudBees Accelerator offers the following unique capabilities:

• Safe and deterministic object file caching
• Optimized dependency management
• Reliable blazing incremental builds
• Fault-tolerant workload distribution
• Enterprise scalablity
• Flexibility and manageability
• Rich analytics and monitoring

CloudBees Accelerator works on the basis of an agent model . Agents are daemons that run on agent boxes, usually 1.5 x #_of_cores. These agents are managed by a Cluster Manager , which represents the gateway to the machines running the agents, or what is referred to as a cluster. eMake and tools are installed on build machines owned by developers or build managers, and that is how builds are initiated. In a very simplistic explanation, eMake talks to the Cluster Manager and asks for agents for a task. Once it receives a list of the agents and their IP's, it then orchestrates the parallelization of the build.

Accelerating Ninja Builds

In the case of builds that use Ninja , eMake is not an option because it does not know how to read Ninja build files. Ninja does support parallelization in a manner similar to GMake, with both default automatic parallelization based on the number of cores, and a -j option that can be used to override the default value, forcing Ninja to produce N number of simultaneous processes to parallelize a build. That means that if you want to parallelize using just Ninja, and you want blazing fast speed, you have to be prepared to spend blazing big money for a large server with high numbers of cores. CloudBees Accelerator, however, can give you the ability to pile up large numbers of inexpensive boxes with i7 processors, for example, to build a cluster of unlimited size. What it also provides is an alternate way to access the parallelization infrastructure of the cluster without having to use the eMake tool. The key to this feature is a tool called Electrify which is included with CloudBees Accelerator at no extra charge. This tool can run an executable and monitor it as it spawns processes. These processes can then be selectively "intercepted" and sent to the build cluster for parallel execution. Now, this sounds simple on its face, but the requirements for successful parallelization are strict and can present challenges:

• The processes themselves must be just that, processes that are distinct and can run on their own without having to operate under some underlying runtime. This requirement excludes Java threads, because they cannot be separated from their underlying JRE without breaking the thread.
• The processes must also not have dependencies between each other that would cause them to step over each other during execution, such as sharing files in read/write mode.
• In addition. parallelization itself benefits from tasks that can be broken up into small pieces that can run simultaneously. Some large tasks, such as linking libraries or executables, creating .tar or .deb packages or other similar tasks cannot be broken up, so they do not contribute to the performance of the build as a whole if they are parallelized. All they do is add the overhead of sending the task to the agent, creating network traffic, etc. and as such are better off running on the machine from which the build was initiated. Those are referred to as local tasks, and in ElectricMake they are designated as such with a #pragma runlocal statement in a Makefile. More on how Electrify can keep those running on the build machine in a minute.

How to parallelize Ninja builds with Electrify

Because ElectricCloud has customers who like Ninja and don't want to switch to Makefiles, but who would also like to take advantage of our parallelization tool, I decided to see if I could parallelize Ninja builds with Electrify. The short answer is that it works, and it works very, very well. The longer answer is that it takes some prep to be able to do this. This is a rough list of what needs to be done and it assumes you already have a cluster of agents and a Cluster Manager installed. If you are using the peer-to-peer CloudBees Accelerator Huddle  it will work as well, but of course it will be limited to the number of cores available in your peer-to-peer networked setup.

1. Install CloudBees Accelerator tools on the machine where you are currently running builds.
2. Install whatever dependencies your builds needs (such as shared libraries) in the machines where the agents reside. For the Google Chrome build, for example, this is done with the build/install-build-deps.sh script. If you don't do this, Electrify will still send those to the agent when it needs them, but it will be more work for you and more work for the network and agents.
3. You will need to identify what CloudBees Accelerator refers to as the "eMake root" of the build. If you are using eMake already you know what this is -- one or more directories from which eMake will identify files that the agents need to complete an assigned task. In the case of the Google Chrome build, it's the./chromium directory.
4. My recommendation is to build a script with the command line you will use to launch the parallelized build. The command line I use to launch the Google Chrome build is this:

electrify --emake-annofile=build-@ECLOUD_BUILD_ID@.anno 
--emake-annodetail=basic 
--emake-annoupload=1 
--emake-root=<home_dir_where_chrome_build_lives>
--emake-cm=<ip_of_your_huddle_main_server>
--electrify-allow-regexp=".*"
-- ninja-linux64 -j 300 -C out/Release chrome

In order, the arguments mean:

• Write the annotation file to a file named build_(build-number).anno .
• Include the only annotation detail possible with Electrify, basic annotation.
• Upload the annotation to the cluster manager at the end of the build.
• Set the emake-root to a specified folder from which Electrify will serve files to the agents.
• Intercept all spawned processes and send them to the cluster.
• -- indicates the end of the Electrify command line and the beginning of the command line for Ninja
• Launch the 64-bit build of Ninja directly (because ninja is usually just a script).
• Launch up to 300 processes at at time (ElectricMake will que them appropriately, adjust the number generally to 3x the number of agents available in your cluster).
• Go tor  the out/Release directory where you will find the build.ninja file.
• Build the Chrome target.

Adjust as necessary. The documentation for Electrify is in the CloudBees Accelerator Electric Make User Guide .

Some things to note:

I said Electrify can be told to keep selected processes from being sent to the cluster. You do that with the --electrify-not-remote=   or --electrify-deny-regexp=   command line options, explained in the User Guide. Also, if you are familiar with Electric Insight and its fantastic visualization capabilities into your builds (see below), you will be disappointed to know that Electrify does not produce the annotations necessary for Insight's more advanced features, such as ElectricSimulator. Note that all of this can also be done with the free peer-to-peer CloudBees Accelerator Huddle. Finally, if you do have Electric Insight, this is what you want to see as a result of a Ninja build parallelized with Electrify. It's a thing of parallelized beauty that would make Pink Floyd proud, the brick wall. This is the full clean Google Chrome build, shrunk to a mere nine minutes and a handful of seconds.

And the really awesome part of the story? The acceleration was done with agents on virtual machines running on Skytap  at a fraction of the cost of standing up, maintaining and operating an infrastructure. From a template, I had a 96-core cluster up and building in minutes. How awesome is that!? Any questions? CloudBees Contact is your friend. Have fun with your new warp speed engine!