ElectricMake Temporary Directory Settings

This post was originally included as part of the 2008 Customer Summit presentation on ElectricAcclerator Performance Tuning

In some earlier posts , we've looked at some of the performance metrics that CloudBees Accelerator agents collect and how you can use that information to understand the performance of your build. In this post, we'll start to explore the performance metrics collected by ElectricMake itself. In particular, we'll look at the metrics related to the emake's temporary directories, how you can identify situations in which your settings are suboptimal, and how you can address the problem to improve build performance.

As a build runs, generated files are written to a temporary location, and only when the creating job is determined to be not in conflict are the files relocated to their final, correct location. This is required in order to preserve the atomicity and transactional nature of the system. If emake did not handle files this way, it would be impossible to roll back reverted jobs, for example.
Files are committed to their final location using one of two techniques: commit by copy, which is guaranteed to work in all cases but which may be slow; and commit by rename, which is very fast but which may not be possible in all cases. Specifically, commit by rename is prohibited by the operating system whenever the temporary location and the final location are on separate physical disks. Because it is much faster, emake uses commit by rename whenever possible.
Emake reports the portion of files committed using each mechanism in emake performance metrics , which you can obtain by adding the --emake-debug=g option to the command-line:

emake --emake-debug=g --emake-logfile=emake.dlog

When the build completes, you will find that the file emake.dlog contains hundreds of individual performance metrics, such as network bandwidth and lock contention within emake. In this case, we're interested in the Commit by counters:

Terminator:
 Commit by copy:             10 files 1048.6 MB,   9.1 MB/s overall
 Commit by rename:            0 files   0.0 MB,   0.0 MB/s overall

Ideally, Commit by copy will be small relative to Commit by rename . In this example, you can see that Commit by copy is much larger than Commit by rename , which indicates that emake is using a suboptimal set of temporary directories for the generated files. By default emake creates one temporary directory for each emake root, which usually results in good performance. However, that may be inadequate if a single root spans multiple physical disks. For example, a build that uses files from several network file systems like /net/src and /net/tools might simply specify /net as an emake root. This is convenient, but forces emake to use copying to commit files in those locations, because a temporary directory created in /net is likely not on the same disk as /net/src or /net/tools .
You can direct emake to create temporary directories in specific locations with the --emake-tmpdir command-line option. In this example, a better temporary directory yields the following improvement:

Terminator:
 Commit by copy:              0 files   0.0 MB,   0.0 MB/s overall
 Commit by rename:           10 files 1048.6 MB,  29.3 MB/s overall

The practical impact of this tweak was a better-than-3x improvement in build runtime:

One place this problem shows up frequently is in ClearCase environments, where the emake root typically includes /view or /vobs . Every vob and view in ClearCase is considered a different disk, so emake is unable to use Commit by rename to efficiently move files between them. You can address this by explicitly declaring a temporary directory for each vob/view with --emake-tmpdir , or as of CloudBees Accelerator 4.x, you can leverage the built-in ClearCase integration to automatically create a temporary directory for each by adding --emake-clearcase=vobs to your emake command line.