Visualizing Build Processes

The other day I was waiting for my continuous integration build to finish. Think about that a minute. See anything wrong with that statement? I was waiting for a continuous integration build to finish. The whole point of continuous integration is to detect integration errors as quickly as possible . After years of growth, our builds are too long to really live up to that definition.
Obviously I have to do something about this, but what? Where do I start? The build is composed of several distinct phases: checkout, compile, unit test, package, install and system test. In addition, the complete process is actually replicated on several platforms simultaneously. In total, the process encompasses over 200 distinct steps, running in parallel on dozens of machines.

Fortunately, ElectricCommander tracks the duration of every step in the process, so I can find the "long pole" and focus my efforts there:

<div style="background: none repeat scroll 0 0 #deffde; border: thin solid; width: 60ex; margin-left: auto; margin-right: auto;"><span style="font-family: Courier New;">&#x3C;job>
  &#x3C;jobId>218513&#x3C;/jobId>
  &#x3C;jobName>ecloud-5.1.33976-201006251150&#x3C;/jobName>
  </span> <span style="color: red;">&#x3C;elapsedTime>7988201&#x3C;/elapsedTime></span> 
  <span style="color: red;">&#x3C;finish>2010-06-25T15:18:08.234Z&#x3C;/finish></span> 
  <span style="color: red;">&#x3C;start>2010-06-25T13:05:00.033Z&#x3C;/start></span> 
  &#x3C;jobStep>
    &#x3C;jobStepId>2796108&#x3C;/jobStepId>
    &#x3C;stepName>build-windows&#x3C;/stepName>
    <span style="color: red;">&#x3C;elapsedTime>363518&#x3C;/elapsedTime></span> 
    <span style="color: red;">&#x3C;finish>2010-06-25T13:25:12.528Z&#x3C;/finish></span> 
    <span style="color: red;">&#x3C;start>2010-06-25T13:19:09.010Z&#x3C;/start></span> 
  &#x3C;/jobStep>
  &#x3C;jobStep>
    &#x3C;jobStepId>2796141&#x3C;/jobStepId>
    &#x3C;stepName>build-linux&#x3C;/stepName>
    <span style="color: red;">&#x3C;elapsedTime>277503&#x3C;/elapsedTime></span> 
    <span style="color: red;">&#x3C;finish>2010-06-25T13:14:08.008Z&#x3C;/finish></span> 
    <span style="color: red;">&#x3C;start>2010-06-25T13:09:30.505Z&#x3C;/start></span> 
  &#x3C;/jobStep>
  ...
&#x3C;/job></div> 

Unfortunately , with hundreds of steps it's not at all practical to work with the data in this format. What I need is something that allows me to visualize the build process, so that I can see at a glance where the time is going. So I created the Plotter Plugin for ElectricCommander (click to view full-size):

Now instead of digging through hundreds of lines of timing information, I can see immediately where the problems are. Briefly, here's what you're looking at in this image:

• Each colored box represents a single step in the build process.

• Hovering over a box will display the name of the step.

• Clicking on a box will load the Commander "Job Step Details" page for that step.

• Boxes are colored according to the outcome of the step (red, yellow or green, corresponding to failure, warning or success, respectively).

• Boxes are positioned horizontally according to the start time of the step relative to the start time of the entire process. The correlary to this is that steps that are running in parallel "line up" vertically, at least in part.

• Steps which run subprocedures are not colored, and use a dashed outline to group the child steps.

In my case, the first problem is the two large, serialized steps near the top of the image, which correspond to execution of our system test suites on Windows. Now armed with this information, I can make intelligent decisions about how to improve my end-to-end build time.
If you want to try the Plotter Plugin, make sure you're running ElectricCommander 3.5 or later, then download the plugin from our public plugin repository , and use ectool installPlugin to install it. If you're not currently using ElectricCommander, you can get an evaluation copy.