Data Mining CloudBees Accelerator Annotation: Bill of Materials

CloudBees Accelerator annotation files contain a gold mine of information about your build, such as the dependencies between jobs in the build, the time required to run each job, the exact command-line and environment used to invoke each command in the build, and even every file read and written by each job in the build. Many people have correctly speculated that they could use the file access data in annotation to create a bill of materials for the build, similar to so-called "configuration records" in ClearCase. In this post, we'll look at how we can do that using the annolib library.

Getting Started

The first step is to generate an annotation file with an appropriate level of detail. Annotation is not enabled by default, and even if it is enabled, not all information is included by default, because there is a performance impact when using some of the more verbose types of annotation. In this case, we need only file level annotation, which incurs less than a 2 performance penalty. To enable file level annotation, you must add the following options to your emake command line:

--emake-annodetail=file --emake-annofile=emake.xml

When your build completes, you'll find the file emake.xml contains an XML annotated build log. Each job is identified by a <job> tag, which includes an <opList> tag that contains an <op> tag for each file read or written by the job. For example:

<opList>
<op type="read" file="/blog/subdir" filetype="dir" isdir="1"/>
<op type="create" file="/blog/baz" found="0"/>
<op type="read" file="/blog/foo"/>
<op type="read" file="/blog/symlink_to_bar" filetype="symlink"/>
<op type="read" file="/blog/bar"/>
</opList>

Each operation is tagged with a type attribute that tells you the nature of the operation; in addition to the read and create operations in this example, you may see lookup , modify , unlink , rename , link , modifyAttrs , and append operations. These are very closely related to the usage types described in the agent performance metrics that we've previously explored . For purposes of generating a simple bill of materials, we're not so much concerned about the differences between the types of operations except to group them into two buckets: read operations, and everything else.
Besides the type of the operation, the <op> tag gives the full path to the file and the type of the file, one of dir , symlink or file . Many operations do not explicitly declare the file type, because the default value is file according to the annotation DTD, so we save a little bit of space by not including that attribute when it is redundant.

Constructing the bill of materials

The term bill of materials has many different definitions in the build space, but here we are specifically defining it as the list of source files read in the process of generating the final output of the build, subject to the following constraints:

1. Directories and symlinks should be excluded.

2. Files created during the build should be excluded, even if they are read by a later job in the build.

3. Makefiles should be excluded, for compatibility with ClearCase.

You may find that you can do a lot of analysis with standard utilities like grep , but I find that beyond relatively simple tasks, it's easier to use annolib , an annotation processing library created for this purpose. Annolib is implemented as a loadable Tcl extension, so using it means writing a short Tcl script leveraging the facilities in the library. Based on the requirements above, I wrote the following annolib script:

switch -glob -- $tcl_platform(os) {
    Windows* {set InstallDir C:/ECloud/i686_win32   ; set ext dll}
    Linux    {set InstallDir /opt/ecloud/i686_Linux ; set ext so }
    SunOS    {set InstallDir /opt/ecloud/sun4u_SunOS; set ext so }
}

load $InstallDir/bin/annolib.$ext

set anno 
set xml   r]
$anno load $xml

set bom {}

foreach file  {
    if {  != "file" } {
        continue
    }

    foreach op  {
        foreach {job type file} $op { break }
        if { $type != "read" } {
            break
        }

        if {  == "parse" } {
            continue
        }

        lappend bom $file

        break
    }
}

foreach file $bom {
    catch {puts $file}
}

You can run the script like this tclsh bom.tcl emake.xml ; it will print to standard out a list of all the source files read by the build. Here's how it works, line by line:
Line 07 Load the annolib library, using the install directory we determined previously.

An exercise for the reader

This simple script just scratches the surface of what you could do with a bill of materials report. For example, after generating the list of input files, you could query your SCM system for version information on each file and include that in the output. Or you could restrict the output to only those inputs that contribute to a specific output of the build, rather than all the outputs as this script does. What can you do with the gold in your annotation files?

About CloudBees

CloudBees powers Continuous Delivery . We help organizations developing mobile, embedded systems and enterprise web/IT applications deliver better software faster by automating and accelerating build, test, and deployment processes at scale. Industry leaders like Qualcomm, SpaceX, Cisco, GE, Gap, and E*TRADE use CloudBees solutions and services to boost DevOps productivity and Agile throughput.