Dependency Finder – A Use Case for CloudBees Accelerator Annotation Files

A colleague recently asked if there was a way to identify all the source files that went into building a target, i.e. a way to programmatically determine all dependencies for a specific target. The use case my colleague presented was that this list of sources will allow his customer to identify open source files or files that correspond to packages with specific licensing terms. This is not necessarily the same thing as listing all source files used in the build but instead only specific ones that were read for the purpose of building the specific target. For example, in a full build, we may only be interested in source files that produced an intermediate target which might be a library or a package which eventually got integrated into the final product. In CloudBees Accelerator, this is made possible by two technologies: (1) the ability to generate annotation file for each build and (2) the ability to programmatically dissect this annotation file easily via the “annolib” Tcl package. You can read about both of those in this blog post by Eric Melski: Data Mining CloudBees Accelerator Annotation: Bill of Materials . The algorithm used to develop the solution looked like this: 1. Find the job(s) that wrote the target file --> 2. Find the files read by that job. --> 3. Find the jobs that wrote those files --> again. 4. Find the files read by those jobs --> again. 5. Repeat until you've checked every job that precedes the original job in serial order. Nominally, every file that was read by any job in that set could be considered an "input", either directly or indirectly. The resulting Tcl script is available on CloudBees’s GitHub community called “electriccommunity” in the public repository called “CloudBees Accelerator” in the folder called “dependencyFinder”. Usage: dependencyFinder.tcl <annotation_xml_file> <target_file> Example: ./dependencyFinder.tcl einsight.demonstration.xml c:/mysql/mysql-4.1.22/sql/mysqld.exe A demonstration XML file "einsight.demonstration.xml" is provided in the .7z (zipped) format on GitHub's "dependencyFinder" folder. Here is the response a user is expected to see:

*************** INPUTS ****************************
Annotation file:      einsight.demonstration.xml
Target file    :      c:/mysql/mysql-4.1.22/sql/mysqld.exe
***************************************************
c:/mysql/mysql-4.1.22/sql/mysqld.exe
  c:/cygwin/etc/passwd
  c:/cygwin/etc/group
  c:/cygwin/tmp
  c:/cygwin/bin/rm.exe
  c:/cygwin/bin/cygwin1.dll
  c:/mysql/mysql-4.1.22/sql
  c:/cygwin/bin/sh.exe
  c:/mysql/mysql-4.1.22/sql/my_time.c.lnk
      c:/cygwin/bin/cygintl-3.dll
      c:/cygwin/bin/cygiconv-2.dll
      c:/cygwin/bin/cygreadline6.dll
      c:/cygwin/bin/cygncurses-8.dll
      c:/mysql/mysql-4.1.22/sql/mini_client_errors.c.lnk
          c:/cygwin/bin/ln.exe
          c:/mysql/mysql-4.1.22/sql/pack.c.lnk
              c:/mysql/mysql-4.1.22/sql/client.c.lnk
  c:/cygwin/bin/sed.exe
  c:/cygwin/bin/tr.exe
  c:/cygwin/bin/grep.exe
  c:/mysql/mysql-4.1.22/sql/.libs
      c:/cygwin/bin/cygpcre-0.dll
      c:/cygwin/bin/mkdir.exe
      c:/mysql/mysql-4.1.22/zlib/libz.la
          c:/mysql/mysql-4.1.22/zlib/adler32.lo
              c:/cygwin/bin/cat.exe
              c:/cygwin/bin/gcc.exe
              c:/cygwin/lib/gcc/i686-pc-cygwin/3.4.4/specs
              c:/cygwin/lib/gcc/i686-pc-cygwin/3.4.4/cc1.exe
              c:/cygwin/lib/gcc/i686-pc-cygwin/3.4.4/include
              c:/cygwin/usr/include
              c:/cygwin/usr/include/w32api
              c:/mysql/mysql-4.1.22
              c:/mysql/mysql-4.1.22/zlib/adler32.c
              c:/mysql/mysql-4.1.22/zlib/zlib.h
              c:/mysql/mysql-4.1.22/zlib/zconf.h
              ...
              ...
  c:/mysql/mysql-4.1.22/innobase/ut/libut.a
      c:/mysql/mysql-4.1.22/innobase/ut
      c:/mysql/mysql-4.1.22/innobase/ut/ut0byte.o
          c:/mysql/mysql-4.1.22/innobase/ut/ut0byte.c
          c:/mysql/mysql-4.1.22/innobase/include/ut0sort.h
      c:/mysql/mysql-4.1.22/innobase/ut/ut0dbg.o
          c:/mysql/mysql-4.1.22/innobase/ut/ut0dbg.c
      c:/mysql/mysql-4.1.22/innobase/ut/ut0mem.o
          c:/mysql/mysql-4.1.22/innobase/ut/ut0mem.c
      c:/mysql/mysql-4.1.22/innobase/ut/ut0rnd.o
          c:/mysql/mysql-4.1.22/innobase/ut/ut0rnd.c
      c:/mysql/mysql-4.1.22/innobase/ut/ut0ut.o
          c:/mysql/mysql-4.1.22/innobase/ut/ut0ut.c
  c:/cygwin/lib/libdl.a
*****************************************************************
There are 1476 files that directly or indirectly
serve as input to make "c:/mysql/mysql-4.1.22/sql/mysqld.exe" file.
*****************************************************************