Automatic Dependency Management with CloudBees Accelerator

One of the well known problems with make is that it's a real nuisance to completely specify all the dependencies in your project. For example, if you have a file main.c in your project, you probably already have a dependency like this in your makefile:

main.o: main.c

However, if main.c includes logging.h, you technically need to have a dependency like the following too -- but you probably don't have it:

main.o: logging.h

The kicker is that without this additional dependency, make will fail to realize that it needs to rebuild main.o if there is a change in logging.h. Of course, this wreaks havoc on your ability to do reliable incremental builds.

makedepend and compiler-generated dependencies

The classic solution to this problem is to use one of the automatic dependency generation mechanisms, such as makedepend , or compiler-generated dependencies, if your compiler supports it (g++ does, for example, via the -MM switch ). For example, you can add basic makedepend support to your makefile by adding the following lines to the end of your makefile:

SRCS=main.c
depend:
        @makedepend -f -s $(SRCS)

# DO NOT DELETE THIS LINE -- make depend depends on it.

Then, you can invoke make depend to generate dependencies; makedepend will append them to the end of your makefile after the magic "DO NOT DELETE" line shown above. This certainly gets the job done, but it has some drawbacks. Owen at The Grimoire has a more thorough exploration of the shortcomings, but basically the problems are:

• It's a manual process; if you forget to run make depend periodically, your dependencies will get stale.

• It's clumsy, because it relies on magic tokens in and modifies your makefile.

• It adds complexity to your makefile.

• It's slow.

Some of these deficiencies can be addressed with a more sophisticated integration. My good friend Mr. Make has a great article exploring that option, as well as compiler-generated dependencies, but keep in mind that going that route will add even more complexity to your makefiles.

ElectricAcclerator autodepend

If you use CloudBees Accelerator, you have another alternative: autodepend . This feature serves the same purpose as makedepend, etc, in that it automatically generates dependency information for you, so that you don't have to manage that data yourself. The big difference is that it's better, of course, in every conceivable way:

• It's truly automatic: add --emake-autodep=1 to your command-line and never worry about it again.

• It's transparent: the additional dependencies are tracked in a separate file, so you don't have to modify your makefile.

• You don't have to change your makefiles or tool command-lines at all.

• It's completely toolchain, platform and language independent.

• Best of all: it's fast .

Autodepend uses the filesystem usage information that we already collect over the course of the build to get perfect dependency information for every file generated by the build. If the command that produces main.o reads logging.h , we'll see that and record the implicit dependency, just as if you had added main.o: logging.h to your makefile. Since we're already collecting the data anyway, autodepend adds almost no overhead, but you don't have to take my word for it. I set up a test environment that let me easily switch between makedepend, g++ -MM dependencies, and autodepend, then ran a small build using each mechanism three times. The build consisted of about 75 C++ source files, referencing about 150 header files. The times shown here (in minutes:seconds format) reflect the total time to generate dependencies and run the build itself:

I think the results are pretty compelling: using autodepend added no appreciable overhead to the build time, while the alternatives added 20-30 to the total build time. If that's still not enough to convince you to give it a try, maybe this will: just last week, one of our customers switched from compiler-generated dependencies to autodepend and saw their build times drop from 32 minutes to just 18 minutes. So give it a try: you have nothing to lose -- except for long builds, of course. Update (19-DEC-2008): Added benchmark data for g++ -MD , a variant of compiler-generated dependencies that emits the dependency file at the same time the compile itself is performed, rather than as a separate step, which improves the efficiency of that technique.

Build Acceleration and Continuous Delivery

Continuous Delivery isn’t continuous if builds and tests take too long to complete. Learn more on how CloudBees Accelerator speeds up builds and tests by up to 20X, improving software time to market, infrastructure utilization and developer productivity.