Literate Builds Make Configuring Jenkins Easier

Stephen Connolly's picture

Every software project has two kinds of information:

  • Information that is of a wider scope than the project’s source code repository; 

  • Information that is implicitly tied to the project’s source code.

This split is important, because if you store one kind of information in the other place you open yourself up to a world of pain. Let’s look at an example: “When the FooBar project is built, we need to deploy it to a staging environment.” This piece of information is wider in scope than just the FooBar project’s source code. The source code does not know the IP address of the staging environment. The source code does not know the credentials used to deploy to the staging environment. Storing those things in source control is tempting… of course it is, we’ve all done it at some time after all. Ultimately, though, it is the wrong thing to do: you can’t redeploy old revisions to verify bugs because the IP address or credentials have changed, etc.

Another example: “We build the FooBar project by running the SuperBuzzDing build tool with the Bargle command line parameter.” This piece of information is very tightly coupled to the source code you are building. It may be that when you get to version 2.0 of FooBar you need to add some more bits to the command line. Or maybe the build process gets completely refactored to a different build tool entirely.

Traditionally, Jenkins has taken the approach that it does not require anything in the source code repository that you are building. A sort of loose coupling if you like. This is good up to a point. It allows you to keep the information that is greater than a single project outside of that project. Changing the staging environment does not necessitate source code changes. But that flexibility comes at a cost. We have no standard location in which to store the information that is tightly coupled to the source code. Literate builds provide such a location, and in so doing enable some very useful features.

Literate builds use the README.md file in the root of a software project to store the build instructions. In most normal projects the README file is either missing or hopelessly out of date. By using this file as the source of build instructions, we ensure that the file is kept correct and up to date. If the build tool changes from a batch file / shell script to a Makefile to an Ant build.xml to a Maven pom.xml, it doesn’t matter, because the README.md will be updated with the code (or Jenkins will report a failed build).

Another important piece of information that is very relevant to the project’s source code is the exact toolchain that you need to build the source code. Version 1.0 of the project may need Ruby 1.8 on Windows, version 2.0 may need Ruby 1.9 on Windows or Linux and version 3.0 may be built on Ruby 2.0 but only on Linux. This is the kind of information that people typically keep in their README files anyway. By leveraging this information, Jenkins is able to validate the build instructions against all the target environments and ensure that not only are the instructions correct, but that they are correct on all the supported platforms.

The final piece of the puzzle that literate builds provide is that you can now let Jenkins auto-create jobs for all the branches of your project. Because each branch now stores its own relevant build instructions, you can have the branch project jobs be created (and destroyed if you choose) as each branch gets created. This enables a much more flexible style of development, where creating a branch is significantly less of an administrative overhead.

To paraphrase a few Jenkins administrators: “I hate branches because I have to go and create five or six jobs for each branch, and then two weeks later the branch is gone again.” With the literate builds plugin, Jenkins will automatically create these jobs and remove them again when they are gone. The Jenkins administrator is free to go back to his or her primary job.

I will finish off this post with an example of what a literate build looks like:

FooBar project
==============
This is the FooBar project that allows you to block Foo from coming into your computer.
Environments
------------
The project can be built on the following environments:
* `ruby-2.0`, `java-1.7`, `maven-3.0.5`, `rake-10.0.3`
 * `linux`
 * `windows`
 * `osx`
How to build
------------
* On `windows`
 call mvn.bat clean verify
 call rake.bat
* On `linux` 
 mvn clean verify; rake
* On `osx`
 mvn clean verify; rake

The above example is both a nice, clean README.md file and a description of the build for Jenkins to use.

For more information about literate builds see the following resources:

—Stephen Connolly
CloudBees
www.cloudbees.com

Stephen Connolly has over 20 years experience in software development. He is involved in a number of open source projects, including JenkinsStephen was one of the first non-Sun committers to the Jenkins project and developed the weather icons. Stephen lives in Dublin, Ireland - where the weather icons are particularly useful. Follow Stephen on Twitter and on his blog.

 

Comments

wow, great idea!

Is there a way to define multiple Jenkins jobs from within the same literate build description? The git-client-plugin now includes coverage reporting as part of its build description, but I'd also like to propose that it include findbugs reporting. I think the findbugs reporting should be run on a different job that the coverage reporting so that it does not slow the throughput of test execution. Also, is there a plan to move the literate build plugin from the experimental update center to a fully released plugin?
Stephen Connolly's picture

There are two point five ways to defining multiple jobs from within the same literate description. The first part is that the `build` section name should be configurable (may not be yet... but that's because there is still some more important functionality to add)... so once the UI support for that (the model builder has support for it: https://github.com/jenkinsci/literate-api/blob/master/src/main/java/org/cloudbees/literate/api/v1/ProjectModelRequest.java#L127) you can have multiple jobs by using different section headers. The next way is that there is going to be lightweight promotion process support in the 1.0 release. That allows other sections to be exposed as "promotions" just like you get with the promoted builds plugin... only not quite so process heavy... most likely manual only triggers... with perhaps support for parameters... though that might come later... I might support some profiles being "automatic" i.e. they run if the build succeeds... remains to be seen if I can do that without bloating the UI too much. The final .5 way is through build profiles. So by default we pick up publishers from the `.jenkins` directory. You can however specify build profiles... which mean that the `.jenkins/profile-name/` directory has preference. I think it would make sense if the `.jenkins/profile-name/README.md` file was considered as a preferential source of build instructions when building a specific profile, assuming it exists that is. That would allow multiple jobs from the same SCM... but it would not be from the same description... though, again, this is yet to be implemented... another route to the same is to change the marker file name, as the marker file is a preferential source of the build description. There are about 2-3 major features that I think are needed to be complete before I declare the plugin out of "experimental". The reason for keeping it in "experimental" is that there may be some changes required to data format in order to get the last 2-3 features delivered... and once we move out of "experimental" I would view the on-disk format as more rigid (which does not mean I won't try and migrate existing data while in experimental mode, just if I get really stuck I have an "out" ;-) )

@Mark @Stephen - I think one more thing to get it out of experimental (on top of your list): a better name. I like the term "literate" but I think the real feature is the branch discovery and build definition. If it uses a literate style or not is probably less of a concern. But - ideas for a name? Some people would like this to be a peer level to the current freestyle - ie instead of freestyle you may well just use this (given that many freestyle builds simply say "./build.sh" in their build step).

Add new comment