A brief History of the Literate Builds plugin for Jenkins

In 2011, I had an idea. I was fed up having a nice Jenkins job for the controller branch, but losing that tooling every time I wanted to fork off feature development to a branch. GIT made branching cheap and easy, but Jenkins was not letting me keep that. Oh sure there was the hacks in the Jenkins GIT plugin where it would build all the branches, but then my trend graphs were gone. I could copy the project and create a new one for each branch, but that leaves a scattering of projects all over the place and more work for me to maintain them… plus the branch might only last a couple of days.

So there was pain and suffering. And I would just say, “Meh! I'll live without a Jenkins job for this branch”.

My idea was that you could have a Jenkins project type which automatically picks up all the branches and creates sub-jobs for every branch. That would be perfect. Now every time I create a branch I get the job for the branch. I knocked out a plugin, partly in my spare time and partly as a proof of concept of what it would look like as a feature for CloudBees. This resulted in what I called the cloudbees-multibranch-plugin , and it did exactly what it said on the tin… you could create a project and (because svnkit was easier to hack with) it would create a sub-job for every branch in the Subversion repository that the project followed. And that's where I hit the problems:

• Authentication is important, and the way Jenkins stored authentication was a complete mess. This spurred my development of the Credentials plugin to help try and tidy up that mess.

• Lots of Jenkins plugins make the assumption that their configuration screens will be displayed only by an AbstractProject class instance… and multi-branch projects could not inherit from that class and work the way I wanted them to. This gives lots of UI glitches with the form validation. Probably a solvable problem… but the next issue was the show-stopper.

• Very often one of the things people want to work on in a branch is refactoring the build process.

Once I realised that the build procedure will most likely not be exactly the same for every single branch, I realized that there was a more important problem. How do you describe how to build your project?

So I had a vision. A vision of a file that you would check into source control and it told Jenkins how to build the project. I came up with a snappy name: crux , being the most important part. How to build, test and run your project is the most important thing. I did up fancy slides. I created a toolchain to build projects using crux . I fleshed out project model in the toolchain to see how much was actually needed. (See if you can spot the mistake in this next one) I created an XML schema from the project model and XML parsers to read an XML format of the model so that I could test things out while I tried to sort out a DSL for the crux files.

The XML thing kind of killed it. No matter how much I said: “Ignore the XML, that's just the quickest thing I could throw together to show all the rest of this, the real thing will need a nice simple DSL rather than ugly XML” everyone just saw the XML and barfed all over it.

In any case, I had other things to work on.

Fast forward to this summer (if you are northern hemisphere), and I was chatting with Michael Neale and crux came up again. Somehow between the two of us we hit upon a brainwave… we were marvelling how nice GitHub formats the README.md files at the root of your GitHub repository when…

What if the README.md was the crux file?

Literate builds were born. I dusted off the old cloudbees-multibranch-plugin GIT repository and started to refactor it into something that could be released. We did a demo to the rest of the team. Some people got it, some people didn't, but we got the OK to push it a little further.

After a while we came to the conclusion that this feature was just too important to keep it closed source. Thankfully we have a cool CEO who was raised on Open Source and just gets that there are times when things need to be Open Source, so we got the OK…

The rest of this story is rather boring with lots of testing and careful but tedious adding of license headers to source code, only interrupted by a fun day's hacking with myself, Kohsuke and Jesse in Dublin where the API got a good shakedown.

The culmination of this story was this commit when the literate multi-branch project type was finally open sourced.

This is not a feature we are going to leave for dead. I believe this is the basis on which the next wave of Jenkins innovation will be built. I know I have lots of ideas for things that can be built on top of this… but the crux of this is out there now… and “literate” is a much better name than “crux” ever was!

The following plugins were involved in this story:

• Credentials plugin

• SSH Credentials plugin

• SSH agents plugin

• SCM API plugin

• Git Client plugin

• Git plugin

• Mercurial plugin

• Subversion plugin

• Branch API plugin

• Literate plugin

—Stephen Connolly
CloudBees

Stephen Connolly has over 20 years experience in software development. He is involved in a number of open source projects, including Jenkins . Stephen was one of the first non-Sun committers to the Jenkins project and developed the weather icons. Stephen lives in Dublin, Ireland - where the weather icons are particularly useful. Follow Stephen on Twitter and on his blog .