Advanced Git with Jenkins

This is part of a series of blog posts in which various CloudBees technical experts have summarized presentations from the Jenkins User Conferences (JUC). This post is written by Harpreet Singh, VP Product Management, CloudBees about a presentation given by Christopher Orr of iosphere GmbH at JUC Berlin.

Git has become the repository of choice for developers everywhere and Jenkins supports git very well. In the talk, Christopher shed light on advanced configuration options for the git plugin. Cloning extremely large repositories is an expensive proposition and he outlined a solution for speeding up builds with large repositories.

Advanced Git Options

The are three main axes for building projects: What, When and How.


Git plugin options

What to build:

The refspec option on Jenkins lets you choose on what you need to build. By default, the plugin will build the master branch, this option can be supplanted by wildcards to build specific features or tags. For example:

  • */feature/* will build a specific feature branch
  • */tags/beta/* will build a beta version of a specific tag
  • +refs/pull/*:refs/remotes/origin/pull/* will build pull requests from GitHub

The default strategy is usually to build particular branches. So for example if refspec is */release/*, branches release/1.0, release/2.0 will be built while branches feature/123, bugfix/123 will be ignored. To build feature/123/ and bugfix/123, you can flip this around by choosing the Inverse strategy.

 
Choosing the build strategy

When to build:
Generally, polling should not be used and webhooks are the preferred options when configuring jobs. OTOH, if you have a project that needs to be built nightly only if a commit made it to the repository during the day, it can be easily setup as follows:

 

 

How to build:
A git clone operation is performed to clone the repository before building it. The clone operation can be speeded up by using shallow clone (no history is cloned). Furthermore by using the “reference repo” during the clone operation, builds can be speeded up. In the reference repo option, the repository is cloned to a local directory and from there on, this local repository is used for subsequent clone operations. A network access is made only if the repository is unavailable. Ideally, you line these up, so shallow clone for the first clone (fast clone) and reference repo for faster builds subsequently.

 

Equivalent to git clone —reference option

 

Working with Large Repositories

The iosphere team uses the reference repository approach to speed up builds. They have augmented this approach by inserting a proxy server (git-webhook-proxy [1]) between the actual repo and Jenkins. Thus, a clone happens to this proxy server. The slave setup plugin copies the workspace over to the slaves (over NAS) and builds proceed there on. Since network access is restricted to the proxy server and each slave does a local copy, this speeds up builds considerably. 

 

git-webhook-proxy: to speed up workspace clones

The git-webhook-proxy option seems a compelling solution, well worth investigating if your team is trying to speed up builds.

[1] git-webhook-proxy

 

— Harpreet Singh
 
Harpreet is vice president of product management at CloudBees
 

 

 

 

Comments

This comment has been removed by the author.

How can I configure Jenkins to clone a build process for each newly created branch? We currently enjoy this capability in Bamboo and TeamCity. We use the GitFlow model so branches are created and removed frequently.

You can use wildcards in the refspec, but then builds from the various branches are commingled. To show separate linear histories for each branch, your best bet is to use a project type supporting the Branch API plugin; currently this is limited to the Literate plugin, though others may be added.

Add new comment