Advanced Git with Jenkins

Written by: Harpreet Singh
3 min read

This is part of a series of blog posts in which various CloudBees technical experts have summarized presentations from the Jenkins User Conferences (JUC). This post is written by Harpreet Singh, VP Product Management, CloudBees about a presentation given by Christopher Orr of iosphere GmbH at JUC Berlin .

Git has become the repository of choice for developers everywhere and Jenkins supports git very well. In the talk, Christopher shed light on advanced configuration options for the git plugin. Cloning extremely large repositories is an expensive proposition and he outlined a solution for speeding up builds with large repositories.

Advanced Git Options

The are three main axes for building projects: What, When and How.

What to build:

The refspec option on Jenkins lets you choose on what you need to build. By default, the plugin will build the controller branch, this option can be supplanted by wildcards to build specific features or tags. For example:

  • _*/feature/* will build a specific feature branch_

  • _*/tags/beta/* will build a beta version of a specific tag_

  • _+refs/pull/*:refs/remotes/origin/pull/* will build pull requests from GitHub_

The default strategy is usually to build particular branches. So for example if refspec is */release/* , branches release/1.0, release/2.0 will be built while branches feature/123, bugfix/123 will be ignored. To build feature/123/ and bugfix/123, you can flip this around by choosing the Inverse strategy.

Choosing the build strategy
Choosing the build strategy

When to build:
Generally, polling should not be used and webhooks are the preferred options when configuring jobs. OTOH, if you have a project that needs to be built nightly only if a commit made it to the repository during the day, it can be easily setup as follows:

setup nightly built

How to build:
A git clone operation is performed to clone the repository before building it. The clone operation can be speeded up by using shallow clone (no history is cloned). Furthermore by using the "reference repo" during the clone operation, builds can be speeded up. In the reference repo option, the repository is cloned to a local directory and from there on, this local repository is used for subsequent clone operations. A network access is made only if the repository is unavailable. Ideally, you line these up, so shallow clone for the first clone (fast clone) and reference repo for faster builds subsequently.

Equivalent to git clone reference option
Equivalent to git clone --reference option

Working with Large Repositories

The iosphere team uses the reference repository approach to speed up builds. They have augmented this approach by inserting a proxy server (git-webhook-proxy [1]) between the actual repo and Jenkins. Thus, a clone happens to this proxy server. The agent setup plugin copies the workspace over to the agents (over NAS) and builds proceed there on. Since network access is restricted to the proxy server and each agent does a local copy, this speeds up builds considerably.

git-webhook-proxy speed up workspace clones
git-webhook-proxy: to speed up workspace clones

The git-webhook-proxy option seems a compelling solution, well worth investigating if your team is trying to speed up builds.

[1] git-webhook-proxy

-- Harpreet Singh

Harpreet is vice president of product management at CloudBees.

Follow Harpreet on Twitter .

Stay up to date

We'll never share your email address and you can opt out at any time, we promise.