Faster Pipelines with the Parallel Test Executor Plugin

This is the first in a series of posts showing ways to use Jenkins Pipeline.In this post, I'll show you how to speed up your pipeline by using theParallel Test Executor Plugin.

So Much To Do, So Little Time

In my career, I've helped many teams move to continuous integration and delivery. One problemwe always encounter is how to run all the tests needed to ensure high-qualitychanges while still keeping pipeline times reasonable and changes flowingsmoothly. More tests mean greater confidence, but also longer wait times.Build systems may or may not support running tests in parallel, but they still only use onemachine even while other lab machines sit idle. In these cases, parallelizingtest execution across multiple machines is a great way to speed up pipelines.The Parallel Test Executor plugin lets us leverage Jenkins do just that with nodisruption to the rest of the build system.

Serial Test Execution

For this post, I’ll be running a pipeline based on theJenkins Git Plugin. I've modifiedthe Jenkinsfile from that project to allow us to compare execution times to ourlater changes, and I've truncated the "mvn" utility method since it remainsunchanged. You can find the original filehere.

LANG:code
node {
stage 'Checkout'
checkout scm

stage 'Build'

/* Call the Maven build without tests. */
mvn "clean install -DskipTests"

stage 'Test'
runTests()

/* Save Results. */
stage 'Results'

/* Archive the build artifacts */
archive includes: 'target/*.hpi,target/*.jpi'
}

void runTests(def args) {
/* Call the Maven build with tests. */
mvn "install -Dmaven.test.failure.ignore=true"

/* Archive the test results */
step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml'])
}

/* Run Maven */
void mvn(def args) { /* ... */ }

NOTE: This pipeline expects to be run from a Jenkinsfile in SCM. To copy and paste it directly into a Jenkins Pipeline job, replace the checkout scm step with git 'https://github.com/jenkinsci/git-plugin.git'.

This is a Maven project, so the Jenkinsfile is pretty simple.I’ve split the Maven build into separate “Build” and “Test”stages. Maven doesn’t support this split very well, it wants to run allthe steps of the lifecycle in order every time. So, I have to call Maven twice:first using the “skipTests” property to do only build steps in the first call,and then a second time with out that property to run tests.

On my quad-core machine, executing this pipeline takes about 13 minutes and 30seconds. Of that time, it takes 13 minutes to run about 2.7 thousand tests inserial.

Parallel Test Execution

This looks like an ideal project for parallel test execution: a short buildfollowed by a large number of serially executed tests that consume the most ofthe pipeline time. There are a number of things I could try to speed this up.For example, I could modify test harness to look for ways to parallelizethe test execution on this single machine. Or I could try speed up the teststhemselves. Both of those can be time-consuming and both risk destabilizing thetests. I'd need to know more about the project to do it well.

I'll avoid that risk by using Jenkins and theParallel Test Executor Plugin toparallelize the tests across multiple nodes instead. This will isolate the testsfrom each other, while still giving us speed gains from parallel execution.

The plugin reads the list of tests from the results archived in the previous execution of thisjob and splits that list into a specified number of sublists. I can then usethose sublists to execute the tests in parallel, passing a different sublist toeach node.

Let’s look at how this changes the pipeline:

LANG:code
node { /* ...unchanged... */ }

void runTests(def args) {
/* Request the test groupings. Based on previous test results. */
/* see https://wiki.jenkins-ci.org/display/JENKINS/Parallel+Test+Executor+Plugin and demo on github
/* Using arbitrary parallelism of 4 and "generateInclusions" feature added in v1.8. */
def splits = splitTests parallelism: [$class: 'CountDrivenParallelism', size: 4], generateInclusions: true

/* Create dictionary to hold set of parallel test executions. */
def testGroups = [:]

for (int i = 0; i < splits.size(); i++) {
def split = splits[i]

/* Loop over each record in splits to prepare the testGroups that we'll run in parallel. */
/* Split records returned from splitTests contain { includes: boolean, list: List }. */
/* includes = whether list specifies tests to include (true) or tests to exclude (false). */
/* list = list of tests for inclusion or exclusion. */
/* The list of inclusions is constructed based on results gathered from */
/* the previous successfully completed job. One additional record will exclude */
/* all known tests to run any tests not seen during the previous run. */
testGroups["split-${i}"] = { // example, "split3"
node {
checkout scm

/* Clean each test node to start. */
mvn 'clean'

def mavenInstall = 'install -DMaven.test.failure.ignore=true'

/* Write includesFile or excludesFile for tests. Split record provided by splitTests. */
/* Tell Maven to read the appropriate file. */
if (split.includes) {
writeFile file: "target/parallel-test-includes-${i}.txt", text: split.list.join("\n")
mavenInstall += " -Dsurefire.includesFile=target/parallel-test-includes-${i}.txt"
} else {
writeFile file: "target/parallel-test-excludes-${i}.txt", text: split.list.join("\n")
mavenInstall += " -Dsurefire.excludesFile=target/parallel-test-excludes-${i}.txt"
}

/* Call the Maven build with tests. */
mvn mavenInstall

/* Archive the test results */
step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml'])
}
}
}
parallel testGroups
}

/* Run Maven */
void mvn(def args) { /* ... */ }

That’s it! The change is significant but it is all encapsulated in this onemethod in the Jenkinsfile.

Great (ish) Success!

Here's the results for the new pipeline with parallel test execution:

The tests ran almost twice as fast, without changes outside pipeline. Great!

However, I used 4 test executors, so why am I not seeing a 4x? improvement.A quick review of the logs shows the problem: A small number of tests are taking upto 5 minutes each to complete! This is actually good news. It means that Ishould be able to see further improvement in pipeline throughput just by refactoringthose few long running tests into smaller parts.

Conclusion

While I would like to have seen closer to a 4x improvement to match to numberof executors, 2x is still perfectly respectable. If I were working on a group of projectswith similar pipelines, I'd be completely comfortable reusing these same changeson my other project and I'd expect to similar improvement without any disruption toother tools or processes.

Links

• Parallel Test Executor Plugin

This is the first in a series of posts showing ways to use Jenkins Pipeline. Follow along with this entire series through the links below:

• Faster Pipelines with the Parallel Test Executor Plugin
• Publishing HTML Reports in Pipeline
• Sending Notifications in Pipeline
• Browser-testing with Sauce OnDemand and Pipeline
• Continuous Security for Rails apps with Pipeline and Brakeman
• xUnit and Pipeline