JUC Session Blog Series: Robert Fach, JUC Europe

A Reproducible Build Environment with Jenkins

Robert Fach, TechniSat Digital GmbH

TechniSat develops and produces consumer and information technology products.

In this talk Robert introduced what build reproducibility is and explained how TechniSat has gone about achieving it.

What is binary reproducibility : same “inputs” should always produce the same outputs, today tomorrow, next month and in 15 years time! 15-20 years is needed for TechniSat to support the automotive industry.

TechniSat has a rare and unique constraint where the customer can dictate what modules a feature can impact, but a release contains all modules and they are all rebuilt and tested so you need to ensure unchanged modules are not impacted.

You need to identify and track everything that has influence on the input.

• Source code, toolchains and build system validation, and everything else….

The benefit of a reproducible build environment gives a new level of trust to the customer – that you are tracking things correctly to know what has gone into each build. Then you can support them in the future (so you can make a bug fix without bringing in any extra variability into the build)! It can also be used to find issues in the builds (random GUIDs created and embedded for the build can be detected as what should be binary identical and what shouldn't be).

Why is it hard?

Source code tracking: it is an easy and "bread and butter" method of managing sources (tags…), but what about if the source control system changes over time? (you need to make sure that the SCM stays compatible over time).

OS tracking: File system – large code base with 1000's of files – some File systems may not perform well, but changing file systems can change file ordering which can affect the build. Locale issues can affect the build as well (marcos based on __DATE__, __TIME__ etc..)

Compiler: Picking up a new version of the compiler for bug fixes may bring in new libraries or optimizations (branch prediction) that could change the binary. You need to know about anything based on heuristics in the compiler and the switches that control the features so you can disable them, since after the fact it can be too late! You can create a seed for any random generations (namespace mangling -frandom-seed)

Dealing with complexity & scale.

As you scale out and distribute the build, it needs to be tracked and controlled even more.

This adds a requirement for a “release manager,” a system that controls what, how and where (release specification). This system maps the requirements onto the Jenkins jobs, which use a special plugin to control the job configuration (to pass variables to source control, scripts etc.). The Jenkins job maps to a Jenkins agent.

For each release, the release manager creates a new release environment. This includes a brand new Jenkins controller configured with the agents that are required for the build. The agents are mapped onto infrastructure. The infrastructure is currently managed SQA Systems, artefact repository, KVM cluster (with Openstack coming soon) and individual KVM hosts.

After the release the infrastructure is archived (os tools Jenkins etc…). Also record the salt commands used). (provides one level of way to reproduce). The specification provides another way to recreate the environment (but it is not always reliable as something may have been missed).

To create new builds in a change, you can clone an archived set of infrastructure so Jenkins can show trend history.

Performance Lessons learned (a little bit random at the end of the talk).

• Use tmpfs inside VMs for fast random I/O file systems.

• Try to use nfs read-only cache to save network bandwidth

• Put Jenkins Workspace in a dedicated lvm in the host rather than network

To learn more, you can view Robert's slides and video from his talk.

We hope you enjoyed JUC Europe ! Here is the abstract for Robert's talk, "A Reproducible Build Environment with Jenkins." Here are the slides for his talk.

If you would like to attend JUC, there is one date left! Register for JUC U.S. West, September 2-3 .