Why Smart, Efficient Backup and Restore Techniques Are Essential with Jenkins Production Server

Tips from Jenkins/Hudson Founder Kohsuke Kawaguchi:

If you’re like me or other typical folks out there, you’ve probably been postponing backups because you have more important things to worry about. But as you surely know, it’s very important to have a backup, and better late than never!

In addition to disaster recovery, backups are useful insurance against accidental configuration changes, which might be discovered long after they were made. A regular backup system lets you go back in time to find the correct settings. A key tip to ensure reliable, optimized production operation with Jenkins is to make sure you keep up on backups. Even if you are already running Jenkins, it’s not too late to start taking backups.

First, let’s look at backup planning.

Jenkins stores everything under the Jenkins Home directory, $JENKINS_HOME (to find the $JENKINS_HOME location, go to the Configure System menu), so the easiest way to back it up is to simply back up the entire $JENKINS_HOME directory. Even if you have a distributed Jenkins setup, you do not need to back up anything on the slave side.

Another backup planning issue is whether to do backups on live instances without taking Jenkins offline. Fortunately, Jenkins is designed so that doing a live backup works fine – configuration changes are atomic, so backups can be done without affecting a running instance.

Now, let’s look at how you can optimize backups.

Optimization 1: Back up a subset of $JENKINS_HOME
Although $JENKINS_HOME is the only directory you need to back up, there’s a catch: this directory can become rather large. To save space, consider what parts of this directory you really need to back up and back them up selectively.

The bulk of your data, including your job configuration and past filed records, lives in the /jobs directory. The /jobs directory holds information pertaining to all the jobs you create in Jenkins. Its directory structure looks like this:

/jobs/*

- builds (build records)

- builds/*/archive (archived artifacts)

- workspace (checked out workspace)

The /builds directory stores past build records. So if you’re interested in configuration only, don’t back up the builds. Or perhaps you need to keep build records but can afford to throw away archived artifacts (which are actually produced binaries). You can do this excluding builds/*/archive; note that these artifacts can be pretty big, excluding them may introduce a substantial savings.

Note that the following directories contain bits that can be easily recreated, so you don’t need to include these in the backup:

- /war (exploded war)

- /cache (downloaded tools)

- /tools (extracted tools)

Finally, the workspace directory contains the files that you check out for the version control systems. Normally these directories can be safely thrown away. If you need to recover, Jenkins can always perform a clean checkout, so there’s usually no need to back up your workspace.

Optimization 2: Use OS-level Snapshots
If you want maximum consistency in your backups, use the snapshot capability in your file system. Although you can take live backups, they take a long time to run, so you run the risk of taking different data at different time points… which may or may not be a real concern.
Snapshots solve this problem. Many file systems let you take snapshots, including Linux Logical Volume Manager (LVM) and Solaris ZFS (which also lets you take incremental backups). Some separate storage devices also let you create snapshots at the storage level.

Now, let’s test and restore.

Nothing is worse than thinking you have a backup and then when disaster hits, finding out you can’t actually recover. So it’s worth testing to make sure you have a proper backup.

The JENKINS_HOME directory is “relocate-able” – meaning you can extract it anywhere and it still works. Here’s the easiest way to test a restoration:

- Copy the backup Home directory somewhere on your machine, such as ~/backup_test

- Set JENKINS_HOME as an environment property and point to backup_test; for example, export JENKINS_HOME=~/backup_test

- Run java -jar jenkins.war —httpPort=9999

This sequence of commands will pick up the new JENKINS_HOME with the backup_test directory. You can use this instance of Jenkins to make sure your backup works. Be sure to specify a random HTTP port so you don’t collide with the real one – otherwise the server won’t start!

While Jenkins is not difficult to set up or configure, you will get better results, support more projects and save administration time if you know the tips, tricks and optimal settings that can make your installation function most effectively.

This is just one tip, but I share several more in our article, “7 Ways to Optimize Jenkins”. If you prefer slides and my voice, we also have the recorded webinar on the CloudBees resource page (scroll down to bottom), as well as a list of the top questions asked by attendees during the webinar. I’ll also be in the San Francisco Bay area on April 5th running a training course for those who want to master Jenkins.

If there are other topics related to Jenkins that you’d like me to address, feel free to leave a comment!