Hi-speed Linux Builds for DEV@cloud

buggati.jpgToday, we’re proud to announce the general availability of our next generation of Linux build machines for DEV@cloud. These new build machines come online faster, provide higher build speeds and set the stage for advanced capabilities moving forward.

We support two speeds: standard & hi-speed. The standard speed is designed to provide performance equivalent to our previous m1.large. Hi-speed is designed to provide 2x the performance of the standard speed (or more), depending on how well your build takes advantage of multiple cores.

In the graph below, we compare the build speeds of three different projects. Jenkins is a relatively light-weight Maven project, Asgard is a medium-sized Grails project, and OpenJDK is a relatively large C++ project.

                                                                                      Smokin!

The Container Era Is Here

We’ve been using LXC in production at CloudBees for the past three years (i.e., before it was cool).  The big change here is that we’re now taking huge amounts of CPU, network bandwidth and disks and splitting them up among several simultaneous users. Not only does LXC isolate the network, processes and filesystems between each user, it allow us to do cool things such as dynamically increase the amount of CPU available to each tenant.

All of this means that you get a high-speed virtual machine which boots and connects in just a few seconds. And yes, this extensive isolation means that things like VPN access are only the beginning.

The other part of this equation is ZFS — an advanced filesystem originally developed by Sun Microsystems. ZFS makes your build fast in several ways. First, we can throw a bunch of disks at it, and ZFS will pool them together so that your writes are spread across the fastest devices — meaning that writes almost never block. Second, ZFS aggressively caches your data for reading, both in memory and on special cache devices. Because we give ZFS some juicy Flash devices (100’s of GB) for its second level cache, your build can avoid hitting a spinning disk more often than not.

«A COW

ZFS is a Copy On Write (COW) filesystem. This means that changes to the filesystem are always written to newly allocated blocks. As a result, ZFS has all the data it needs to make a instantaneous snapshots — a feature we use to quickly save off your data when a build completes. Another advantage of COW is that we can provide write access to base images — for instance, the Android SDK — and every user can customize it as needed, while we are only storing the “diff.”

This is the same technology behind our recently announced OSX build machines, and much of the technical benefits mentioned here apply equally well to OSX. Plus, this cross platform support is really exciting —- guess which platform we are considering next!  

There’s a lot of potential here, including advanced features such as image customization. Going forward, you should expect to see us use cloud-stuff and Jenkins-stuff to generally make your world more awesome.

How to Upgrade

Hint: You only need to do this if you’re already a customer, and can’t wait to use this stuff. New and existing free users are automatically opted-in! You can also just sign up for DEV@cloud.

If you have an existing DEV@cloud account, you can upgrade now by installing the “CloudBees Cloud Connector” plugin in your Jenkins Update Center.

 

If you want to try your job on hi-speed, just edit the job and tie it to the hi-speed label.

These new build machines come in sizes of ‘hi-speed’ and ‘standard.’ While new customers will use hi-speed by default, existing customers keep the existing default of small*. You can change this default by clicking on the “DEV@cloud Slave Templates” on your Jenkins menu, and click on tool (configure) icon next to the “DEV@cloud Standard Linux” template.

 

* small size only available to existing customers.

On this page, you can upgrade your default to “hi-speed” or “standard”. Click here to learn more about the difference between these. To get the most out of the hi-speed executors, you’ll want to follow these tips on increasing your build performance.

These new Linux images are designed to be backwards compatible with our old ones in every respect. If you see any problems, please let us know. If you don’t choose to upgrade now, your account will be automatically upgraded in the coming days and weeks.

 

Ryan Campbell
Engineering Manager
CloudBees

 

Blog Categories: 

Comments

Zfs and not btrfs, that was surprising.

Joel - yes that is interesting. Hopefully the thoughts around ZFS specifics are worth a follow up blog!

A COW.

Yes, a cow. Beef.

Add new comment