Dependency Cache and Sphinx Support on Codeship

This week we are happy to announce a big change that we released two weeks ago on Codeship: Our new Dependency Cache.

Why cache dependencies

For every build we provide a clean virtual machine. The advantage is that our customers can be sure that their tests run under the same conditions every time. Any change done during the build is automatically removed from our system after the build finishes. This makes installing dependencies slow as we need to install them for every build.

We pre-installed thousands of Ruby Gems in the past, but the dependency installation still took a considerable amount of time for every build. This was not just a bad experience for all of our users, but also used a lot of our server’s resources. Furthermore we wanted to speed up dependency installation for other languages we support.

Whenever a build finishes now, we store a tar archive of all dependencies. For the next build we extract that same archive back into the virtual machine. This happens completely transparent. Only when you add a dependency it needs to be installed. The next time it will be used from the cache.

The difference is enormous. The average bundle install to install Ruby Gems took 220 seconds on average in the week before we released the cache. It went down to 10 seconds on average.

Installing pip packages for python went down from 97 seconds on average to 12 seconds. And the same goes for other languages as well. For our customers this means up to double the speed for their build and they are loving it.

Customer statements about our new Dependency Cache

Mixcloud"FYI loving the dependency cache, so so so much better!"Mat Clayton - Co Founder & CTO

Ambassador"We're amazed at how fast our builds are now. Fetching dependencies used to take minutes and slowed us down as we waited for feedback on our commits. Thanks to Codeship's dependency caching it takes seconds, and we can get right back to work."Chase Lee - CTO

How the Dependency Cache works

We store the tar archives on S3. We make sure they aren’t broken by checking their MD5 sums whenever we upload or download an archive. If the download fails, we retry. If we cannot restore the cache several times, we run the build without the cache. This will slow down the build, but make sure that it can proceed without interruption.

The cache was in beta for a couple of weeks. We pushed it into production two weeks ago and it works very well.

Language support

Ruby

We overwrite the default path bundler installs gems into and cache that folder.

Python

All dependencies installed through pip install are cached.

Node.js

All packages installed through npm install are cached.

PHP

PHP is still giving us headaches on how to cache dependencies properly. We have been trying out various ways, but so far we haven’t found a method that works for all of our customers. We are working on this and will ship an update soon.

Java

We are currently caching the ~/.m2/repository folder for dependencies installed through Maven.

Sphinx full text search engine integration

Sphinx 2.0.4 runs for every build on the default port.

ZeroMQ support

Libraries for ZeroMQ 2.2.2 are now installed on our system.

Conclusions

With our new Dependency Cache you will get even faster feedback for your builds. Give it a try and see how Codeship can speed up your build and deploy times.