Immutable Servers, FTW!

— Prathap G. Dendi (@pDendi) March 17, 2015

A couple of weeks ago I got to attend - and present at - my very first SXSW . And man, what a trip.

St. Patrick's Day Street Party at SWSX
On St. Patrick's day, I was also rocking it with the masses. sort of.. as I got to give my first talk on Immutable Servers to a room full of developers, who were also trying to balance the celebrations with learning something new. It was fun.
My presentation was around how we've adopted Immutable Servers in our development and QA environments at CloudBees as a helpful pattern to software delivery. I covered some free open source tools we use as part of our process: Vagrant, Packer and Docker. I also discussed the benefits as well as the technical and cultural challenges when implementing these across a globally distributed team.

What is Immutable?

The Dark Days Before Immutable Servers

I started the talk with our development practices around 2012-2013. Back then, Engineering all fit in one room and everyone had the same schedule. As the company grew we slowly found out that the practices and processes we had in place would not be able to grow as we did. This became painfully evident as we added developers and QA engineers in different locations. One aspect we took a look at was our tooling and how it could match our growth.

Partly Cloudy Days: Adopting Better Tooling (~2013)

This lead us to investigate Vagrant . We migrated our wiki page instructions to Vagrantfiles that we checked into source control.
We found the Vagrantfile to be the best form of documentation for setting up development environments for globally distributed teams. The key advantage is that the Vagrantfiles are actively exercised by developers and in CI which has a tendency to keep them up to date. Wiki pages do not have that characteristic.

After using Vagrant to model multiple machines, we started to monitor and record start-up times. The next problem we tackled was how to cut down the time spent starting and configuring all of the machines.
For this, we turned to Packer and started creating Vagrant boxes. This cut our time in half and we were thrilled. At the same time, the blueprints for constructing the boxes was version controlled and was a repeatable process.

Right around the same time, Docker started becoming popular and showed incredibly fast start up times. Whereas we measured in minutes, Docker containers could be started in milliseconds.

No DevOps session is complete without the picture of a puppy @therealnikhil #sxsw #ContinuousDelivery pic.twitter.com/uTWNwRtOEg — Prathap G. Dendi (@pDendi) March 17, 2015

We decided to dip our toes by creating containers for 3rd party tools with which we needed to integrate. The example I showed was a Perforce Server inside a container.

Getting everyone on board:

I definitely faced some hurdles trying to convince coworkers to use Vagrant. A lot of people were skeptical and could not see the immediate value.

Gathering consensus and persuading an organization to adopt a new tool or solution is difficult. My suggestion is to avoid chasing new technology for the sake of new technology. Instead try to identify a problem, formulate a hypothesis and then look for a tool/solution.

A warning: even if you have followed the problem solving steps people may still feel like you’re just chasing new technology. But if the ‘new way’ really does solve a problem you will see gradual adoption – at least that was my experience.

TA-DAAAA!!

Sequentially starting our 6 nodes cluster initially took 53 minutes. At the time Vagrant did not support parallel ‘vagrant up’ so we could not use it. After creating images ahead of time with Packer, we were able to cut that time to 30 minutes.

These open source tools have helped our development team stay on the same page and focused on figuring out ways to make our product better instead of troubleshooting the dreaded “worked on my machine” problem.

After spending a day troubleshooting configuration issues- we used to feel like this:

Now, we get to code more, be more agile, and worry less about all the annoying things that make us freak out - like documentation, setting up of the infrastructure, configuration drift, etc.

If you want to share your experiences around Immutable Servers or have any questions feel free to leave a comment or contact me @therealnikhil .

Besides Immutable Servers, there was more fun to be had at SXSW

Two talks really stood out for me. The first was @thinkmariya 's talk about Rapid Iteration on Mobile . After listening to her talk I realized I've made the mistake of starting to think about the UI for a mobile app before addressing some deeper design questions. Namely, what is the feedback loop that keeps users engaged and returning to the app.

The second was How to Build 3 Michelin Star Restaurants + Make a Doc . It was a panel discussion on the documentary, For Grace , which you can think of as a cross between Jiro Dreams of Sushi and The Hundred-Foot Journey. I was inspired by how Chef Duffy and Michael Muser have broken down the overall dining experience into small chunks that they then painstakingly optimize - with no detail being too small: from the way a tablecloth is laid down to how ingredients are sourced. Passion and extreme attention to detail helped them attain their dream of 3 Michelin stars.