SSL vulnerability fix roll out and platforms vs servers

Some time in the last day a particularly nasty bug around SSL was dropped on the world - called Heartbleed . This had the potential to do a lot of nasty things but what troubled me most of all was someone took the time to craft a logo for it:

I can't be the only one to find it odd that a fresh dangerous exploit can have a nice website and logo so quickly - a conspiracy? (joking).

The vulnerability affects many recent versions of the openssl library on linux - all but the latest (basically) - and leaves SSL open to possible snooping of private memory (you can read more here ).

In any case - we tried to jump on it as quickly as we could - we had some services that were quickly patched - but the vast majority of our platform is users - who have many many SSL services (and servers) subscribed. Our approach was to treat it as an upgrade: rather than patching in place, we bring up new servers, check their health, and then flip over the IP address to ensure continuity (once we are sure, can shut the old ones down). Here is me explaining it with a bad drawing:

What a rubbish drawing, that doesn't help at all. Move along.

This is all heavily automated - and somewhat routine - but the scale of doing a supervised roll out like this in a hurry was a little amusing. We came up with a fictional drinking game (no actual drinks were drunk) - you would watch 50 servers start, boot, and flip over - and you have a drink**.

For systems that have a relatively static revproxy layer - upgrades like this should be pretty easy (but you will notice the rest of the world is taking its time to update things) - this was mostly the case for us - except for our beloved users.

But this is what a platform, or a service, is all about. While you were sleeping (or not) - we go ahead and do these things so you don't have to (and hopefully don't notice).

Current status:

• US & EU RUN@cloud service has been fixed.

• Console and API: fixed.

• Repositories and Nexus services: fixed

• Build Services: Pending on ELB upgrade.

We use Amazon ELB for some services - which as at the time of writing still was having issues (you can track this here ). There are reports of a roll out of fixes that some people are seeing, so expect that fixed very soon.

Someone helpfully built an app that can check the vulnerability of any websit e*** - you might want to try this on services you use day to day yourself - contact them if you are concerned:

Here is a good result:

here is a bad:

I hid the true identity of this other site. I expect they are waiting for ELB to be patched - vs servers they control directly themselves.

Now, what you may need to do if you subscribe to our SSL service:

As mentioned in the bug write up - there is a possibility that a private key for your SSL site was exposed - and may mean your SSL cert needs to be revoked. Unfortunately we can't manage that bit for you - you will need to go back to the provider that sold you the SSL cert - and possibly revoke the old - and get it refreshed - the process you then go through to update your SSL service with the new one is fairly simple from that point (sorry, we can't do this for you, as SSL certs are tied to identity - they belong to *you* and identity *you*).

We will be sending out more notes on this in the coming days to those affected via email, with instructions (and we will help you out).

** No actual braincells were harmed in the making of this update.
*** Troublingly even my own internet banking is affected.