Choosing a Deployment Strategy: A Manager's Guide

Some time ago, I was in charge of deploying to the main system in the company I was working at. Back in those days, I had the bad habit of doing deployments manually by copying/pasting assemblies and then RDPing into the servers to update it with the latest changes. One. Server. At. A. Time. Luckily for me, that system only needed four servers. Those were bad days because something always went wrong---files were in the incorrect folder, configuration values or file permissions were missing, or I was so tired that I forgot to update one of the servers. Things are way different now; there are more good days than bad days. For example, what I call a "bad" day now is because of something as simple as a missing test case. Now that we live in the DevOps era, deployments are also different in the industry. Some organizations do several deployments per day. There are some strategies to increase the frequency of deployments that help to reduce the risks of downtime without customers noticing that something changed.

The blue/green deployment strategy consists of having two production versions of a particular system running at the same time. The catch is that only one is receiving live traffic. This means that the new version doesn't necessarily have to be backward compatible because the blue and green deployments won't be live at the same time. It represents a challenge for app dependencies like databases, but you can use the blue/green strategy for them too. Let's say for example that your blue side is the running version. The new version with the changes you want to deploy then is the green side. Because the green side isn't receiving live traffic yet, you have the chance to perform real testing in a production environment before pushing it live. When the green side is ready to be released, you just need to change the router that's pointing to the blue side so that it points to the green side instead. If something goes wrong for some reason, you can just switch the router back and diagnose what went wrong calmly. Rollback speed depends on how quickly you can switch versions, but it's usually fast. The downside is that you're duplicating infrastructure, which has an impact on cost---although this might get reduced with containers. But the more important thing is that you're reducing risk by having a production version that you can test before deploying to real users.

The rolling update strategy consists of pushing the new version little by little. You won't be replacing everything at once, as you do with blue/green deployments, so backward compatibility is a must. This challenges developers because they need to make sure that the new version can keep working with the previous version for some period of time. But this will also give them a sense of how things actually work in production and, as a bonus, it fosters the habit of practicing deployments early in the workflow---which will shift deployments to the left. Similar to the blue/green strategy, you start by spinning up the new version first when it's ready to go live. Then, connections from the now-previous version start draining to be rolled out gracefully. The cool thing about this is that you do deployments gently. You also get to define whether you want to spin up and replace one or several instances at a time. The downside is that you aren't able to run a set of tests on the new version before going live. Or rather, it's not that you can't, but it will make the deployment slower and more complex. Before deployment finishes, you might see some problems with the new version. It will take some time to do rollbacks depending on how far along the update is. (It's usually slow because you need to spin up new instances of the previous version again and then do the rotation.) An alternative that will help you reduce deployment and rollback time is to use containers---as long as there's enough capacity do the rollout. The benefit of this method is that costs stay low because you're just creating copies of the new version in small portions, not a full copy every time you deploy.

Canary deployments make the new version live for just a small fraction of the users. Technically, you decide which users will see the new version first, and you can always change the decision of which users to have in your test group. It's usually as simple as redirecting a small portion of user traffic to the new version and the rest to the current version. When deployment is finished, both versions will be the same. But the good thing is that you get the chance to test changes with a portion of the real traffic before a full rollout. Deployment could be done either with a blue/green deployment or a rolling update, as long as you only release the new version to a control group of users at a time. You also define how you want to split the traffic. It could be 5/95, meaning that, initially, only five percent of users will see the change. You could also use more than two groups, like 5/15/80. Or you can even start with a group of beta testers who are willing to look for problems and provide feedback. You can keep the new version for some time (minutes, hours, or even days) to see how it goes when receiving real traffic. Again, it depends on how you want to do it. If nothing goes wrong, you can continue by making the new version available to the rest of the users. Or you can roll back to the previous version. The good thing is that only the few users who tested the new version will notice it. The benefits and risks, rollback time, and costs (besides the ones that I just mentioned) will depend on whether you choose the blue/green or rolling update strategy.

A/B testing is similar to the canary release, but how you split the traffic is a business decision and is usually more complex. This strategy is commonly used for experimentation---when you want to try something that you're not sure will work. You define groups based on user preferences, location, age, gender, or other factors that will tell you which version had the biggest positive impact on revenue. You can then decide if A or B is ready for all your users. With this deployment strategy, two versions of the system compete with each other. It could be new vs old or A vs B. Whoever is responsible for making executive decisions in your company could choose which version converted the user to a customer the most, and that's the version you keep. For that reason, it's important that you have data to support your theory rather than make a decision simply because you think a certain version will work better. Implementing this strategy is usually complex, but if you make use of feature flags, it gets simpler. You can be as creative as you want when testing your hypothesis. For example, you might start by dark launching using any of the previous strategies. Then, you'd turn on the feature flag to make only version A available. After some time, you'd turn it off and then turn on version B for the same amount of time as A. From there, you'd collect metrics and decide which version to leave running.

It doesn't matter much which strategy (or a mix of strategies) you choose for your deployments. It depends on the outcomes you want. The key thing here is that you stop doing deployments manually and you have as little human interaction as possible. You also need to stay consistent, meaning that you'll have to apply the chosen deployment strategy for development and any other environment in your workflow. It might be challenging to have production-like environments, but you need to practice deployment before updating the environment where your real users interact. Don't treat your production environment any differently. Every strategy has its trade-offs. But one benefit common to all of them is that, if you implement the strategy well, you'll have zero downtime. You won't have to wait for specific times when your users won't notice that you're changing something. Now, it won't be easy to get to that point; your staff will have to put in some extra effort to achieve it. But they'll appreciate not having deployments cut into their nights, weekends, or family time. Last but not least, practice doing deployments frequently. If it hurts, do it more.