The 5 Best Out of the Box A/B Testing Tools

So you've heard about A/B testing, eh? You feel ready to bypass rounds of "business analysis" and get information straight from your customers' behaviors? Here's the next step: finding a tool that will help you get started. In this post, I've laid out an array of tools that make it easy—and in some cases cheap—to get the ball rolling on A/B testing. These tools are some of the easiest to use right out of the box, and some contain some out-of-the-box ideas as well.

To ensure these tools are strong out of the box, I am measuring all tools based on the following criteria:

• Cost: What financial investment do you need to make to get value out of this tool?

• Platforms: Some tools are only good on a browser, some can go on a server, and some can be installed on phones. Which installations does the tool support?

• Ease of implementation: How easily can you embed an A/B test in your application code? How disruptive is it?

• Ease of administration: How easily can you monitor and change the variables in an A/B test?

• Additional notes/features: Not everything fits into one box. The tool may have extra features that should be called out, or perhaps it is part of a larger suite.

We start with Optimizely, one of the more well-known A/B testing tools. Optimizely focuses on keeping a low barrier of entry to A/B testing, in both cost and complexity. It specializes in landing page optimization but can support more general A/B experiments. Cost: Starts at $17/month, but cost scales with usage. This can easily grow to $2000/month for enterprise users. That said, it has a low cost of entry for experimentation. Platforms: Optimizely is focused on client testing, although it recently introduced support for mobile and server apps. Ease of implementation: Relatively easy. You need to add a snippet of JavaScript to your app. You can then add JavaScript selectors to the elements for which you want to create variations. Ease of administration: Again, relatively easy. You set a goal for the experiment, e.g., "I want 30K page views." Then you set the audience you want. Optimizely let's you deal with common variations, such as user agent, region, etc. You can also target specific URLs. Finally, you determine what percentage of traffic goes to the experiment until the goal is met. Additional notes/features. Optimizely is the baseline for a solid A/B testing tool. It's effective, simple, and cheap (up until a certain point). A great entry-level tool.

Visual Website Optimizer is a more advanced tool than Optimizely. It's not the cheapest tool out there, but it has some useful analytics features that make it top of its class for certain types of experiments, such as ad campaigns. Cost: Starts at $49/month, which can grow to $999/month for the enterprise testing plan. If you want their conversion optimization platform, the price spikes even higher. Not the easiest get-out-of-bed price of the options in this post. Platforms: Like most of these tools, VWO only supports client-side testing. Ease of implementation: VWO uses a visual editor to let you make changes. It's very easy to use, but the visual aspect means a lack of flexibility. Ease of administration: Similar to Optimizely, VWO has an intuitive dashboard that lets you segment your audience, set up experiments, and analyze their progress. Additional notes/features: VWO seems similar to Optimizely at first glance, but it has a few key differences. Perhaps the largest one is individual user session tracking, where you can start to monitor an individual user's interactions: gathering heat maps, looking at their navigation flow, etc. This can lead to powerful insights.

We've arrived at Yours Truly. CloudBees Feature Management.io offers A/B testing as one of its many features. We specialize in Feature Flag Management, which includes features that can be toggled based on audience for a set of experiments. We are also one of the few tools that supports a wide range of platforms. Cost: Differs based on usage, usually hovering somewhere in the double digits. Large organizations may find themselves spending over $100/month. You can also get started for free. Platforms: CloudBees Feature Management.io supports all three platforms: browsers, servers, and mobile phones. This includes a variety of server languages such as Java and .NET. Where we really shine is on mobile devices. CloudBees Feature Management.io can actually hot swap your code on mobile. Ease of implementation: CloudBees Feature Management.io is pretty easy to plug in. It's not visual like VWO. Because we support a variety of platforms, we use an approach that works across all of them. You make a container and add toggles as classes to that container. This guide on getting started shows just how easy it is to plug in your app. Ease of administration: As the guide shows, it's fairly easy to get an experiment up and running once you add the feature flag to your application. The UI is very flexible in how you can segment the audience and run the experiment. In some places, it favors flexibility over ease of use. Additional notes/features: As stated above, CloudBees Feature Management.io is one of the few solutions that can support you across multiple platforms and has first-class support for hot-swappable experiments on mobile devices.

Unbounce is another unique beast among the offerings here. It is solely focused on conversion optimization through landing pages. And in that area, it is unparalleled. Cost: Unbounce's pricing is reasonable at $79/month. The price goes up from there, but the Essential package can last you a long while. It's a high price for a limited feature set, but if you're doing a lot of digital marketing, this may be the tool for you. Platforms: Since Unbounce only deals with landing pages, it only supports client-side website management. Ease of implementation: You have to use their editor to make your landing pages, but Unbounce makes this a breeze. It has thousands of templates to choose from and is very flexible in how you can lay out your pages. You can even preview how Unbounce works before you buy. Ease of administration: Unbounce's administrative interface is simple to use. It lets you set up sites for various clients if you want to. Like its overall offering, the interface is very limited in terms of what you can do. Additional notes/features: Unbounce takes specialization to a new level. If you are into digital marketing and only care about conversion rates, this is the best tool for the job. But if you want your A/B testing software to suit any other purpose, this may be too costly an option.

Sitespect's claim to fame is that it is is middleware for your server; it lets your server collect data in ways most other tools cannot. It also provides a secure data collection experience. Cost: It's likely you won't be able to nail down a specific price from Sitespect unless you request a demo. It seems their pricing is more towards VWO's end of the spectrum. Platforms: They have a client-side package, but their cream of the crop is their server-side package, installed as middleware onto your application. Ease of implementation: Because of its server-side nature, implementation is more advanced and requires some level of technical knowledge. Ease of administration: Once everything is up and running, Sitespect's dashboard and analytics can be very intuitive, capturing all sorts of metrics about your experiments. Additional notes/features: If you need a tool to monitor what is happening on your server, you'd be hard-pressed to find a better one than Sitespect. If that's not your main priority, one of the alternative tools listed here may be a better fit.

This is just the tip of the iceberg when it comes to A/B testing options. Each option makes it easy to start A/B testing straight out of the box, and each one caters to different needs. We take pride in CloudBees Feature Management.io's ease of use and wide variety of platform support options, but it's not the only game in town.