Archives for November 26, 2013

Radical A/B Testing

Optimizely argues that A/B testing should be used to test radically different designs and that multi-variant testing should be used to test the combinations of smaller variations.

The main problem with A/B testing only small changes is that you don’t know what you’re leaving on the table. Small incremental changes most likely yield small incremental returns. But radical changes can yield radical returns. Changing a button color may get you a 3% increase, but changing your entire page may get you a 100% increase.

This article summarizes it well:

Button color tests. Font size tests. Headline tests. Run them all you want. But remember that your website’s conversion rate is only as limited as the risks you take. Small tweaks = small wins. If you’re craving for big wins, you’ll have to make big changes.

Many people argue that you should start with radical and move to incremental.

Here’s another argument for radical A/B testing. Here’s one from Moz’s Rand Fishkin, in which he advises, “Start with the big ideas and big projects, nail down the grand plans worth testing, let your audience pick a winner and then try to tweak, tune and improve.”

A few other articles and tools with checking out: