Chapter 2.2: What exactly is A/B testing, anyway? Part 2: Multivariate testing

by James Young

Chances are, you’ve heard (or even used) the term “multivariate testing.” It sounds a little scientific and scary — but there’s nothing to fear, and much to gain from it.

So what is it? Multivariate testing is a process by which more than one component of a website may be tested in a live environment. Translation: in an A/B test, you can find out what happens when you change one variable. Multivariate testing lets you look at several variables at once.

Here’s where the magic happens

In multivariate testing, your testing software provides you with an algorithm that lets you run traffic through a variable set of elements on a page.

 In the example below, we’re testing four different variables at once:

1. Headline
2. Images
3. Side offers
4. Body copy


Shown above is a test with three variations of four components on the page. 

Ideally, when the test is complete, you’ll know the combination of variables that produces the best conversion. From there, you can choose to roll out with that combination of variables as your control. (Or if it’s already the control, come up with more ideas for your next multivariate test!)


Above: the winning combination!

Some websites benefit from continuous optimization, as responses to creative approaches and layouts may differ by time of day/week or even season.

Not for the faint of heart

Putting together and running multivariate test is more involved than doing a simple A/B test. Sometimes unknown factors can affect results, and it takes longer to get to statistical significance (that is, to run enough visitors through the test to make it valid). But don’t let words like “unknown factors” and “statistical significance” dissuade you; just be sure you’re working with an analyst (or analytical team) that’s “been there, done that.” The potential rewards are well worth the time and effort!

Still more to come

Next time: lean testing. For now, I’ll just say, “believe the hype!”

Ideas by James Young. Structure by Chuck Vadun.


See also:

Chapter 1: Infusing the culture of A/B testing

Chapter 2.1: What exactly is A/B testing, anyway?

Chapter 2.3: What exactly is A/B testing, anyway? Part 3: Lean Testing

Chapter 3: What to Test For

Chapter 4: What is a hypothesis, and why is it important?

Chapter 5.1: Where do we start: Let’s just start with basic marketing

Chapter 5.2: Where do we start: Look to Your Research