Chapter 1: Infusing the culture of A/B testing

by James Young

You’ve likely read and heard a lot about A/B testing, and how it can help boost your marketing ROI. It can and does, but the companies who benefit the most from A/B testing have something in common.

Those companies have infused the culture of testing into their organizations. They make testing what their teams are all about.

Trust me, I’m a professional

Here’s how I came to this point of view — a little bit about who I am, where I’ve been, and why you can and should believe me. I started in the field of graphic design before the advent of the Internet and World Wide Web, and consider myself a user experience person who’s drawn to marketing opportunities.

As such, I have three passions:

1. Helping teams extract creativity and ideas from the deep spaces of their minds
2. Being an advocate for the customer/end user
3. Taking advantage of my experience in items 1 and 2 above to help companies make more money

No slide rule required

When I talk about A/B testing, it’s mainly from a design perspective, not a development/analysis perspective. So if the mere mention of “analytics” or “metrics” makes you break out in hives, don’t worry — I won’t be getting into details of math and technology.

I will say that A/B testing feels much more like a game to me than a brain-wrenching exercise. It proves there’s real ROI attached to my creativity and hard work. And, it feels great when I win. What’s not to like about that?

A brief history of testing

Keep in mind, A/B testing isn’t new. It’s older than the Web, and has its roots (so to speak) in dead-trees media (direct mail and magazine ads). There’s also testing in direct response TV and radio. But A/B testing on the Web is more fun than any of those.

Why? Because you can get results fast — sometimes even immediately. Which is perfect for impulsive, impatient, and generally A.D.D. people like myself (and like a great deal of the people reading this blog, I’d bet).

Getting hooked

I first got involved with A/B testing about eight years ago. I didn’t realize how powerful a tool it was (and is) for about a year, largely because it was hard to cut through all the analytics jargon. Once I understood that a 107% index (7% increase in conversion) could mean a gain of $20 million, I became a believer.

Everyone’s doing it

Now, of course, A/B testing is commonplace, especially for marketers, but even eight years ago it was revolutionary to have such a, well, “tangible” way to prove ROI. A/B testing is mainly used for marketing because marketing sites are relatively easy to set up and run tests on. Giant companies like, Google, and Netflix are among the few who test UX experiences   on Web applications and mobile apps — that’s because of the huge staffing resources such tests require.

Beware the pitfalls

By now you’re likely with me on the idea that A/B testing is a powerful tool for increasing conversion on the Web. But the mistake many organizations make is treating an A/B test like a one-off campaign. They’ll put lots of effort into a test one week and forget about it the next.

Or, companies will invest big money in software tools, but fail to invest in the team(s) that will be coming up with the very ideas to test.

Don’t think of A/B testing as a quick source of short-term gain. Rather, look at it as a long-term effort that will pay off with much greater rewards.

Do the right thing

I work with organizations large and small, and have seen how companies succeed (and fail) at it. They infuse the very concept of testing into their company culture. The company I started doing A/B testing with took this approach, and went from an under-a-billion-dollar company to the $4 billion company it is today. A/B testing ignited that growth and I am proud to have been a part of it.

Now, it’s your turn. Keep watching this blog for the tips and advice that will set you and your company up for incredible jumps in conversion and growth!

Ideas by James Young. Structure by Chuck Vadun.


See also:

Chapter 2.1: What exactly is A/B testing, anyway?

Chapter 2.2: What exactly is A/B testing, anyway? Part 2: Multivariate testing

Chapter 2.3: What exactly is A/B testing, anyway? Part 3: Lean Testing

Chapter 3: What to Test For

Chapter 4: What is a hypothesis, and why is it important?

 Chapter 5.1: Where do we start: Let’s just start with basic marketing

Chapter 5.2: Where do we start: Look to Your Research