Chapter 2.3: What exactly is A/B testing, anyway? Part 3: Lean Testing

by James Young

Here’s a testing concept I just love: lean testing. It’s perfect for testing the idea of a product or service offering, without have to actually FINISH the product or service offering. Think of it this way: before you spend time and money getting an offering to market, wouldn’t it be great to see if anyone would even be interested in it?

Send it up the flagpole

Lean testing came into vogue with Erik Ries’ book, “The Lean Startup.” Ries writes that by releasing what’s called a “minimum viable product,” you can gather customer feedback that will help you improve your offering — by tailoring it to better meet the needs of your target customer.

So, you can use lean testing to assess what your customers demand — and then determine how to meet that demand using the least amount of resources possible.

What’s lean got to do with it?

How do lean testing and A/B testing work together? Say you want to test an offering that, for whatever reason, isn’t ready for prime-time. Maybe you don’t have the resources or the time to build out the full offering, but you’d still like to know if the concept of it has legs. No problem — create an A/B test recipe as if you actually had the offering. Then put it up against your control version, and see what kind of a response you get.

“Vet” your idea

Let’s say we wanted to test the idea of providing free vet care with a pet adoption, before we were actually able to do so. We could get an idea of how much “lift” it would provide, and determine from there if we wanted to continue putting together the offering.


Creating the offering shown in Recipe B could take a lot of time, effort, and money — and might not even get the results you want. Why not do a quick A/B test to find out first?

Here’s how you handle it, if you’re not ready to deliver on the promise you just made in Recipe B: if a visitor clicks to learn more, they’ll see a message saying that the offering isn’t yet available. You can even give the visitor the opportunity to provide their email address, so you can notify them when (if) you make the offering available.

Lean_popup But now you will have a solid understanding of what works and what won’t saving your organization a lot of time and money finding out. Lean_results

So now you know: the offer has some legs (four of them, like our furry friend?). You can proceed with developing the whole program. If Recipe B had been a flop, you’d have lost very little in the grand scheme of things.

Danger, Will Robinson …

There’s a caveat to testing like this. My writer pal Chuck Vadun says, rather bluntly, “It’s very likely you’ll piss people off.” This is because you’re putting an offer out there, which you have no ability to fulfill, right at the moment when someone wants to jump on it.

… Quid pro quo, Clarice

There are ways to mitigate this possible hazard, including a) testing it with a very small amount of your web traffic, and b) offering something of value to the visitor who shows interest — a discount code for one of your current products, for example.

Whatever you provide in return, it’s a small price to pay for information you can’t get any other way: in-market testing of interest in your offering. Focus groups and customer interviews may give you confidence in what you’re planning to offer, but there’s nothing like seeing how many people with vote their interest with a tap or a mouse-click.

Ideas by James Young. Structure by Chuck Vadun.

See also:

Chapter 1: Infusing the culture of A/B testing

Chapter 2.1: What exactly is A/B testing, anyway?

Chapter 2.2: What exactly is A/B testing, anyway? Part 2: Multivariate testing

Chapter 3: What to Test For

Chapter 4: What is a hypothesis, and why is it important?

Chapter 5.1: Where do we start: Let’s just start with basic marketing

Chapter 5.2: Where do we start: Look to Your Research