Why do we Test & Target?

by James Young

As I consult with various companies I have grown to see the value of A/B traffic testing. This is the ability to test one page or creative campaign over another and see, with actual data, which one gives you the best results based on your business goals. I have mostly worked on projects that have used systems such as Omniture’s Test & Target software, but there are many others out there.

Oftentimes we creative and marketing professionals think we already know what will move the needle to increase conversion or revenue. The solution seems obvious to us. Testing, however, can prove us wrong and produce a more effective outcome.

So say I redesign a website for a client that is based upon good user data and brilliant analysis of the user experience issues. Let’s launch it. It’s brilliant and will surely make a huge difference to the bottom line of the company. But wait a minute, how will I know if it is brilliant? I really cannot measure the improvement over last month’s site because things change seasonally in the marketplace. The competition is bigger or the product line is different or there is too much variability. I cannot truly understand the positive or negative results of the new site.

Only A/B testing gives us the ability to send a portion of the client’s traffic through the existing experience and then send another portion through the new experience. The results give us a statistically sound picture of how the new creative campaign is doing. We can then know exactly how much improvement is due to the new experience, and can extrapolate that out to dollar figures or conversion rates. Once a department has these numbers they can really crow about the value they are bringing to the company. This can allow them to grow into a powerhouse division that gets the respect it deserves.  As consultants, we can really prove the return on investment to the company for using our services.

Tests that show a decrease in conversion or revenue, while not what we ever work towards, are just as valuable. A/B testing can show us how big of a bullet we just dodged. These results are disappointing, but I like to remind the client, “We just avoided a X-million dollar mistake.” I encourage them that we can learn from the loss and keep moving ahead with testing new ideas.

In my earlier career, before I began the practice of A/B testing, I really felt that even without data, qualitative or quantitative, I could figure out what was best for the user. Testing has humbled me quite a bit. Designs that I thought were so smart and “sure to win” failed or were flat. Conversely, some of worst ideas have really moved the needle. When we are discussing which recipes to test or not to test, my new mantra is, “Well… dumber things have won…” Just test it.

All this is to say that we creatives finally have objective, reliable tools at our disposal to help us produce the best experience for our users and the greatest improvement to the bottom line for our clients. A/B testing takes the guesswork out of the process.