The Trouble with A/B Split Testing

Background

Split testing is very popular right now, and many people have the perception is that without hardly lifting a finger a company can achieve game-changing lift. This one post alone sent scores of designers and marketing folks running to change the text on their signup buttons, and in many cases pursue split tests of their own.

For a few more common and interesting data point on the space of split testing, check out this Doug Bowman post, this Jason Putorti post (and some good commentary) and peruse some Jonathan Mendez posts.

Common Sense

Reality check folks: you are not going to buy that house in Cabo next month because you changed the text on your sign up button. Sorry, it’s just not going to happen.

There is no such thing as a free lunch. There are no silver bullets in marketing.

If you were a restaurant, would you try to get more diners by making the front door physically wider? The same kind of thinking is at play here, and it’s nonsense.

What would you do to get more people to eat at your restaurant? You’d cook really good food. You’d try to serve food that is not easily ordered in that neighborhood. You’d be really kind to your customers so they would write you up positively on yelp and refer their friends. Think: Seth Godin with a chef’s hat.

Devil’s Advocate

“Ok wiseguy, this is the internet, this isn’t a restaurant. If restaurants were run online, you could publish different menus, and find out what dishes and cuisines will be popular by testing menus against each other. What about that, huh?”

Yeah, but …

“What if there is another Chinese food restaurant next door already, and your food sucks?”

Menu testing would be a waste of time.

Real Challenge #1: What works for Company A might not work for Company B

Example: Someone emailed our team the other day and asked us to make the search box on our site twice as big because there was a Marketing Sherpa article about how doing this raised revenue 68% overnight. Smells like bullshit; my guess is that the test subject’s site architecture stinks and people can’t browse to anything they are looking for and the search box they had probably was tiny. I’m happy with our search box thank you very much, it’s lovely.

Real Challenge #2: Upstream improvement does not equal downstream results

This is a huge one. Example: split testing messaging on a free-download registration page to optimize registration by playing up the ‘Free’ aspect of the offer. It’s possible that this would raise registration and hurt eventual conversion if you cannibalize the value of the premium product in your messaging. I’ve seen this happen and I have hard evidence from using professional split testing tools.

Real Challenge #3: People change

That terrific holiday messaging you came up with doesn’t do that well in the Spring. Not a huge surprise.

Things to watch for: seasonality, interaction of various campaign efforts, deltas in the competitive landscape.

Real Challenge #4: Traffic is limited

You’ll see below that one recommendation I make is to focus on downstream testing. One challenge here is simply that you have far fewer visitors at the bottom of the funnel than at the top! If you are performing split tests on a shopping cart, you might have dozens of daily participants as opposed to thousands, for example.

Split Testing is a Powerful Tool

When used correctly, split tests can put you on the scent, and ultimately lead you to game-changing business realizations. Here are a few stakes in the ground, advice that might help point testing efforts in the right direction.

Good Idea #1: Start at the Cart

In a world where it is very difficult to get people to take out their wallets, start by talking to those who already have.

At the top of the funnel, you are by definition dealing with a population that will mostly not end up purchasing your product or service. As stated above, optimizing early-step funnel metrics is extremely tempting, but not the right idea. What is your cart abandon rate? 50%? 30%? Start there. This is the population that was interested enough in your product and offer to begin the checkout process.

If you can reduce cart abandon rates by 50%, you can raise your revenues by 50%. And this absolutely is overnight, bottom-line lift. Get pumped.

Good Idea #2: Product

Are you in the early stages of a startup? Are you considering a new product line or product configuration? Split test the ^$%@ out of the ideas you have! For results to be really meaningful, remember that you should come as close to booking an order as possible; if the user really isn’t ready to swipe a credit card, how real is that conversion?

Good Idea #3: Offer

Test discounts and special offers against each other. Look to net revenue or NPV to determine the winner, of course (as opposed to units sold).

Good Idea #4: Population

This one is less about landing page split testing, but is worth mentioning: focus on where you get your converting traffic. The perfect product isn’t perfect unless it fits the needs and worldview of the people in front of whom you are expending effort to place your product. #duh.

Comparing behavior, reaction and conversion of various populations is an extremely worthwhile activity. Top-of-the-line tools like Performable show conversion reports by referring source; this is huge, take advantage of it, and don’t bother using a tool that doesn’t provide this insight.

Good Idea #5: Run Each Test for a Minimum of a Week

You know that weekly ‘roll’ in your analytics, the weekly ebb and flow of traffic volume? That same ebb and flow exists for behaviors other than visit volume, and likewise tends to normalize out when comparing weeks to weeks. Don’t run a test for less than a week, and then roughly speaking your tests will be comparable to each other.

Bonus: Whenever possible look at previous-year benchmarks, and remember to adjust for day-of-week as well for daily metric.

Conclusion: Find Your Resonant Frequencies

To be honest, there is some value in split testing button text, and split testing marketing copy, and bullet lists and layouts and all the other obvious presentation stuff you can think of. Think for a minute about how Mint.com might have gained huge business value from split testing … it really was about finding product-market fit. Do people want what we offer? Why? Which people? What gets them excited to buy it?

Adhere to a disciplined approach that focuses on answering these questions, and split testing will be your best friend.