markitechture

Why Test? Because You’re Wrong.

A couple of months ago we started a split test that I was extremely excited about. We had been testing basic hypotheses by removing top-level navigation, adding/subtracting Learn More buttons, as well as running some of the other plays in the basic landing page optimization playbook. This new test was to be a departure, a redesign of the site chrome as well as some of the page elements.

Our design team dug in and took a week putting together a new chrome just for that page, some new icons to represent the product features (which we really thought was missing from the current design), and an absolutely drop-dead sexy call to action button.

Frankly, we were so sick of the old page we could hardly bear to look at it.

Guess what.

The old page won by a mile.

The Test

Here is the original page:

Here is the challenger:

The Results

The new page lost the test by 14%! What about the sample size, you ask? We put about 30,000 uniques across the two branches, and there was way more than enough data for the last step of the funnel to call this conclusive.

The Takeaway

So we dug into the results of course, to try to figure out what may have happened. At first we thought it could have been the position of the call to action button on the page; was the new button too far below the fold? We used Google’s BrowserSize to check:

Geez, well if anything it was actually higher up on the page! So what could the answer be?

One thought that stands to reason is that the rest of the marketing funnel, all the other pages that the potential customer sees, are more or less in the site chrome of the ‘old page.’

Our takeaway form this observation is that changing the site chrome for one step in a flow is probably dangerous. Now, in our defense the users seeing this page are coming from a heavy desktop client piece of software, not (usually) other pages on the site, so we thought it would be fine to ‘start fresh’ with a new design. Perhaps it is the case that the user really does hold a pretty clear picture of the site in his or her head for months, and breaking the expectation that other messaging-focused properties related to the given product or brand would share this presentation can be quite troublesome to conversion. That’s pretty interesting if you ask me.

The other thing we noticed is that the new version did not have the McAfee secure site badge. Hmmm. Could it really be that this had a significant negative impact on the results? That will be the subject of our next test.

Conclusion

There are those in the anti-testing camp that are fond of the message: we’ll get it right, we could test but we’ll probably never come up with anything much better than the initial idea that we like the most ourselves. There are people in this camp. Believe me.

Jerry Springer time. My final thought(s):

1) You might be wrong. We all fall in love with our work at times, and testing can keep us honest by forcing us to reflect and digest the success and failure of our projects and ideas.

2) The professional lifestyle of testing will lead individuals and organizations to have better, more validated instincts when it comes to design-with-a-purpose.