markitechture

A/B Testing for Product Market Fit: Resonant Emotional Messaging

At a talk I recently attended, the speaker advocated the use of split testing testing and pointed to the visual beauty and quality user experience of a well-known (and recently acquired) web application. The message he left the audience with was “see, this is why you test, because you care about ux and quality design.”

My belief is that this particular company, well-known for their use of testing, gained most of the benefit from A/B testing while working toward product/market fit, and that their capacity to build intuitive and emotionally-pleasing (not easy to do) user experiences really came from having world-class design/UX/IxD talent.

How might you use A/B testing to achieve product/market fit?

Start by thinking of the most important outcome, the single greatest user benefit of your product. Now, think how different people might describe that ‘same’ outcome; how do your users describe the coolness of the app in “their words”?

A rough example: let’s take 3 generic apps: a productivity/GTD tool, a money management tool, and some sort of a social relationship manager. The outcome for each is straightforward (respectively): use your time better, manage your finances more efficiently, maintain better contact with people you already know.

In each of these ‘generic consumer web application’ scenarios, it’s easy to imagine a small scrappy team of designers and developers who have successfully built an appealing, working application and are currently driving traffic and focusing on measuring and optimizing adoption conversion metrics.

Herein lies the crux of the debate at hand, for there seems to be a paradox between delivering a cohesive, unified user experience, and testing minutia at every stage.

How does testing help here? Testing can help you find the resonant message(s) for your product.

Who people are, and why they would want to use your product are two sides of the same coin. If your product has a clear user benefit, you must not assume that you know best how to articulate this benefit to users.

Back to our example. Your productivity tool is gaining traction, and you are seeing mounting volume of referred traffic from social sources. Your signup rates are mediocre, and you wish to optimize them. First of all DO NOT SPEND PRECIOUS WEEKS TESTING MINUTIA! Instead, you might try to find out which of the following outcomes (which ‘dream’) resonates with your potential users:

(nb. these are concepts, not headlines)

1. Get ahead: Get more done at work so you can take on additional responsibility and further your career
2. Take control: Manage the various chaotic aspects of your life centrally (career, family, hobbies, school)
3. Perform: Kick ass at school by focusing on what matters, and executing quickly

What about the money management app? Who might be the potential dreams that your product could make come true for those users? Maybe:

1. Be disciplined: You know your income, you know your goals, so let’s keep you honest about how well you apply your funds
2. Get rich: All it takes to build wealth is insight and intelligence about your personal finances, and to start early
3. Lose the guilt: Rest assured you are doing everything you can with your money, use this app and you will be a good person

The third app was our social relationship manager; why might people fall in love with the idea of using it? Perhaps:

1. Get ahead: Talk to the right people at the right time about the right things, to create opportunity
2. Get married: 90% of happily married couples met through friends; who do your friends know?
3. Don’t be a stranger: Stay in touch with the amazing people you already know well, wherever they may be

See where this is going?

What A/B testing lets you do is to take the same interaction and visual design, the same ‘flat’ user experience, and collect feedback on the ‘3rd dimension,’ that of emotional connection, as related to your users’ value systems.

On a tactical note, I am recommending that you create landing page tests, with each message driving the copy and graphical content for one variation, and that you point sources of new-potential-user traffic to this page (see note below on context, segmentation, and testing). Monitor the most ‘downstream’ metric you have access to (purchase, ‘activation’, etc) and allow each test branch (variation) to collect at least 75-100 of this downstream goal. This will not be a waste of your time.

Even the exercise of imagining (or surveying to find!) the ‘driving dreams’ of your happy users is worth the sweat. Try to pull people into the variation-/message-creation process who are not as close to the product as you are.

A worthwhile recommendation for those who implement this approach: test messages by referral source. At the coarsest level test messages for people coming from Twitter, separate from those coming from Facebook shares, separate from those coming from SEM activities, separate from SEO if possible (it’s always possible). If your product can support more extensive SEM-based acquisition, you ought to test for message by Campaign/AdGroup.

I hope this adds a useful perspective to the current and valuable debate about split testing, specifically regarding its purposes, approaches, and challenges.