Top 6 Mobile app A/B Testing Mistakes and How to Avoid them

NEW YORK - Dec. 23, 2013 - PRLog -- So my boss asked me to use this A/B testing tool, telling me it’s supposed to do all these wonders for my mobile app. You know, things like a sudden burst in click-throughs, an exponential increase in my conversion rate, soaring revenues and all.

Well, it didn’t quite go that way. It was kind of like having a superman costume with the cape, but still waiting for the 8 o’clock local to reach office!

So where was I going wrong?

See, we’re all humans and thus we are bound to make certain mistakes. No matter how elementary and obvious they may be. You don’t agree? Allow me to share what I learned about A/B testing, the hard way.

#6: It doesn’t hurt to wait (Keep the test going until a statistically significant no. of conversions is achieved)

So, it went like this – I created the variations and then came the sudden conversions meaning, Success! Now onto the next test!

Well, just hold it for a bit. You see, at each point in time there is always more than one unseen factor at play. Actually, more than you keep track of.

So, when that sudden conversion came for you, it may have been for an entirely different reason other than your variation.

What you need to do, is keep the test going until a statistically significant number (http://en.wikipedia.org/wiki/Statistical_significance) of conversions is achieved. Only then can you pinpoint the reason for your conversions and be safe that it is not because of chance alone.

Frankly, it doesn’t hurt to wait.

#5: One drop is not sufficient for an explosion(In the arena of Split Testing, you’ve got to be bolder. Try radical changes)

Alright, I was really careful. I just changed the color of that navigation bar a hint darker and I waited for the magic…

A/B testing your app is not like those chemistry experiments you did in high school, where all it took for the bubbles to emerge was that one single drop! In the arena of split testing, you’ve to be bolder. You don’t think your action button looks good? Go ahead and give it a complete makeover. Change the text too if you wish and then see what happens.

#4: Don’t narrow your vision(Your goal is to get maximum conversions. Click-throughs, scroll downs, etc mean nothing if they do not equate to more conversions)

Always remember, your goal is to get maximum conversions. Click-throughs, scroll downs, visits etc. mean nothing if they do not equate to more conversions.

Conversions are what brings more money to the table. Suppose a red background color is getting you more hits but are giving you weak conversion numbers, then it is time to try something different.

So, don’t get distracted by little birdies when you should be looking at the stars.

#3: Your customers are not chicken(Always segment them in accordance with their age group, user platform, geographic location etc)

I once thought of my customers as chickens bred from the same farm. Life hasn’t been the same again.

The analogy here is pretty accurate. While formulating an A/B test, we often make the mistake of treating our target audience as being multiple copies of one ideal customer(chicken). What we forget, is that these chickens too have individual personalities.

Remember, there is no such thing as an ‘ideal customer’ with ‘ideal needs’.  What you need to do is divide or segment them in accordance with the different variables that may define their personality i.e. age group, user platform, geographic location etc.

#2: Climb off the hill to reach that mountain(Multivariate testing allows you to try various possible combinations for your app’s appearance)

When I’m standing on top of a hill, I tend to think that’s the highest one can go. If I take a step to my right, I realize it’s a downward journey; a step left, I get the same feeling. Finally, I gladly settle down with a cold pint of beer, blissfully thinking this is as high as high goes.
What I missed  was that far ahead(or maybe, not that far!) there was mountain peak that dwarfs my hill by miles and the only way to reach it is to climb down from the hill I’m standing on.

When you think of a design for your app, a similar thing happens. Suppose your landing page has a certain color combination, say green and orange, with the background color green and the action button in the color of orange. The combination works well for your conversions but definitely it is not the best. Now, there also exists a color combination of yellow and blue which would show a much better conversion rate but you don’t know that yet!

Like always, you start your A/B test by changing the your button color from orange to blue. But, your conversion rate falls. Next, you keep the action button color as orange and change the background to yellow. But once again, you see a fall in your conversion rate and thus, you never got to try that magical combination of blue and yellow.

We call the orange and green combination a local maxima(the hill) and the black and blue combination the global maxima(the mountain). So, can we ever know which combination is the best? This is where multivariate testing comes into play, allowing you to experiment with several attributes in one swoop.

So, try out all crazy combinations. Who knows, you might just find yourself on top of the Everest.

#1:  Stats tell the ugly truth(The app should be aimed at fulfilling its purpose, even if it means compromising on its ‘prettiness’)

I’d like to take action on what the test results are saying, but it just looks so… ugly.

Sure, it may not be that aesthetically pleasing to your eye, but you knew that your app was never in a beauty pageant anyway!

Whatever your app might do, it is there in the app store for a purpose, and each minute segment, each layout and each design of the app should be aimed at fulfilling that purpose, even if it means compromising on your app’s ‘prettiness’.

I guess by now you should already be seeing the world of app A/B testing very differently. Just keep the basic pointers in mind when you formulate your next test and hopefully you won’t have any troubles taking off with all the new superpowers you have just received.

Now, use your cape and fly like Superman.

APPY ITERATING!