7 Steps to Running Effective A/B Tests

March 5, 2015

Ofer Shai

blog_AB_testingAs a mobile marketer, we’re sure you are always looking for new ways to create fresh campaigns to increase your user base and ultimately drive more revenue. A/B testing is a great way to take small risks with your marketing campaigns, learn what resonates with your users and produces the highest conversion rates.

If you haven’t already incorporated A/B testing into your mobile marketing strategy then now is the time. You will be surprised, small changes to your marketing campaigns can produce immediate improvements in your key business goals. Not sure where to begin? We’ve got you covered.

Here are the basic steps to running effective A/B tests:

  1. Plan the goals you want to achieve and improvement in conversion rate that is important to you. Then, decide the variants you want to test. 
    • Example goals: Do you want your users to make a purchase? Share the app with a friend or on social media sites? Try a new app in your portfolio? etc.
    • Example conversion rate: How much of an improvement to your key metric do you consider worthwhile to detect? If 1% of your users are currently converting on the original variant A, what percent above 1% is important to you in order to choose variant B, C, D, etc. as the winning variant?
    • Example variants: changes to the messages (the top game vs number 1 game), the call to action (download now vs try it today), the image, etc.
  2. Determine the amount of risk you’re willing to take.
    • Are you willing to risk 10% of your users seeing a creative that does not perform well? What if that creative performs disastrously?
    • Are you willing to risk choosing the wrong winner because you didn't take the time to show the variation to a high enough number of users to produce accurate statistics.
    • These two risks have to be weighed against each other.
  3. Build a segment of users to send the A/B test.
    • Identify the user attributes that are important to achieving your overall goals. Common attributes used to build segments include language spoken, date of the first and last time they used your app, the total amount of money spent, and the amount of time spent in your app. Then, you can use unique combinations of those attributes to create a segment of users.
    • Example Segment: A common frustration among mobile marketers is figuring out the best way to to encourage non-spending users to make their first in-app purchase. So, using the attributes above, an example segment may be defined as users who have spent $0.00 in your app in their first five times using it. Then you could test two different types of Virtual Goods Promotions on the  segment and use the winning variant on the remaining segment of valuable users going forward.
  4. Calculate the size of the segment and duration you need to run the A/B test in order to produce accurate results.
    • You can use an online calculator to calculate how many results you need (number of users you need to see the tests) and the duration of the campaign.
    • Remember, the smaller the size of the segment of users you decide to test, the longer you will have to run the test. There’s opportunity cost to weigh now as well.
  5. Run the A/B Test for the full duration, NO PEEKING!
    • Running your A/B test for the full duration is imperative to achieving accurate results. Read more about why this is key to your A/B test in the best practices in our white paper listed below.
  6. Choose a winning variant.
    • Once the A/B test is complete, let the data from the results guide you in choosing a winning variant.
    • To do this, you’ll want to use a significance calculator. There are several available online.
    • Keep in mind that the winning variant might be the original baseline variant (A) and that sometimes, none of the variants are significantly better than the others.
  7. Show the winning variant to the rest of your chosen segment.
    • Watch your KPIs skyrocket!

Happy Testing!

mobileab_feature_01Now that you have the steps down to run an A/B test, read our white paper and we will take you further in depth with best practices related to each of these steps.


In: Product, Marketing

Subscribe To Our Newsletter:

Recent Posts