What it is
A/B testing is a powerful optimisation tool that tests two different offers side by side to increase conversion metrics.
-
It’s used to test hypotheses about your customers
-
Testing two or more variants of an offer allows us to see the relative performance
-
Learnings are carried forward to build performance over time.
On average, you can achieve four times more referrals over the first six months.
AB testing is used to test hypotheses
Our Engineering team has built a sophisticated platform that lets us A/B test every element of your referral scheme, from rewards and sharing methods to copy and design.
By testing hypotheses, you can better understand customer behaviour and drive performance.
-
Does Stu like £ or %?
-
Does Andy like Pink or Red?
-
Does Simon like Bees or Ants?
AB testing puts hypotheses concurrently against one another to answer questions. It takes the guesswork out of optimisation.
Bayesian statistics model
By using statistics, it gives an unbiased view of your customer cohort.
-
Randomised AB testing allocates customers into cohorts of each variant
-
Statistics are used to determine whether the results are representative of the population
We have a Bayesian statistics tool built into our platform.
Baysian statistics: based on what we’ve seen so far, how confident are we that the difference between the metrics are significant? This gives us a degree of certainty that the key metric we’re testing has been improved – and this result is representative of the whole population.
The Bayesian tool is in the campaigns section and highlights metrics that have a statistically significant difference.
A/B testing by cohort
One of the most powerful optimisation tools for a referral programme is A/B testing by cohort. The ‘by cohort’ means you can present one variation of your offer to one cohort (group) that they and their friends can benefit from while showing the next cohort a completely different offer at the same time.
Experiment with every element of your referral customer journey, including incentive, design, copy, or imagery, so you can learn what resonates with your customer segments and optimise performance.
How to run a test
To run a robust test, there are a few things to consider
-
What is my hypothesis? Ahead of testing, you should define specifically what you are wanting to prove or disprove. For example…
-
“reducing the share options will improve customer experience and increase share rate”
-
“showing branded copy will increase referral metrics”
-
“displaying an incentive to encourage customers to sign up for an email will drive more sign-ups”
-
“providing a higher incentive will encourage more people to come back and purchase again”
-
-
What metric am I measuring to validate my hypothesis? This will need to be defined when the test is launched and will likely be part of your hypothesis. For example:
-
Share rate
-
Purchase rate
-
AOV
-
Repeat revenue
-
Number of email sign-ups
-
-
Where in the funnel is this? Do I have enough volume?
-
How long will I need to run this to get a significant result?
-
Is my experiment set up to test only my hypothesis?
Testing referral against use cases isn’t advised as the messages are so different and it is unlikely the goals will align. In the case of serving a message to drive email sign-ups, the key goal is to increase the email database. Whereas serving a message with a discount on the next order is encouraging customers to come back and purchase again, therefore repeat revenue would be a logical metric to track.
We recommend testing specific features of the message instead. For example:
-
Message to encourage sign-ups vs incentive to sign up for the newsletter
-
Discount on next order with no minimum spend vs discount on next order with minimum spend
-
7-day validity period for referral offer vs 14-day validity period for referral offer
Keep testing:
-
Once you’ve found a winner, you’re not going to improve unless you try a new test
-
So it’s a good idea to prepare an AB testing roadmap!
In this article you can read how to set up your first A/B test, and you can also watch this video.
