How A/B Testing Works at Mention Me

A/B Testing: Drive Optimisation Through Smart Experimentation

A/B testing is the most effective way to improve referral programme performance. Whether you're testing imagery, rewards, copy, or CTA buttons, Mention Me's A/B system helps you experiment, learn, and grow with ease.

What Is A/B Testing in Mention Me?

A/B testing lets you show different variants of a referral experience to different customer groups—allowing you to identify what performs best. For example, you might compare:

• 10% vs. £10 off
• Image A vs. Image B
• Button colour variations
• Tone of voice in share messaging

Key Benefits

• Boost metrics like share and conversion rate
• Understand preferences across customer segments
• Test any element across the referral journey
• Build insights over time to drive sustained gains

How Our A/B Testing Works

How We Determine a Winner

Our system uses Bayesian modelling, not traditional statistical significance. It evaluates:

• Probability to be best — how confident we are in a variant's performance
• Pre-set confidence thresholds to conclude tests

Tests auto-end after 8 weeks or if a winner is found earlier.

Why We Don’t Use Traditional Significance

We focus on probability-based evaluation to:

• Run faster tests
• Support low-volume campaigns
• End underperforming tests early

What Metrics We Use

1. Variation Conversion Rate (default): best for high volumes, measures purchasing behaviour directly
2. Variation Share Rate: useful for low-volume campaigns to evaluate share behaviour

Conversion Rate is preferred where viable, as it correlates directly with revenue.

Planning Your A/B Test

1. Start with a Clear Objective

Examples:

• "Will fewer share methods improve share rate?"
• "Is £10 off more effective than 10% discount?"
• "Does urgency-focused messaging improve referee purchases?"

2. Isolate One Variable

Test one change at a time to keep insights measurable and actionable.

3. Check Your Volume

We'll help you select the right metric and testing conditions based on your referral traffic. For very low volumes, we may suggest best-practice alternatives.

How Long Do Tests Run?

• Maximum duration: 8 weeks*
• Most winners found within 4 weeks
• Faster tests use lower confidence thresholds
• Slow tests allow for more confident decisions

What Happens When Tests Conclude?

At the end of a test, results are evaluated and your chosen action is applied automatically or manually:

1. A winner is found: that variant is promoted
2. No clear winner: default to Control (can be changed)
3. Probability difference but low confidence: promote higher-performing variant by default (can be adjusted)

When and Why Tests May End Early

1. Clear winner emerges: test ends early, variant promoted
2. No meaningful difference: test ends early to save time and refocus efforts

Build a Testing Roadmap

Continuous testing is key. Plan experimental cycles based on data and hypotheses. A simple monthly approach:

FAQs

Why does the A/B Testing Dashboard show higher customer numbers than downloadable reports?

The A/B testing dashboard includes all customers who enrolled, including those who shared their name only. CSV reports only reflect customers who clicked to share a referral link.

To get a full list of enrolled customers, contact your Client Success Manager. Learn more about automatic post-purchase enrolment here.

Key Takeaways

• Uses Bayesian probability, not legacy statistical significance
• Test every stage: rewards, journey, images, copy
• Default cap: 8 weeks (average: ~4 weeks)
• Conversion rate is the primary metric, but share rate is an option
• Smart test creation + performance insights help low-traffic brands

Ready to Optimise?

Log in to Mention Me or speak to your Client Success Manager to launch your first test.

*Note: Some tests may take 12 weeks due to cooling-off requirements (e.g. approval delays or refund windows).