A/B Testing Metrics Update (Summer 2025)
As part of the improvements to A/B testing introduced in summer 2025, Mention Me has updated the metrics used to evaluate test performance. This article explains the changes, why they were introduced, and how they differ from existing metrics.
What’s the Change?
Mention Me is introducing three new metrics for tests:
- Variation share rate
- Variation purchase rate
- Variation conversion rate
These will be used specifically within A/B tests or single variant testing. The existing share, purchase, and conversion rates remain unchanged and will continue to appear at the campaign, market, and programme levels.
The original metrics are calculated from a broader user base—those enrolled in any offer—regardless of which variation they saw. The new metrics, however, only consider customers who were shown a specific test variant, offering more targeted and reliable insights during experiment evaluation.
Metric Comparison
| Existing | New | |
|---|---|---|
| Share Rate | Shares from enrolled customers ÷ Total enrolled customers | Shares from customers with an impression ÷ Unique impressions |
| Purchase Rate | Purchases from shares ÷ Shares from enrolled customers | Purchases from shares ÷ Shares from customers with an impression |
| Conversion Rate | Purchases from shares ÷ Enrolled customers | Purchases from shares ÷ Unique impressions |
How Does This Help?
The new metrics more accurately measure the impact of a specific offer by exclusively tracking users who:
- Were shown that exact variant
- Decided to refer (share)
- Earned purchases from their shared offers
This filtering prevents unrelated impressions, enrolments, and conversions from skewing test results. In A/B testing, this reduces noise and allows faster, clearer conclusions.
While the new metrics are used within experiments only, the original metrics still serve as a complete picture of campaign success and continue to be used in programme-level reporting.
You may notice that unique impressions can exceed total enrolments. Along with the metric changes, this might result in numerical differences compared to what you’re used to—but is expected and more accurate for experimentation.
Why Would a Customer See One Offer and Be Enrolled in Another?
Referral links are valid for a set period (typically 3–6 months). If a referee visits after this time, they’re shown the most current offer—so they may convert via a different offer than the referrer originally promoted.
In A/B testing, this leads to an attribution ambiguity: Should the original offer be credited, or the new one shown? For experiment integrity, Mention Me’s updated metrics attribute only to the experience actually displayed to the customer. This makes A/B test evaluation more precise.
These scenarios are still valuable and are fully included in overall campaign metrics outside of test performance reports.