When evaluating the impact of Smart Experiments, accuracy and statistical significance are essential. Experiments that remain ‘unconcluded’, those without a statistically significant difference between test groups, are excluded from overall value calculations.
This article explains:
- What an “unconcluded” experiment is
- How we determine statistical significance using the Repeat Rate Significance Test
- Why conclusive results matter when measuring incremental value
What is an ‘unconcluded’ experiment?
To evaluate a Smart Experiment’s value, we compare test groups using the Repeat Rate Significance Test to determine if results are statistically meaningful.
The Repeat Rate is the rate at which customers make a second purchase. The test tells us whether any difference observed between the Control Group and Low Propensity to Refer group is statistically valid and not due to chance.
Only when this difference is statistically significant do we consider the experiment ‘concluded’.
Experiments without statistically significant differences remain ‘unconcluded’.
Why can’t we use ‘unconcluded’ data to calculate incremental value?
Impact on Repeat Revenue
To determine repeat revenue generated by showing an alternative offer to the Low Propensity group, we compare its repeat rate to the Control group’s.
If a test is ‘unconcluded’, no significant difference means we can’t confidently calculate additional repeat revenue.
When a test does conclude:
- If Low Propensity wins: additional repeat revenue was driven by showing the alternative offer.
- If Control wins: showing referral to everyone would have driven more repeat revenue.
Impact on Estimated Referral Revenue Missed
To assess the value lost by not showing referral, we use:
- Share rate of the Control group
- Number of customers shown the alternative offer
- Historical average value of a share
Why are unconcluded tests excluded from overall value?
To calculate total incremental value, we subtract Missed Referral Revenue from Repeat Revenue Gained. If a test is inconclusive, we can’t calculate repeat revenue gained, and thus can’t confidently determine incremental value.
In such cases, we show only the missed revenue from referrals on the results dashboard.