A/B Test Sample Size Calculator
Community
00
Determine the required sample size per variant for an A/B test given uncertainty in baseline performance and the desired sensitivity.
NOTEAbout This Model
## Determine the required sample size per variant for an A/B test given uncertainty in baseline performance and the desired sensitivity. This model explores how **Required Sample Size Per Variant** could play out, depending on Baseline Conversion Rate, Minimum Detectable Effect. Hit **Run** to see the range of possibilities.
VARIABLEBaseline Conversion Rate
beta(2.2, 17.8)
Typical conversion rates for web events range from 2% to 20%. We avoid 0 to prevent division errors.
baseline_conversion_rate
VARIABLEMinimum Detectable Effect
beta(2, 19.45)
The smallest absolute change in conversion rate you care about detecting. Usually 0.5% to 5%.
minimum_detectable_effect
FORMULAPooled Variance Estimate
baseline_conversion_rate * (1 - baseline_conversion_rate) + (baseline_conversion_rate + minimum_detectable_effect) * (1 - (baseline_conversion_rate + minimum_detectable_effect))
Calculate the pooled variance for the two proportions based on the baseline and the effect.
pooled_variance_estimate
FORMULARequired Sample Size Per Variant
(pow(1.96 + 0.84, 2) * pooled_variance_estimate) / pow(minimum_detectable_effect, 2)
Standard sample size formula for a two-sided test with alpha=0.05 and beta=0.20 (Z-scores 1.96 and 0.84).
required_sample_size_per_variant
OUTPUTRequired Sample Size Per Variant
required_sample_size_per_variant
Track the required sample size per variant output
What is Carlo?
Carlo is a visual tool for Monte Carlo simulation. Model uncertainty by dragging probability distributions, connecting them visually, and running thousands of scenarios instantly.