A/B Test Sample Size Calculator
00
Determine the required sample size per variation for an A/B test given uncertainty in baseline performance and desired sensitivity.
NOTEAbout This Model
## Determine the required sample size per variation for an A/B test given uncertainty in baseline performance and desired sensitivity. This model estimates **Required Sample Size Per Group** based on uncertain inputs: Baseline Conversion Rate, Minimum Detectable Effect. Run the simulation to see the distribution of possible outcomes.
VARIABLEBaseline Conversion Rate
beta(2.1, 17.9)
The current expected performance of the control group.
baseline_conversion_rate
VARIABLEMinimum Detectable Effect
beta(2, 19.49)
The smallest absolute lift in conversion rate worth detecting.
minimum_detectable_effect
FORMULAPooled Variance Factor
baseline_conversion_rate * (1 - baseline_conversion_rate) + (baseline_conversion_rate + minimum_detectable_effect) * (1 - (baseline_conversion_rate + minimum_detectable_effect))
Calculate the pooled variance multiplier based on baseline and detectability.
pooled_variance_factor
FORMULAZ Score Constant Term
pow(1.96 + 0.84, 2)
Standard normal z-score approximations for power (0.84 for 80%) and significance (1.96 for 95% two-sided).
z_score_constant_term
FORMULARequired Sample Size Per Group
round(z_score_constant_term * (pooled_variance_factor) / pow(minimum_detectable_effect, 2))
Calculate the required sample size per variation using the standard formula for difference in proportions.
required_sample_size_per_group
OUTPUTRequired Sample Size Per Group
required_sample_size_per_group
The minimum users needed per test group to detect your expected effect size with statistical confidence. Larger samples reduce false negatives but cost more time.
What is Carlo?
Carlo is a visual tool for Monte Carlo simulation. Model uncertainty by dragging probability distributions, connecting them visually, and running thousands of scenarios instantly.