A/B Test Significance Calculator
Check whether the difference between two versions of a page, email or ad is statistically significant.
About the A/B Test Significance Calculator
An A/B test compares the conversion rates of two variants with a two-proportion z-test. A significant result means the difference is unlikely to be random noise; the confidence interval shows the plausible size of the real difference.
The formula
z = (p_B − p_A) ÷ √(p(1 − p)(1/n_A + 1/n_B))
- p
- pooled rate = (c_A + c_B) ÷ (n_A + n_B)
- Lift
- (p_B − p_A) ÷ p_A
Worked example
Example: Relative lift (B vs A) = +22.727%
- Inputs: Visitors to A: 5,500; Conversions in A: 220; Visitors to B: 5,500; Conversions in B: 270; Confidence level: 95%.
- Formula: z = (p_B − p_A) ÷ √(p(1 − p)(1/n_A + 1/n_B))
- Result: Relative lift (B vs A) = +22.727%
- Conversion rate A: 4%
- Conversion rate B: 4.9091%
- z: 2.31082
Things to keep in mind
- Decide the sample size before starting and don't stop the test early when it first looks significant; peeking inflates false positives.
Questions people ask
How long should I run a test?
Until you reach a pre-planned sample size, and for at least one full business cycle (usually a week).
What confidence should I use?
95% is standard; use 99% for costly, hard-to-reverse changes.