Ab Test Calculator

Imagine flipping a coin that you suspect is slightly biased. After 10 flips you get 6 heads — is it biased, or just luck? After 1000 flips you get 600 heads — now you are confident. A/B testing works the same way. Your control and variant are two coins, and the question is whether the difference in their conversion rates reflects a real property of your change or just the natural randomness of human behavior.

The calculator computes a two-proportion z-test. It takes the observed conversion rate for each variant, calculates a pooled standard error that reflects how much random fluctuation you would expect given your sample sizes, and then measures how many standard errors apart the two rates are. That distance — the z-score — determines whether the gap is surprising enough to call real.

Statistical confidence is the complement of the p-value: a 86.5% confidence result means that if there were truly no difference between control and variant, you would observe a gap this large or larger only rarely by chance. The confidence level you set (90%, 95%, or 99%) is the threshold you require before declaring a winner. Raising it reduces false positives but requires more data to reach.

Use this calculator when you have cleanly split traffic between exactly two variants (control and one challenger), you have defined a single binary conversion goal before the test started, and your traffic has been randomly assigned rather than self-selected. It is most reliable when sample sizes are roughly equal between arms and when each group has at least several hundred observations.

Do not use this calculator for tests where the same visitor can appear in both variants (cookie deletion, multi-device users at scale), for continuous metrics (revenue per visitor, session duration), or for tests with more than two variants. It also should not be used when external events (a sale, a news story, a competitor outage) affected one test period but not another — the independence assumption breaks down.

The result is also not appropriate as a post-hoc justification. If you designed the test to run for two weeks, do not stop at day four because significance appeared — planned stopping rules are part of what makes the math valid.

Stopping when you see significance. The most common mistake is checking results continuously and stopping the moment significance is reached. This practice inflates your false positive rate dramatically — the more often you look, the more likely random fluctuation will temporarily cross the threshold. Decide your required sample size before you start and stop only when you hit it.

Treating relative uplift as the main signal. A +15.5% relative improvement looks compelling, but if the sample backing it has low confidence (86.5% confidence), it is not a real finding. Small absolute differences on small samples produce large relative percentages that evaporate at scale. Always anchor judgment in confidence, not just uplift.

Running multiple variants without correction. This calculator tests one control against one variant. If you are running A/B/C tests (three or more variants), you need a correction for multiple comparisons — otherwise the chance that at least one pair shows spurious significance grows quickly. For multi-variant tests, apply a Bonferroni correction or use a dedicated multi-arm tool.

The two-proportion z-test calculates a z-score from your inputs. First, compute each conversion rate: control rate p₀ = control conversions / control visitors (4% in the example), and variant rate p₁ = variant conversions / variant visitors (4.62%).

Next, compute the pooled proportion: p̅ = (control conversions + variant conversions) / (control visitors + variant visitors). The pooled standard error is SE = sqrt(p̅ * (1 - p̅) * (1/n₀ + 1/n₁)), where n₀ and n₁ are visitor counts. The z-score is then z = (p₁ - p₀) / SE.

The two-tailed p-value is derived from the standard normal distribution: p = 2 * (1 - CDF(|z|)). Confidence is 1 - p, displayed as 86.5% confidence. If the absolute z-score exceeds the critical value for your chosen threshold (1.96 for 95%, 1.645 for 90%, 2.576 for 99%), the result is significant. Relative uplift is (p₁ - p₀) / p₀, shown as +15.5%.

The pooled z-test assumes equal variance under the null hypothesis and performs well when conversion rates are not near 0% or 100% and sample sizes are reasonably large. Where it breaks down: very low base rates (under about 1%) cause the normal approximation to the binomial to become inaccurate; in those cases a Fisher exact test or a Bayesian approach is more reliable. Similarly, when traffic splits are highly unequal (one variant gets ten times the visitors of another), the pooled variance estimate becomes unstable. The two-tailed formulation used here is conservative — it detects both positive and negative effects — but if you pre-registered a directional hypothesis you could use a one-tailed test with lower required sample size, accepting the risk that you cannot detect harm.