What confidence level should I use?

95% is the standard in most industries. Use 90% when quick decisions matter more than precision, and 99% when a bad call is very costly (e.g., pricing changes or surgical UI changes affecting millions of users).

The p-value is the probability of seeing a difference this large — or larger — purely by chance, assuming the two variants actually perform identically. A p-value below 0.05 means there is less than a 5% chance the result is random noise (at 95% confidence).

How many visitors do I need before running this test?

You need enough visitors for your sample to be representative. As a rule of thumb, aim for at least 100 conversions per variant before drawing conclusions. Use a sample-size calculator before launching the test.

Does this handle one-tailed or two-tailed tests?

This calculator performs a two-tailed z-test, which is the safer default. It tests whether the variant differs from control in either direction, not just whether it is better.

Why might a large uplift still be non-significant?

A large percentage uplift on a small sample has wide confidence intervals, so the result could easily be noise. Statistical significance depends on both effect size and sample size.

A/B Test Statistical Significance Calculator

Name: A/B Test Statistical Significance Calculator
Availability: InStock
Author: Nham Vu

Enter visitors and conversions for control and variant to find out if your A/B test result is statistically significant at 90%, 95%, or 99% confidence.

Test Parameters

Confidence Level

Control (A)

Visitors

Conversions

Variant (B)

Visitors

Conversions

Fill in visitor and conversion counts,
then click Calculate Significance.

Summary

Enter visitors and conversions for control and variant to find out if your A/B test result is statistically significant at 90%, 95%, or 99% confidence.

How it works

Enter the number of visitors and conversions for your control group.
Enter the number of visitors and conversions for your variant group.
Choose a confidence level: 90%, 95%, or 99%.
Click "Calculate" to run a two-proportion z-test.
Read the result: significant or not, plus p-value, z-score, and relative uplift.

Use cases

Determine whether a landing page redesign lifts conversion rates meaningfully.
Validate email subject line tests before rolling out the winner to your full list.
Check if a new CTA color or copy change moves the needle beyond random chance.
Decide when to call an A/B test and stop collecting data.
Present test results to stakeholders with a confidence score they can act on.

Frequently Asked Questions

Last updated: 2026-07-01 · Reviewed by Nham Vu