What confidence level should I require?

95% is the conventional bar, meaning roughly a one-in-twenty chance of a false positive. Higher-stakes decisions often warrant 99%.

Does a non-significant result mean there is no difference?

No. It means the data cannot distinguish the groups — often because the sample is too small. Absence of evidence is not evidence of absence.

Why is stopping a test early a problem?

Checking repeatedly and stopping when a result crosses the bar dramatically inflates false positives. Fix the sample size in advance and run the test to completion.

Calculator · 116

Statistical Significance Calculator

Test whether a measured difference is real — and decide whether the evidence supports acting on it.

Control conversions

count

Control sample

sample

Treatment conversions

count

Treatment sample

sample

Confidence level

—

Average

Scenario lens Current · Benchmark · Optimized

Leverage

Formula

Two-proportion z-test on control vs treatment conversion rates

Understanding statistical significance

Reference material — the calculator above stays the primary tool.

What significance measures

Statistical significance answers one question: how likely is a difference this large if the two groups were actually identical? The two-proportion z-test compares the observed rates against the variability expected from sampling alone, and returns the confidence that the difference is real.

It does not measure importance. A trivially small difference can be highly significant with enough sample, and a large one can fail significance with too little.

How to read your result

The result is the confidence level, judged against the conventional 95% decision bar:

Low — below 90%; the difference is plausibly chance. Average — 90–95%; suggestive but short of the bar. Strong — 95% or above; unlikely to be chance, reasonable to act on.

What significance does not tell you

Common misreadings worth avoiding. Treat these as orientation.

Context	Typical median
Significant	Not necessarily important
Not significant	Not proof of no effect
Peeking early	Inflates false positives
Many variants	Multiplies false winners

Testing discipline

Fix the sample size before starting, avoid stopping the moment a result crosses the bar, and account for multiple comparisons when testing several variants. Repeated peeking is the fastest way to manufacture a false winner.

Significance in context

Read this alongside the confidence interval and sample size tools, which the related tools cover. Significance says whether a difference is real; lift and intervals say how big it is and how precisely you know it.