Tool 15 — Free, no sign-up

How long before it means anything.

A two-proportion z-test, both ways round. How many impressions per creative you need before a CTR difference is real, and whether the difference you already have is.

Planning the test

CTR, install rate, whatever the test is about.
Use two-sided unless you genuinely do not care about B being worse.
Control included. Extra variants are Bonferroni-adjusted against the control.

The cost of precision

Sample per variant (log scale, both axes)
Effect Target rate Per variant Total Run time

Halving the effect you want to detect roughly quadruples the sample. That is the whole reason small tests on small lifts never finish.

Reading a test you already ran

Equal sample sizes per variant. Uses the confidence and sidedness set above.

—z statistic
—p-value
—Difference
—Confidence interval

Why peeking early is wrong

Every time you look at a running test and decide whether to stop, you give the random noise another chance to cross the significance line, so a test checked daily will produce a “winner” far more often than the 5% your confidence level promised. Fix the sample size before you start, look once when you reach it, and if you genuinely need to stop early use a sequential method built for it rather than the fixed-horizon maths on this page.

How it works / caveats

  1. Sample per variant = (zα√(2p̄(1−p̄)) + zβ√(p₁(1−p₁)+p₂(1−p₂)))² ÷ (p₂−p₁)², with p̄ the average of the two rates.
  2. zα uses α/2 for a two-sided test, and α is divided by the number of comparisons against control for Bonferroni.
  3. The reverse test is the pooled two-proportion z-test; the confidence interval uses the unpooled standard error, which is the conventional pairing.
  4. Normal quantiles come from Acklam’s rational approximation with one Halley refinement; the CDF is Abramowitz & Stegun 26.2.17. Both are accurate to well past the digits shown.
  5. Impressions are assumed independent. They are not: the same user seeing the same creative twice is one person’s opinion counted twice, which makes real tests slightly less powerful than this says.
  6. A significant CTR difference is not an install difference and definitely not a retention difference. Creatives that win on CTR sometimes bring worse players.