The honest answer is: it depends entirely on how small a difference you need to detect. That sounds evasive, but it is the actual principle, and once you understand it you can estimate your own requirement in a couple of minutes.
The rule that governs everything
The smaller the effect you are trying to detect, the more data you need - and the relationship is brutal. To detect an effect half as large, you need roughly four times as much data. This single fact explains most of what confuses people about sample sizes.
It is why detecting "did this double our conversion rate?" takes a handful of days, and detecting "did this improve conversion by 0.2 percentage points?" can take months of traffic you may not have.
Practical starting points
- Directional judgement on a big difference: a few dozen observations often suffice. If 18 of 20 customers say the checkout is confusing, you do not need a statistician.
- Survey of a customer base to within a few percentage points: a few hundred responses, largely regardless of how many customers you have in total. Precision depends on the sample size, not the population size - which surprises people.
- Conversion rate test on a small improvement: frequently thousands of visitors per variant. Run the arithmetic before you start, not after.
- Anything about a rare event (fraud, churn in a loyal base, safety incidents): far more than you expect, because what matters is the number of events, not the number of records.
The four inputs to the calculation
Any sample size calculator - and there are free ones everywhere - needs:
- Your current baseline (e.g. 3% conversion)
- The smallest difference worth acting on (this is a business decision, not a statistical one)
- Confidence level (95% conventionally)
- Power (80% conventionally - the chance of detecting the effect if it is real)
Do this before collecting data. Discovering afterwards that your test could never have detected a commercially relevant effect is a waste of a month.
Two questions that matter more than sample size
Is the sample representative? A biased sample of 10,000 is worse than a random sample of 200, because a large biased sample gives you confident wrong answers. Who is missing matters more than how many are present.
How much does this metric normally vary? A noisy metric needs far more data than a stable one. Look at your own week-to-week variation before deciding anything.
When to stop collecting and decide
Three legitimate stopping points:
- The confidence interval is narrow enough that every value in it leads to the same decision
- The cost of waiting exceeds the value of extra certainty - very common, and a perfectly rational reason to act on partial evidence
- More data cannot resolve the question because the uncertainty is about something else entirely (strategy, competitor behaviour, values)
The judgement underneath
The real skill is deciding how much certainty a decision deserves. A reversible £500 experiment needs almost none. A £2m irreversible commitment deserves the full calculation. Matching the rigour to the stakes is what separates useful analysis from analysis paralysis.
The Statistics for Business course covers sampling and sample size properly, applied to business decisions. Enquire today.