Statistical significance is one of the most used and most misunderstood ideas in business. Here is what it means, in plain English, and the three misreadings that cause real damage.
The idea in one paragraph
You observe a difference - version B converted better than version A. Two explanations exist: B genuinely is better, or B got lucky. Statistical significance is a way of asking: if there were truly no difference at all, how often would random chance alone produce a gap this large? If the answer is "rarely" - conventionally, less than 5% of the time - we call the result statistically significant and treat chance as an unsatisfying explanation.
That is the whole concept. Everything else is mechanics.
What a p-value is
The p-value is that "how often" figure. A p-value of 0.03 means: if there were no real effect, you would see a difference this big or bigger about 3% of the time by chance alone.
Three things a p-value does not mean
It is not the probability that your result is true. A p-value of 0.03 does not mean there is a 97% chance the effect is real. It is a statement about data given an assumption, not about the assumption given the data. This is the single most common misinterpretation, including among people who use it professionally.
It is not a measure of effect size. A p-value tells you a difference is probably not zero. It says nothing about whether it is big. With enough data, a completely trivial difference becomes highly significant - which is why large companies can find "significant" effects that are commercially pointless.
"Not significant" does not mean "no effect". It means you have not gathered enough evidence to rule out chance. Absence of evidence is not evidence of absence, and treating a non-significant pilot as proof that something does not work is how good ideas get killed.
The number that matters more: effect size with a confidence interval
Instead of "B was significantly better (p=0.04)", report: "B improved conversion by 0.6 percentage points, 95% confidence interval 0.1 to 1.1 points."
This tells you the direction, the plausible magnitude, and the precision - everything needed to decide whether to act. A confidence interval that spans from "trivial" to "transformative" is a signal that you need more data, which a bare p-value hides completely.
The business rule of thumb
Ask two questions of every result, in this order:
- Is it big enough to be worth acting on? (effect size, commercial judgement)
- Are we confident it is real? (significance, confidence interval)
If the answer to the first is no, the second does not matter. Most organisations ask them the wrong way round.
Where it goes wrong in practice
Stopping a test the moment it turns significant, running twenty tests and reporting the one that worked, and changing the success metric after seeing the data - all of these manufacture significance from noise. See how to avoid p-hacking for the discipline that prevents it.
Learning it properly
Significance is the point where business statistics becomes genuinely powerful and genuinely easy to misuse. The Statistics for Business course covers it with worked business examples and the common traps. Enquire today.