P-hacking means torturing data until it produces a significant result. In academia it is a scandal. In business it is usually accidental, entirely routine, and expensive - because decisions get made on findings that will not replicate.
The six practices that produce false findings
1. Peeking and stopping early. Checking a test daily and stopping the moment it turns significant. This dramatically inflates false positives, because random fluctuation will cross the threshold sooner or later. It feels like efficiency; it is the most common form of p-hacking in commercial testing.
2. Testing many segments and reporting the winner. Slice by region, device, age band and channel, and something will look significant at the 5% level by chance alone. Reporting only that slice as the finding is manufacturing a result.
3. Changing the success metric after seeing the data. The test was for conversion; conversion did not move, but average order value did, so the report is about order value. Sometimes legitimate as an observation - never legitimate as a confirmed finding.
4. Dropping inconvenient data. Excluding an "unusual" week that happened to weaken the result. If the exclusion rule was not set in advance, it is not a rule.
5. Trying methods until one works. Running several tests or model specifications and reporting the one with the best p-value.
6. Presenting exploration as confirmation. Exploratory analysis is valuable and should generate hypotheses. It becomes dishonest when the hypothesis it generated is reported as though it had been tested.
The discipline that prevents all six
Write the analysis plan before you collect the data. One page, and it needs only five things:
- The hypothesis, stated specifically
- The primary success metric - exactly one
- The sample size or duration, calculated in advance
- The segments you will examine, named up front
- What result will trigger what decision
Then run it and report against the plan. Anything you discover outside the plan is labelled exploratory and requires confirmation in a fresh test before it drives spending.
Additional safeguards worth adopting
- Pre-register internally. Email the plan to a colleague before starting. Astonishingly effective.
- Report everything you looked at. "We examined eight segments; one was significant" is honest and correctly deflates the finding.
- Adjust for multiple comparisons when you genuinely need to test many things.
- Replicate before you scale. A finding worth millions is worth a confirming test.
- Report effect sizes and confidence intervals, not just p-values - the range makes over-claiming much harder.
Why it matters commercially, not just ethically
A false finding is not a neutral error. You roll out the change, incur the cost, and the expected benefit never arrives - and because business rarely re-tests, you may never learn it was not real. Organisations that p-hack accumulate a portfolio of expensive changes that do nothing, and grow steadily more confident in a body of false knowledge.
The cultural condition
P-hacking flourishes where analysts are rewarded for finding positive results. If "the test showed no effect" is treated as a failed piece of work rather than a useful finding, you have built an incentive to manufacture significance. See building a data culture for the behaviours that fix that.
The Statistics for Business course at London School of Business UK covers experimental design and analysis discipline. Enquire today.