Being handed a fifteen-page report and having no idea whether to believe it is an uncomfortable and extremely common experience. The skill is not statistical depth - it is a reading order and a checklist.
Read it in this order, not front to back
- The conclusion or recommendation. What are they asking you to do?
- The method and data section, however brief. Where did the numbers come from and over what period?
- The charts. These carry most of the real content.
- The caveats and limitations. If there are none, that is itself a finding.
- Only then the detailed tables, and only the ones relevant to the recommendation.
Most people read in the opposite order, exhaust their attention on tables, and never scrutinise the claim.
The six questions that expose weak analysis
1. What is the denominator? "45% of respondents" - out of how many? A percentage without its base is not information.
2. Who is missing? Which customers, staff or transactions are not in this data? Non-respondents, churned customers, rejected applicants - the excluded group is often the one that answers your question.
3. What period, and is it typical? Does the window include a holiday, promotion, outage or system change?
4. Compared with what? A number alone means nothing. Compared with last year, with target, with the rest of the market, or with the normal range?
5. Is the change bigger than normal variation? The question that deflates most confident reports. If the metric routinely moves 5% week to week, a 4% move is not a finding.
6. Could something else explain this? Seasonality, a concurrent campaign, a definitional change, a data pipeline fault.
How to read the charts sceptically
- Check the y-axis start. A truncated axis turns a 2% change into a dramatic cliff.
- Check whether you are seeing a rate or a count.
- Look for the comparison line. A single series with no benchmark is decoration.
- Watch for dual axes, which can manufacture the appearance of a relationship between anything and anything.
- Ask what happened before the chart starts. Series often begin at a conveniently flattering point.
Vocabulary you need to not be bluffed by
- Mean vs median - if both are given and differ a lot, the data is skewed
- Confidence interval - the plausible range for the true value; wide means uncertain
- Statistically significant - probably not chance; says nothing about size or importance
- Sample size (n) - always look for it; small n means treat everything gently
- Year on year vs like for like - the second excludes new stores or products, and is usually the honest one
The two most useful things you can say in a meeting
"What is the sample size and who did not respond?" and "how much does that metric normally move anyway?" Neither requires expertise, and both regularly change conclusions. See more of these mistakes and how to catch them.
Getting properly fluent
Reading reports critically is the fastest-paying statistical skill for any manager. The Statistics for Business course at London School of Business UK teaches it from the consumer's side as well as the analyst's. Enquire today.