A metric went up. That is not the same as improvement. Distinguishing the two is possibly the most practically valuable statistical skill a manager can have, and it comes down to four checks.
Check 1: Is the change outside the normal range?
Every metric fluctuates. Before you can interpret a move, you need to know how much it moves anyway.
How to work it out: take the last 20-30 periods of the metric, calculate the average and the standard deviation. Roughly speaking, anything within two standard deviations of the average is ordinary variation. Anything beyond it deserves attention.
Plot this as a chart with the average and the upper and lower bounds drawn on - a control chart. It is the single most useful management chart in existence and takes ten minutes to build in a spreadsheet. Once you have one, most of the drama in your monthly reporting disappears.
Check 2: Is there a run, not just a spike?
One point outside the range might be a one-off. Statistical process control uses simple pattern rules that are worth knowing:
- Seven or more consecutive points above the average - that is a shift, not chance
- Six or more consecutive points trending in one direction - that is a trend
- Any single point far outside the bounds - investigate the event
Sustained patterns are much stronger evidence than a single dramatic reading, and they are what you should be looking for.
Check 3: Compared with the right thing?
Improvement is relative, and three comparisons are worth making every time:
- Against the same period last year - handles seasonality
- Against the trend, not the last point - the previous period may have been unusual in either direction
- Against the wider market or your other units - if every region improved 6%, your region's 6% is not a result of anything you did
That third comparison is the one most often skipped, and it is the one that separates genuine performance from a rising tide.
Check 4: Is it the metric you actually care about, and has something else got worse?
The classic pattern: call handling time falls 20%, and repeat contacts rise 30%. Or conversion rises because a discount was applied, and margin collapses. Always pair an efficiency metric with a quality metric and a financial one - otherwise you will optimise a number while damaging the business.
Set it up properly once
For each of your five to eight key metrics, define and write down:
- The exact definition and data source
- The normal range, calculated from history
- The pairing metric that guards against gaming
- What movement would trigger action, agreed in advance
That document is worth more than any dashboard, because it turns "the numbers moved" into "here is what we decided we would do".
Why this matters commercially
Organisations without this discipline oscillate: they launch initiatives to fix problems that were noise, then declare victory when the metric reverts to its mean - and learn nothing either way. It is exhausting and expensive, and a control chart largely ends it. See the other mistakes in the same family.
The Statistics for Business course at London School of Business UK covers variation, control charts and metric design applied to real reporting. Enquire today.