Foundation · 14 min
Signal or noise: reading a number that moves
After this you can
- plot a run chart from time-ordered data and describe what it shows.
- distinguish common-cause from special-cause variation given a run chart.
- explain why reacting to every fluctuation makes a stable process worse.
The problem
A warehouse team reported despatch errors every Monday. Week 12 came in at 20 — one of the best figures of the year — and the shift lead was thanked in front of everyone. Week 13 came in at 32, a 60% jump, and the same lead spent an hour explaining what had gone wrong. By week 16 it was 19 again and nobody said anything at all. Nothing about the process had changed in any of those weeks. The whole cycle — the praise, the explanation, the silence — was a response to numbers that meant nothing.
The idea
Every process produces numbers that move. The question that matters is whether a particular movement means anything.
Two kinds of variation sit behind any number:
Common-cause variation is the ordinary, ever-present wobble of a process working as it normally does. Traffic, who was on shift, how many people phoned that morning, which supplier's batch arrived. No single one of these is the reason the number moved; all of them together are. Common-cause variation is a property of the system, and the only way to reduce it is to change the system.
Special-cause variation is something that was not there before. A new booking rule. A machine out of calibration. A member of staff who left. Special causes are worth chasing individually, because there is something specific to find.
The expensive mistake is treating one as the other, and it goes wrong in both directions:
- Treating common cause as special is called tampering. You investigate an ordinary fluctuation, find something (you always will), change it, and add a new source of variation to a process that was already doing its best. Deming demonstrated this with the funnel experiment: adjusting after every result makes the spread worse, not better.
- Treating special cause as common means shrugging at a real change and letting it become the new normal.
You cannot tell which is which from two data points. You cannot tell from ten either, if you only look at the last two of them. What you need is the whole series, in order, with a line through the middle — a run chart.
A run chart is time on the horizontal axis, the measurement on the vertical, points joined in order, and a median line through the middle. That is all it is. The median, not the mean, because one extreme value drags a mean towards the very point you are trying to notice.
Then you apply rules that ask: is this pattern one that random variation produces easily?
- A shift — six or more consecutive points on the same side of the median. Random data rarely stays on one side that long.
- A trend — five or more consecutive points all rising or all falling. Worth a look, but the weakest of the rules: it fires on random data more often than people expect.
- Too few or too many runs — a "run" is a block of consecutive points on the same side. Too few means the series sits in one place then another. Too many means it alternates like a metronome, which usually means two different things are being plotted as one.
If none of them fire, the honest statement is: nothing here is distinguishable from ordinary variation. That is not the same as "the process is fine". A stable process producing a result you hate will keep producing it forever until somebody changes the process.
Worked example
An outpatient clinic recorded the median wait from arrival to being seen, once a week, for 24 weeks.
| Week | Wait (min) | Week | Wait (min) |
|---|---|---|---|
| 1 | 46 | 13 | 38 |
| 2 | 52 | 14 | 33 |
| 3 | 41 | 15 | 36 |
| 4 | 49 | 16 | 31 |
| 5 | 44 | 17 | 35 |
| 6 | 55 | 18 | 29 |
| 7 | 43 | 19 | 34 |
| 8 | 48 | 20 | 32 |
| 9 | 51 | 21 | 37 |
| 10 | 45 | 22 | 30 |
| 11 | 47 | 23 | 33 |
| 12 | 50 | 24 | 28 |
Step 1 — the median of all 24 values is 39.5 minutes. Sort them, take the average of the 12th and 13th. (The mean is 40.29, close here, but the median is what the rules use.)
Step 2 — mark each point above or below 39.5. Weeks 1–12 are every one of them above. Weeks 13–24 are every one of them below. Twelve above, twelve below, none exactly on the line.
Step 3 — apply the rules.
Shift: twelve consecutive points above the median, then twelve below. Both are far past the threshold of six. Two shifts.
Runs: the series crosses the median exactly once, so there are 2 runs. With 12 points on each side, ordinary variation would produce somewhere between 8 and 18 runs. Two is so far outside that range that the exact probability of seeing it by chance is under 1 in 600,000 — reported as p < 0.001.
What the chart says, and what it does not. It says something changed at around week 13, and it says so with more confidence than any pair of weeks could. It does not say what changed. That is the next question, and the chart has told you exactly where to point it: what happened between week 12 and week 13?
Now look at what the weekly report would have said. Week 5 to week 6 is 44 to 55 — up 25%, an alarming week. Week 6 to week 7 is 55 to 43 — down 22%, a recovery. Both of those "movements" are ordinary variation inside a period where nothing whatsoever was changing. A report comparing adjacent weeks would have generated eleven stories before reaching the one real event.
Dataset: ds-clinic-wait-weekly — the same data loads in the tool below, so you can reproduce every figure here yourself.
Your turn
The Run Chart tool opens with the clinic data.
-
Confirm the median reads 39.5 and that the tool reports 2 runs.
-
Delete weeks 13–24, leaving only the first twelve. The median recalculates and the signals disappear — because within that period, there genuinely is no signal. This is the single most useful thing you can do with the tool: watch a chart stop claiming something when the evidence for it is removed.
-
Now paste the despatch error data from the Hook:
27, 21, 27, 19, 25, 28, 28, 27, 19, 22, 23, 20, 32, 27, 29, 19, 28, 27, 24, 26Twenty weeks, ten above the median and ten below, 10 runs, no shift, no trend — no signals at all. That includes the week-12-to-13 jump from 20 to 32 that cost the shift lead an hour, and the fall from 32 to 19 that nobody mentioned. Both are noise.
Run Chart
practiceShow the data behind this chart
| # | Label | median wait | Vs median |
|---|---|---|---|
| 1 | Week 1 | 46 | above |
| 2 | Week 2 | 52 | above |
| 3 | Week 3 | 41 | above |
| 4 | Week 4 | 49 | above |
| 5 | Week 5 | 44 | above |
| 6 | Week 6 | 55 | above |
| 7 | Week 7 | 43 | above |
| 8 | Week 8 | 48 | above |
| 9 | Week 9 | 51 | above |
| 10 | Week 10 | 45 | above |
| 11 | Week 11 | 47 | above |
| 12 | Week 12 | 50 | above |
| 13 | Week 13 | 38 | below |
| 14 | Week 14 | 33 | below |
| 15 | Week 15 | 36 | below |
| 16 | Week 16 | 31 | below |
| 17 | Week 17 | 35 | below |
| 18 | Week 18 | 29 | below |
| 19 | Week 19 | 34 | below |
| 20 | Week 20 | 32 | below |
| 21 | Week 21 | 37 | below |
| 22 | Week 22 | 30 | below |
| 23 | Week 23 | 33 | below |
| 24 | Week 24 | 28 | below |
Points
24
Median
39.5
Longest run
12 points
- Above median
- 12
- Below median
- 12
- Runs observed
- 2
- Expected range
- 8–18
shift12 consecutive points above the median (points 1–12). A run of 6 or more is unlikely by chance alone, so something about the process changed here — the chart says when, not what.
shift12 consecutive points below the median (points 13–24). A run of 6 or more is unlikely by chance alone, so something about the process changed here — the chart says when, not what.
too few runs2 runs where 8 or more would be expected (p < 0.001). Too few runs means the points cluster on one side and then the other — the process is not varying around a single stable level.
One observation per line, oldest first: Week 1, 34 — or just the number. Do not sort it. The order IS the data.
How this is calculated
The centre line is the median, not the mean — a single extreme value moves the mean and would drag the centre line towards the very point you are trying to notice.
- Shift — 6 or more consecutive points on one side of the median. Points sitting exactly on it are skipped, neither extending nor breaking a run.
- Trend — 5 or more consecutive points all rising or all falling. Shown, but the weakest of the three: it fires on random data more often than people expect.
- Too few / too many runs — the number of times the series crosses the median, compared against the exact distribution of the number of runs. The critical values here are computed from that distribution rather than looked up in a table.
A fourth classic rule — an “astronomical” point obviously unlike the rest — is deliberately not automated. It is a judgement about the process, and any threshold would get it wrong in both directions.
Source: Provost & Murray (2011), The Health Care Data Guide, Ch. 3 for the rules; Swed & Eisenhart (1943), Annals of Mathematical Statistics 14(1) and Gibbons & Chakraborti, Nonparametric Statistical Inference 5e §3.2 for the runs distribution.
Saved runs can be attached to a project deliverable as evidence. Both what you entered and what the tool computed are stored, so the result can be checked again later.
Your turn
Now the part that is nearly impossible to accept from prose. Tampering is the claim that adjusting a stable process makes it worse — and Deming built an experiment that settles it in about ten seconds.
A marble drops through a funnel aimed at a target. Where it lands varies by an amount nobody controls. The four rules are four things you could do about that.
- Start on rule 1: leave the funnel alone. Note the spread — this is the floor.
- Switch to rule 2: after each drop, move the funnel to correct the miss. Nothing about this is careless; every adjustment is a reasonable response to a real deviation. The spread is about 1.41× wider — the variance exactly doubles, and it stays doubled however long you run it.
- Switch to rule 4: set the funnel to wherever the last marble landed. This is the one organisations actually run — this month's target is last month's actual. Now raise the drops to 500. The spread keeps growing; there is no number it settles on.
- Change the seed. The details move, the ranking never does.
All four rules run on the same drops, which is the only fair comparison: if each had its own randomness, the difference between them might be the luck rather than the rule.
The Funnel Experiment
practiceA marble drops through a funnel aimed at a target. Where it lands varies by an amount nobody controls. Choose what to do about that — and watch what it costs. All four rules run on the same drops, so any difference between them is the rule and not the luck.
Spread vs rule 1
× 1.39
Landing σ
1.352
Drop σ
0.975
Worst miss
4.56
Rule 2After each drop, shift the funnel from its current position by the opposite of the error just seen. Entirely reasonable, and it DOUBLES the variance — because you are now adding an adjustment, itself driven by noise, on top of the noise.
Over these 500 drops the spread is 39% wider than leaving the funnel alone, and the worst miss is 1.3× as far out. Nobody did anything careless. Every adjustment was a reasonable response to a real deviation.
This rule settles at 2× the variance (1.41× the standard deviation) however long you run it. Change the drop count and the ratio stays put — that is what “stationary” means.
How this is calculated
Each drop’s noise is an independent draw from a normal distribution with σ = 1, generated from the seed so the run is reproducible. Write d = landing − target. Every rule acts on d — the distance the marble landed from target — never on the noise itself, because the person adjusting can only see where it went.
- Rule 1: d = e. Variance σ².
- Rule 2: d = en − en−1. Variance 2σ², settled.
- Rule 3: dn+1 = −dn + e. Sign flips each drop, amplitude grows without bound.
- Rule 4: dn+1 = dn + e. A random walk; variance ≈ n·σ².
Source: Deming, W.E. (1994), The New Economics, Ch. 9; Deming (1986), Out of the Crisis, Ch. 11.
Check yourself
No hints. Wrong answers are explained, not softened.
A run chart of 20 weekly values shows 9 runs, no run longer than 4, and no trend. Week 18 is the highest point on the chart. What is the correct reading?
A manager reviews a stable process weekly and adjusts a setting whenever the figure is worse than last week. Over six months, what happens to the variation?
You have 18 monthly values. Someone sorts them from smallest to largest before plotting, saying it makes the chart easier to read. What have they destroyed?
Worth remembering
Why does a run chart use the median rather than the mean as its centre line?
A single extreme value drags the mean towards the very point you are trying to notice. The median barely moves, so the centre line stays where the bulk of the data is.
What is the shift rule on a run chart?
Six or more consecutive points on the same side of the median. Points sitting exactly on the median are skipped — they neither extend nor break a run.
A run chart shows no signals. What can you conclude?
That nothing in the data is distinguishable from ordinary variation. NOT that the process is good — a stable process delivering a bad result will keep delivering it until the process changes.
Can you do this now?
Rate yourself honestly. We compare your rating with how you actually answered — the gap is more useful than either number alone.
I can plot a run chart from time-ordered data and describe what it shows.
I can distinguish common-cause from special-cause variation given a run chart.
I can explain why reacting to every fluctuation makes a stable process worse.
Your rating is recorded alongside your drill results. Neither alone marks the competency as met.