Green Belt · 15 min
Is this difference real? And is it big enough to care about?
After this you can
- choose the right comparison test for how the data was collected, and say why.
- state what a p-value does and does not tell me about a difference.
- report an effect size and a confidence interval beside a p-value and say which decision each one supports.
Assumes you have done PDCA: the Check is the part everybody skips and Capable, or just in control?.
The problem
A software team A/B tested a checkout change on 1,000 users per arm and reported p < 0.001 — highly significant. The change was rolled out to every customer at a cost of six engineer-weeks. The effect it detected was 1.5 units on a spread of 10: about 0.15 standard deviations, which is smaller than the difference between a Tuesday and a Wednesday. The p-value was correct. It was also answering a question nobody had asked.
The idea
Every change ends with the same question: is this difference real? The arithmetic that answers it is one line. Almost every failure in practice is in what the number gets taken to mean.
What a p-value is
The p-value is the probability of seeing a difference at least this large if there were genuinely no difference at all. Small p means the data would be surprising under "nothing is happening", so "nothing is happening" is a poor description of it.
That is the whole content. In particular it is not:
- the probability that there is no difference;
- the probability the result is a fluke;
- a measure of how big the difference is;
- a measure of how important it is.
The last two are the expensive ones, because a p-value shrinks as the sample grows. With enough data, any difference becomes significant — including one far too small to act on. That is not a flaw; it is what "detectable" means.
What to put beside it
The effect size. How big is the difference, in units the process uses, or in standard deviations (Cohen's d) when you need to compare across measures. This is the number the decision actually turns on.
The confidence interval. The range of differences consistent with what you observed. It carries everything the p-value carries and more: if it excludes zero, the test is significant, and its width tells you how precisely you know the answer.
Reported together, the four cases become readable:
| p | Effect | Reading |
|---|---|---|
| Small | Large | A real difference worth acting on |
| Small | Tiny | Real, and probably not worth the effort. Very large samples land here |
| Large | Large | Could not tell — the study was too small for the effect it was looking for |
| Large | Tiny | Some evidence there is nothing much here |
The bottom-left case is the one most often reported as "no difference", and it is the opposite: a difference of practical size is entirely consistent with the data.
Choosing the test
The choice follows from how the data was collected, not from what you want to show.
Paired — the same units measured twice. Before and after on the same machines, the same operators, the same accounts. Pairing removes the variation between units, which is usually the largest source, so it detects far smaller differences from the same number of measurements. Running an unpaired test on paired data throws most of that away.
Two independent samples — different units in each group. Two machines, two shifts, two suppliers.
One sample against a target — is this process centred where it is supposed to be?
For two independent samples there is a second choice, and the common default is the wrong one. The pooled t-test assumes both groups have the same variance. Welch's does not, costs essentially nothing when the variances happen to be equal, and is far more reliable when they are not. Welch should be the default; this tool makes it one.
Two things decided before the data
The alternative. A one-sided test asks whether A is greater than B, and it is roughly twice as easy to pass as a two-sided test. Choosing it after seeing which way the data went halves the p-value for nothing. It has to be a statement about what would have counted as a result, made in advance.
How much data. Detecting a small effect takes a lot of it. At 80% power and α = 0.05 you need about 393 per group for an effect of 0.2 standard deviations, 63 for 0.5, and 25 for 0.8. Deciding the size of difference that would matter to you before collecting is the only thing that turns "not significant" into information rather than an anticlimax.
Worked example
Twelve changeovers under the old setup procedure and twelve under a new one, on the same line. Different changeovers in each group, so these are independent samples — not paired.
| n | Mean | SD | |
|---|---|---|---|
| Old | 12 | 41.500 | 2.747 |
| New | 12 | 35.167 | 2.125 |
Step 1 — choose the test before looking further. Independent groups, so a two-sample t. Welch, because nothing has established that the two procedures have the same variance — and, as it happens, the old one is visibly more variable. Two-sided, because a new procedure could plausibly have been worse.
Step 2 — the test.
t = 6.317 on 20.69 degrees of freedom. The fractional df is Welch's correction and it is reported as computed; rounding it down would quietly change the p-value.
p < 0.001.
Step 3 — the size, which is where the decision lives.
- Difference in means: 6.333 minutes
- 95% confidence interval: 4.247 to 8.420 minutes
- Cohen's d: 2.579 — large
Six minutes off every changeover, and the data is consistent with anything from four to eight. That interval is what you take to a manager: not "significant", but "between four and eight minutes per changeover, and here is how many changeovers we do".
Step 4 — what the pooled test would have given. t = 6.317 on 22 degrees of freedom, p ≈ 0.0000023. Almost identical here, because the variances are close and the group sizes are equal — which is exactly the case where the choice does not matter. It matters when they are not, and you do not know which case you are in until you look. Defaulting to Welch means never having to.
Now the Hook, for contrast. 1,000 per arm, a difference of 1.5 on a standard deviation of 10. p < 0.001, and Cohen's d = 0.15 — negligible. Same verdict word, opposite decision. The only thing that separates the two cases on a report is the effect size, and it is the number that gets left out.
Dataset: ds-changeover-times — the same data loads in the tool below, so you can reproduce every figure here yourself.
Your turn
The tool opens with the changeover data, and puts the two choices — which test, and what would count as a result — above the paste boxes, because both have to be made before the numbers are seen.
- Confirm t = 6.317 on 20.69 df, a difference of 6.333, and the interval 4.247 to 8.420.
- Tick use the pooled-variance test. The df becomes a whole 22 and the p-value barely moves — this is the case where the choice does not matter. Now delete four values from Group B and try again: the two tests start to separate.
- Switch the alternative to the first is greater. The p-value halves. Notice how tempting that is once you have already seen which way the data went — which is exactly why the choice belongs before the data.
- Make the comparison marginal: delete values from both groups until the result stops being significant. Read the sentence carefully. It reports the largest difference still consistent with the data rather than claiming there is none.
- Look at the planning line. For the effect you are seeing, it says how many observations per group you would want — which is the number to argue about before the next trial, not after it.
Hypothesis Test
practiceDifferent units in each group — two machines, two shifts, two suppliers.
The means differ, in either direction.
Distinguishable from zero — p < 0.001, effect large
The difference of 6.3333 is distinguishable from zero at alpha = 0.05. The 95% interval runs from 4.2466 to 8.4201. The effect is large.
Difference
6.333
t
6.317
df
20.69
Cohen's d
2.579
- Test
- Two-sample t (Welch)
- 95% interval
- 4.247 to 8.420
- n
- 12, 12
Planning: to detect an effect of this size (2.58 standard deviations) at 80% power you would want about 3 per group. Approximate — computed from the normal approximation, which understates the requirement slightly at small n because the exact calculation needs the noncentral t distribution. Treat it as a planning figure. It is also the number needed to DETECT an effect of 2.5790679110450445 standard deviations, which is a decision about what size of difference would matter, not a statistical one.
n=12, mean 41.500, sd 2.747
n=12, mean 35.167, sd 2.125
How this is calculated
- Welch (default): t = (x̄₁ − x̄₂) / √(s₁²/n₁ + s₂²/n₂), with Welch–Satterthwaite degrees of freedom, which are fractional. They are reported as computed — rounding them down is a silent change to the p-value.
- Pooled: the same numerator over a common variance estimate, on n₁ + n₂ − 2 degrees of freedom. Valid only when the population variances are equal.
- Paired: a one-sample t on the differences. Pairing removes the between-unit variation, so it detects smaller differences from the same number of measurements.
- Cohen’s d uses the pooled standard deviation regardless of which test ran — it describes separation and is not part of the inference. The bands are Cohen’s conventions, which he described as a last resort for when no substantive basis exists.
- Below 5 observations the normality assumption cannot be assessed from the sample at all, and the tool says so rather than proceeding quietly.
Source: Montgomery, D.C., Introduction to Statistical Quality Control 7e, Ch. 4; Welch, B.L. (1947), Biometrika 34; Ruxton, G.D. (2006), Behavioral Ecology 17(4), on Welch as the default; Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, 2nd ed.
Check yourself
No hints. Wrong answers are explained, not softened.
A test returns p = 0.03. Which statement is correct?
Eight machines are each measured before and after a maintenance change. Which test, and what is at stake?
A trial reports p = 0.42 with a 95% interval for the difference of -3 to +11 units. What is the honest conclusion?
Worth remembering
What is a p-value?
The probability of seeing a difference at least this large IF there were none. Not the probability there is no difference, not the probability it is a fluke, and no information about size.
Why does a very large sample make almost anything significant?
p shrinks as n grows, so any non-zero difference eventually becomes detectable — including one far too small to act on. That is what 'detectable' means, and it is why the effect size belongs beside every p.
Pooled or Welch for two independent samples?
Welch, by default. Pooling assumes equal variances, which almost nobody checks. Welch costs essentially nothing when they are equal and is far more reliable when they are not.
Can you do this now?
Rate yourself honestly. We compare your rating with how you actually answered — the gap is more useful than either number alone.
I can choose the right comparison test for how the data was collected, and say why.
I can state what a p-value does and does not tell me about a difference.
I can report an effect size and a confidence interval beside a p-value and say which decision each one supports.
Your rating is recorded alongside your drill results. Neither alone marks the competency as met.