Green Belt · 15 min

Four suppliers, one answer, and the question it does not answer

After this you can

say what a significant F test has established and what it has not.
explain why running every pairwise comparison inflates the error rate, and quantify it.
read a set of corrected pairwise comparisons and state which groups a study could actually separate.

Assumes you have done Is this difference real? And is it big enough to care about?.

A quality team compared six production lines on scrap rate. They ran a t-test on every pair — fifteen of them — and found two that were significantly different at 5%. The two lines were investigated for three weeks.

The lines were identical. With fifteen comparisons at 5% each, the chance of finding at least one difference where none exists is 54%. They were more likely to find something than not, and they did.

The idea

Comparing two groups is a t-test. Comparing more than two is where it goes wrong, and it goes wrong in a specific, countable way.

Why not just run all the pairs

The error rate is per test. Set α = 0.05 and each individual comparison has a 5% chance of a false positive. But the comparisons multiply: with k groups there are k(k−1)/2 pairs.

Groups	Pairs	Chance of ≥1 false positive at α = 0.05
3	3	14%
4	6	26%
5	10	40%
6	15	54%

That is not a subtlety. At six groups you are more likely than not to find a "significant" difference in data with no differences at all — and you will find it in a specific pair, which makes it feel like a discovery rather than a coin landing heads.

What ANOVA does

Analysis of variance asks one question about all the groups at once: are these means all the same? It works by splitting the total variation in two:

Between groups — how far the group means sit from the overall mean.
Within groups — how much the readings scatter inside each group.

If the grouping means nothing, those two are estimates of the same thing and their ratio F is around 1. If the groups genuinely differ, the between-groups part is inflated and F is large.

One test, one error rate, however many groups.

What ANOVA does not do

It does not say which group. This is the whole reason the lesson exists.

A significant F establishes that the group means are not all equal. It has no opinion at all about the ordering, and it sits directly above a table of means that looks exactly like an answer. Reading that table as a ranking after a significant F is the most common error made with this test, and it is made by people who know better, because the table is right there.

It does not look at spread. ANOVA compares means. Two suppliers with the same mean and wildly different consistency are identical to this test, and are not identical to buy from.

Tukey's HSD: which groups, honestly

The answer is not "run the t-tests anyway". It is to compare every pair with the error rate held across the whole family rather than per comparison.

Tukey's Honestly Significant Difference does that. The intervals it produces are simultaneous: all of them hold together at the stated confidence, which is a stronger claim than a set of separate 95% intervals and the only one you can read as a set.

The cost is power. Some differences an uncorrected test would call significant are not significant after correction — and those are precisely the ones that were most likely to be noise. That is not the correction being conservative. That is the correction working.

Worked example

Nine joints tested from each of four adhesive suppliers, measured as the force needed to pull the joint apart.

Supplier	n	Mean (N)	SD
Alpha	9	48.056	0.914
Borea	9	47.433	1.240
Cyrix	9	48.278	2.273
Delvin	9	43.656	2.381

Step 1 — the F test.

Source	SS	df	MS	F
Between suppliers	126.329	3	42.110	12.753
Within suppliers	105.660	32	3.302
Total	231.989	35

p < 0.001. At least one supplier differs. The adjusted effect size ω² = 0.495, so the supplier accounts for about half the variation in bond strength.

Step 2 — stop, before reading the table of means.

The four means are 48.056, 47.433, 48.278 and 43.656. That is an ordering, and it is not a finding. The F test has just told you that something differs; it has said nothing about which, and three of those four numbers may be separated by nothing but noise.

Running all six pairwise t-tests instead would carry a 26% chance of at least one false positive.

Step 3 — Tukey's HSD. Critical q(0.05; 4, 32) = 3.832, giving an HSD of 2.321 N: any difference smaller than that was beyond what this study could resolve.

Pair	Difference	95% interval	Verdict
Alpha vs Borea	0.622	−1.699 to 2.943	Could not tell
Alpha vs Cyrix	−0.222	−2.543 to 2.099	Could not tell
Borea vs Cyrix	−0.844	−3.165 to 1.476	Could not tell
Alpha vs Delvin	4.400	2.079 to 6.721	Differs
Borea vs Delvin	3.778	1.457 to 6.099	Differs
Cyrix vs Delvin	4.622	2.301 to 6.943	Differs

The finding is one supplier, not a ranking. Delvin is weaker than the other three. Alpha, Borea and Cyrix are not distinguishable from one another — their apparent order is noise, and a purchasing decision made on it would be a coin toss with a spreadsheet attached.

Step 4 — the thing the F test cannot see.

Alpha and Cyrix have almost the same mean: 48.06 against 48.28, a difference of 0.22 N that no test here can separate. Their standard deviations are 0.914 and 2.273 — Cyrix is two and a half times as variable.

To an analysis of variance those two suppliers are identical. To anyone specifying a minimum bond strength they are not remotely identical, because the supplier that scatters more will produce more joints below any limit you set. The tool reports the spread beside the mean for exactly this reason, and the variance-ratio check flags it.

Dataset: ds-adhesive-bond-strength — the same data loads in the tool below, so you can reproduce every figure here yourself.

Your turn

The tool opens on the adhesive data with the pairwise comparisons on the same screen as the F test — not behind a tab, because that is how the ranking gets read.

Confirm F = 12.753 on 3 and 32 degrees of freedom, and the HSD of 2.321.
Read the group table. Note that it is in the order the data was entered and not sorted by mean — and notice how much you want it to be sorted.
Read the Tukey table. Three pairs say could not tell. Those three suppliers are the ranking that does not exist.
Open what uncorrected t-tests would have said. On this data they agree with Tukey, which is worth seeing too: correcting does not always change the answer.
Now load the where the correction bites preset. Uncorrected tests call four pairs significant; Tukey calls two. The two that vanish are marked. That is the 26% arriving.
Change alpha to 0.01 and watch the HSD grow. A stricter error rate is a wider net that catches fewer things, and the intervals show you exactly how much wider.

Comparing several groups (ANOVA)

practice

At least one group differs — and this test does not say which

F(3, 32) = 12.753, p < 0.001. At least one of the 4 groups differs from the others, and the grouping accounts for about 49.5% of the total variation. **This test has not identified which group.** It compares the spread between the groups with the spread inside them and reports that the first is too large; it has no opinion at all about the ranking. Reading the group means as an order of merit at this point is the error the test exists to prevent, and running all 6 pairwise comparisons instead would give a 26% chance of at least one false positive. Use the pairwise comparisons below, which hold the error rate across the whole family.

12.753

3 / 32

p < 0.001

ω² (adjusted)

0.4948

Analysis of variance table: sums of squares, degrees of freedom, mean squares and the F ratio.
Source	SS	df	MS	F
Between groups	126.329	3	42.110	12.753
Within groups (error)	105.660	32	3.302	—
Total	231.989	35	—	—

In the order you entered them, not sorted by mean. Spread is shown beside the mean: two groups can have the same mean and very different consistency, and the F test does not look at that at all.
Group	n	Mean	SD
Alpha	9	48.056	0.914
Borea	9	47.433	1.240
Cyrix	9	48.278	2.273
Delvin	9	43.656	2.381

Which groups actually differ — Tukey’s HSD

3 of 6 pairs differ by more than chance: Alpha vs Delvin, Borea vs Delvin, Cyrix vs Delvin. The error rate is held at 5% across all of them together, not per comparison — so these can be read as a set, which is exactly what a list of uncorrected t-tests cannot be. Any difference below 2.3208 was too small for this study to call.

Critical q (4 groups, 32 df): 3.832
Honestly significant difference: 2.321
Comparisons: 6

Every pair of groups compared, with the error rate held across the whole family rather than per comparison.
Pair	Difference	95% interval	p	Verdict
Alpha vs Borea	0.622	-1.699 to 2.943	p = 0.886	Could not tell
Alpha vs Cyrix	-0.222	-2.543 to 2.099	p = 0.994	Could not tell
Alpha vs Delvin	4.400	2.079 to 6.721	p < 0.001	Differs
Borea vs Cyrix	-0.844	-3.165 to 1.476	p = 0.758	Could not tell
Borea vs Delvin	3.778	1.457 to 6.099	p < 0.001	Differs
Cyrix vs Delvin	4.622	2.301 to 6.943	p < 0.001	Differs

Why the correction: 4 groups means 6 pairwise comparisons. Running each at α = 0.05 without correcting gives about a 26.5% chance of at least one false positive on data with no real differences at all — something would look significant roughly one time in four.

The most variable group has 6.8 times the variance of the least variable one. ANOVA assumes they are equal. The groups are all the same size, which makes the test fairly tolerant of this — but the difference in spread is itself worth investigating, and may matter more than the difference in means.

What uncorrected t-tests would have said

Alpha vs Borea: p = 0.4729 not significant
Alpha vs Cyrix: p = 0.7970 not significant
Alpha vs Delvin: p = 0.0000 significant
Borea vs Cyrix: p = 0.3316 not significant
Borea vs Delvin: p = 0.0001 significant
Cyrix vs Delvin: p = 0.0000 significant

Shown so the cost of correcting can be seen rather than taken on trust. This is the mistake, computed on purpose — it is not an alternative analysis to choose between.

Group, then bond strength

One reading per row. Groups keep the order they first appear in.

Alpha

Applies to the whole family of comparisons, not to each one.

Load an example

How this is calculated

The F test splits the total variation into a between-groups part and a within-groups part, and compares their mean squares: F = MS_between / MS_within on k−1 and N−k degrees of freedom. With two groups this is exactly the pooled t-test, and F = t².
Tukey’s HSD compares every pair while holding the error rate across the whole family. It uses the studentized range distribution, which has no closed form — it is computed here by numerical integration and verified both against Harter’s published tables and against the exact identity q(1−α, 2, df) = √2 · t(1−α/2, df).
Unequal group sizes use the Tukey-Kramer form, which widens the interval for the smaller group. Applying the equal-size formula would understate it, which is the direction that manufactures significance.
ω² is reported rather than η², because η² is biased upward — always, and more so with small samples. A negative ω² is shown as computed: it means the grouping accounts for less variation than chance would.
Equal variances are assumed by the F test. The ratio of the largest group variance to the smallest is checked, and whether the groups are equally sized is stated with it, because that is what decides how much the assumption matters.

Source: Montgomery, D.C., Design and Analysis of Experiments 8e, Ch. 3; Tukey, J.W. (1953), The Problem of Multiple Comparisons; Harter, H.L. (1960), Annals of Mathematical Statistics 31(4), 1122–1147.

Check yourself

No hints. Wrong answers are explained, not softened.

An ANOVA across five machines returns F = 7.2, p = 0.002. What has been established?

Six lines are compared by running a t-test on every pair at α = 0.05. If the lines are genuinely identical, what is the chance of at least one 'significant' result?

After a significant F, Tukey's HSD shows one supplier differing from the other three, and those three not distinguishable from one another. What is the finding?

Two suppliers have means of 48.06 and 48.28, with standard deviations of 0.91 and 2.27. What does the ANOVA say, and what should you do?

Worth remembering

What does a significant F test establish?

That the group means are not all equal. Nothing about which one differs, and no ordering at all. Reading the table of means as a ranking afterwards is the error the test exists to prevent.

Why not just run a t-test on every pair?

The error rate is per test and the pairs multiply. Six groups is fifteen pairs, and at α = 0.05 each there is a 54% chance of at least one false positive in data with no differences at all.

What makes Tukey's intervals different from ordinary confidence intervals?

They are simultaneous: all of them hold together at the stated confidence, so the set can be read as a whole. A collection of separate 95% intervals cannot be.

What can ANOVA not see?

Spread. It compares means, so two groups with the same mean and very different consistency are identical to it — and are not identical against a specification limit.

Can you do this now?

Rate yourself honestly. We compare your rating with how you actually answered — the gap is more useful than either number alone.

I can say what a significant F test has established and what it has not.
I can explain why running every pairwise comparison inflates the error rate, and quantify it.
I can read a set of corrected pairwise comparisons and state which groups a study could actually separate.

Your rating is recorded alongside your drill results. Neither alone marks the competency as met.

The problem