These groups look different — are they, and which ones?

Compare three or more groups at once — with the pairwise comparisons on the same screen, because a significant F never says which group differs.

Comparing Several Groups (ANOVA) · analyze · Green Belt

Use this when

Three or more machines, suppliers, shifts or settings to compare at once
Somebody has run every pair as a separate t-test and found something
You need to know which groups differ, not just that some do

Comparing several groups (ANOVA)

sandbox

At least one group differs — and this test does not say which

F(3, 32) = 12.753, p < 0.001. At least one of the 4 groups differs from the others, and the grouping accounts for about 49.5% of the total variation. **This test has not identified which group.** It compares the spread between the groups with the spread inside them and reports that the first is too large; it has no opinion at all about the ranking. Reading the group means as an order of merit at this point is the error the test exists to prevent, and running all 6 pairwise comparisons instead would give a 26% chance of at least one false positive. Use the pairwise comparisons below, which hold the error rate across the whole family.

12.753

3 / 32

p < 0.001

ω² (adjusted)

0.4948

Analysis of variance table: sums of squares, degrees of freedom, mean squares and the F ratio.
Source	SS	df	MS	F
Between groups	126.329	3	42.110	12.753
Within groups (error)	105.660	32	3.302	—
Total	231.989	35	—	—

In the order you entered them, not sorted by mean. Spread is shown beside the mean: two groups can have the same mean and very different consistency, and the F test does not look at that at all.
Group	n	Mean	SD
Alpha	9	48.056	0.914
Borea	9	47.433	1.240
Cyrix	9	48.278	2.273
Delvin	9	43.656	2.381

Which groups actually differ — Tukey’s HSD

3 of 6 pairs differ by more than chance: Alpha vs Delvin, Borea vs Delvin, Cyrix vs Delvin. The error rate is held at 5% across all of them together, not per comparison — so these can be read as a set, which is exactly what a list of uncorrected t-tests cannot be. Any difference below 2.3208 was too small for this study to call.

Critical q (4 groups, 32 df): 3.832
Honestly significant difference: 2.321
Comparisons: 6

Every pair of groups compared, with the error rate held across the whole family rather than per comparison.
Pair	Difference	95% interval	p	Verdict
Alpha vs Borea	0.622	-1.699 to 2.943	p = 0.886	Could not tell
Alpha vs Cyrix	-0.222	-2.543 to 2.099	p = 0.994	Could not tell
Alpha vs Delvin	4.400	2.079 to 6.721	p < 0.001	Differs
Borea vs Cyrix	-0.844	-3.165 to 1.476	p = 0.758	Could not tell
Borea vs Delvin	3.778	1.457 to 6.099	p < 0.001	Differs
Cyrix vs Delvin	4.622	2.301 to 6.943	p < 0.001	Differs

Why the correction: 4 groups means 6 pairwise comparisons. Running each at α = 0.05 without correcting gives about a 26.5% chance of at least one false positive on data with no real differences at all — something would look significant roughly one time in four.

The most variable group has 6.8 times the variance of the least variable one. ANOVA assumes they are equal. The groups are all the same size, which makes the test fairly tolerant of this — but the difference in spread is itself worth investigating, and may matter more than the difference in means.

What uncorrected t-tests would have said

Alpha vs Borea: p = 0.4729 not significant
Alpha vs Cyrix: p = 0.7970 not significant
Alpha vs Delvin: p = 0.0000 significant
Borea vs Cyrix: p = 0.3316 not significant
Borea vs Delvin: p = 0.0001 significant
Cyrix vs Delvin: p = 0.0000 significant

Shown so the cost of correcting can be seen rather than taken on trust. This is the mistake, computed on purpose — it is not an alternative analysis to choose between.

Group, then bond strength

One reading per row. Groups keep the order they first appear in.

Alpha

Applies to the whole family of comparisons, not to each one.

Load an example

How this is calculated

The F test splits the total variation into a between-groups part and a within-groups part, and compares their mean squares: F = MS_between / MS_within on k−1 and N−k degrees of freedom. With two groups this is exactly the pooled t-test, and F = t².
Tukey’s HSD compares every pair while holding the error rate across the whole family. It uses the studentized range distribution, which has no closed form — it is computed here by numerical integration and verified both against Harter’s published tables and against the exact identity q(1−α, 2, df) = √2 · t(1−α/2, df).
Unequal group sizes use the Tukey-Kramer form, which widens the interval for the smaller group. Applying the equal-size formula would understate it, which is the direction that manufactures significance.
ω² is reported rather than η², because η² is biased upward — always, and more so with small samples. A negative ω² is shown as computed: it means the grouping accounts for less variation than chance would.
Equal variances are assumed by the F test. The ratio of the largest group variance to the smallest is checked, and whether the groups are equally sized is stated with it, because that is what decides how much the assumption matters.

Source: Montgomery, D.C., Design and Analysis of Experiments 8e, Ch. 3; Tukey, J.W. (1953), The Problem of Multiple Comparisons; Harter, H.L. (1960), Annals of Mathematical Statistics 31(4), 1122–1147.

How this is calculated

One-way analysis of variance: the total sum of squares is split into between-groups and within-groups parts and their mean squares compared as F = MS_between/MS_within on k-1 and N-k degrees of freedom, with p from the exact F distribution. SS_total is computed independently rather than by addition, so the ANOVA identity is a testable claim rather than an assumption. Effect size is reported as omega-squared as well as eta-squared, since eta-squared is biased upward; a negative omega-squared is reported as computed rather than clamped. Pairwise comparisons use Tukey's HSD, holding the family-wise error rate at alpha across every pair; unequal group sizes use the Tukey-Kramer form. The studentized range distribution has no closed form and is computed by 96-point Gauss-Legendre quadrature over the infinite-df case averaged across the chi distribution of the scale factor, verified against Harter's published tables and against the exact identity q(1-alpha, 2, df) = sqrt(2) t(1-alpha/2, df). The uncorrected pairwise t-tests are also computed and shown, so the cost of the correction is visible rather than asserted.

Source: Montgomery, D.C., Design and Analysis of Experiments 8e, Ch. 3 (the fixed-effects model and Example 3.1, used as the verification fixture); Tukey, J.W. (1953), The Problem of Multiple Comparisons; Harter, H.L. (1960), Tables of Range and Studentized Range, Annals of Mathematical Statistics 31(4), 1122-1147; Hays, W.L. (1963) for omega-squared.

Learn the method

Four suppliers, one answer, and the question it does not answerGreen Belt · 15 min · free