Skip to content
Leantensify Learn

Is this difference real, or could it be chance?

Compare two groups, a before and after, or a process against a target — with the effect size and the confidence interval next to the p-value rather than below a fold.

Hypothesis Test · analyze · Green Belt · free, no account needed

Use this when

  • A change was made and you need to show whether it worked
  • Two machines, shifts or suppliers appear to differ
  • Somebody has quoted a p-value and you want to know how big the difference actually is

Hypothesis Test

sandbox

Different units in each group — two machines, two shifts, two suppliers.

The means differ, in either direction.

Distinguishable from zero — p < 0.001, effect large

The difference of 6.3333 is distinguishable from zero at alpha = 0.05. The 95% interval runs from 4.2466 to 8.4201. The effect is large.

Difference

6.333

t

6.317

df

20.69

Cohen's d

2.579

Test
Two-sample t (Welch)
95% interval
4.247 to 8.420
n
12, 12

Planning: to detect an effect of this size (2.58 standard deviations) at 80% power you would want about 3 per group. Approximate — computed from the normal approximation, which understates the requirement slightly at small n because the exact calculation needs the noncentral t distribution. Treat it as a planning figure. It is also the number needed to DETECT an effect of 2.5790679110450445 standard deviations, which is a decision about what size of difference would matter, not a statistical one.

n=12, mean 41.500, sd 2.747

n=12, mean 35.167, sd 2.125

How this is calculated
  • Welch (default): t = (x̄₁ − x̄₂) / √(s₁²/n₁ + s₂²/n₂), with Welch–Satterthwaite degrees of freedom, which are fractional. They are reported as computed — rounding them down is a silent change to the p-value.
  • Pooled: the same numerator over a common variance estimate, on n₁ + n₂ − 2 degrees of freedom. Valid only when the population variances are equal.
  • Paired: a one-sample t on the differences. Pairing removes the between-unit variation, so it detects smaller differences from the same number of measurements.
  • Cohen’s d uses the pooled standard deviation regardless of which test ran — it describes separation and is not part of the inference. The bands are Cohen’s conventions, which he described as a last resort for when no substantive basis exists.
  • Below 5 observations the normality assumption cannot be assessed from the sample at all, and the tool says so rather than proceeding quietly.

Source: Montgomery, D.C., Introduction to Statistical Quality Control 7e, Ch. 4; Welch, B.L. (1947), Biometrika 34; Ruxton, G.D. (2006), Behavioral Ecology 17(4), on Welch as the default; Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, 2nd ed.

How this is calculated

Welch's two-sample t by default: t = (mean1 - mean2) / sqrt(s1^2/n1 + s2^2/n2) with Welch-Satterthwaite degrees of freedom, reported fractional because rounding them down silently changes the p-value. The pooled-variance test is available and states on the result that it was used; it assumes equal population variances. Paired comparisons run a one-sample t on the differences. Cohen's d uses the pooled standard deviation regardless of which test ran, since it describes separation rather than forming part of the inference. p-values come from the exact t distribution and are never rendered as zero.

Source: Montgomery, D.C., Introduction to Statistical Quality Control 7e, Ch. 4; Welch, B.L. (1947), Biometrika 34; Ruxton, G.D. (2006), Behavioral Ecology 17(4), on Welch as the default choice; Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences 2nd ed.

Learn the method