Chapter 3 · Comparing groups beyond the t-test

After ANOVA, which groups differ?

Post hoc tests: Bonferroni, Holm, Tukey, Dunnett, and the false discovery rate.

Read first: Choosing a test, Hypothesis testing

The problem first

You are a second-year OBG resident. Your thesis compares the rise in haemoglobin at 8 weeks in women with mild postpartum anaemia given dietary advice alone, daily oral iron, alternate-day oral iron or intravenous iron, 6 women in each group. One-way ANOVA gives p < 0.001.

To find which groups differ, you run a t-test for every pair of groups: six t-tests, each at 5%. Five are significant, and you report all five. The external examiner circles the six p-values in your results table.

“You made six comparisons at 5% each. Which post hoc test did you use, and why that one?”

Start here

Why not run a t-test for every pair of groups?

Each test at 5% has a 1 in 20 chance of a false positive when the groups do not really differ. The more tests you run, the higher the chance that at least one of them is falsely significant.

Every pair of groups compared with its own t-test at 5%, when no group really differs. Dark bars: 1 − 0.95^m for m independent tests. Red bars: 100,000 simulated studies with 10 patients per group, where the pairwise tests share groups and are therefore not independent. The chance is lower than the formula suggests, but it is still far above 5%.

With k groups there are k(k − 1) ÷ 2 pairs: 6 pairs for 4 groups, 15 for 6. A one-way ANOVA answers only one question at 5%: does at least one mean differ? It does not say which ones. Multiple comparison procedures answer the second question while keeping the chance of any false positive under control.

One-way ANOVA is explained in the choosing a test chapter, and type I error and p-values in the hypothesis testing chapter.

Two error rates

What is a family, and what is the family-wise error rate?

A family is the set of comparisons that together answer one research question. The family-wise error rate (FWER) is the probability of at least one false positive anywhere in that family.

The family must be defined before the analysis. If the question is which groups differ from each other, the family is all six pairs. If the question is whether each treatment beats the control, the family is only three comparisons, and a smaller family needs less adjustment.

Per-comparison error rate: the false-positive rate of each single test (usually 5%)
Family-wise error rate (FWER): the probability of one or more false positives among all the tests in the family
False discovery rate (FDR): the expected proportion of false positives among the results called significant

SPSS describes the unadjusted least significant difference (LSD) test as using t-tests for all pairwise comparisons with “no adjustment ... made to the error rate for multiple comparisons”. That is the resident's six t-tests.

Adjusting p-values

How do Bonferroni and Holm share out the 5%?

Bonferroni gives every test the same share of α. Holm starts with the same share for the smallest p-value and allows a larger share at each step.

Bonferroni: test each p at α ÷ m, or multiply each p by m

Holm: test the i-th smallest p at α ÷ (m − i + 1)

Holm works through the sorted p-values from the smallest and stops at the first one that fails its threshold. That one and all larger p-values are not significant.

m: number of comparisons in the family (6 in the worked example)
i: rank of the p-value, 1 for the smallest

Worked example, six pairwise p-values. Bonferroni tests every p-value at 0.05 ÷ 6 = 0.0083 and calls four comparisons significant. Holm tests the smallest at 0.0083, the next at 0.05 ÷ 5 = 0.010, and so on up to 0.05. Oral A vs control (p = 0.0133) passes its Holm threshold of 0.025, so Holm calls five comparisons significant.

Holm (1979) showed that the procedure protects against an error of the first kind at the chosen level “for any combination of true hypotheses”. The R documentation for p.adjust states that there seems no reason to use the unmodified Bonferroni correction “because it is dominated by Holm's method, which is also valid under arbitrary assumptions”.

Interactive

Edit the p-values and watch which results stay significant after each adjustment

Try this: in the worked example, change the p-value for oral A vs control from 0.01331 to 0.03. Holm no longer calls it significant at 0.05, but Benjamini-Hochberg still does.

Worked example in this chapter: six pairwise comparisons of haemoglobin rise, with unadjusted p-values from t-tests that use the pooled SD.

α = 0.05α = 0.01

Unadjusted and adjusted p-values
Comparison	Unadjusted p	Bonferroni	Holm	Benjamini-Hochberg
Oral A vs control	Unadjusted p-value for Oral A vs control	0.080	0.027 significant	0.016 significant
Oral B vs control	Unadjusted p-value for Oral B vs control	0.007 significant	0.005 significant	0.002 significant
IV vs control	Unadjusted p-value for IV vs control	< 0.001 significant	< 0.001 significant	< 0.001 significant
Oral B vs oral A	Unadjusted p-value for Oral B vs oral A	1.000	0.307	0.307
IV vs oral A	Unadjusted p-value for IV vs oral A	0.003 significant	0.002 significant	0.001 significant
IV vs oral B	Unadjusted p-value for IV vs oral B	0.030 significant	0.015 significant	0.008 significant

5of 6 significant with no adjustment

4with Bonferroni

5with Holm

5with Benjamini-Hochberg

Bonferroni: p × 6. Holm: smallest p × 6, next × 5, and so on, never lower than the one before. Benjamini-Hochberg: p × 6 ÷ rank, never higher than the one after.

α itselfBonferroni: α ÷ 6Holm: α ÷ (6 − rank + 1)Benjamini-Hochberg: rank × α ÷ 6

Red dots: significant with Holm. Rings: significant with Benjamini-Hochberg only. Dark dots: not significant with either.

After ANOVA

Tukey and Dunnett: how do they compare groups after ANOVA?

Tukey's test compares every pair of means. Dunnett's test compares each treatment with one control. Both use the error mean square from the ANOVA and control the family-wise error rate for their own family.

Tukey's test is based on the studentized range: the largest group mean minus the smallest, divided by √(MS error ÷ n). If the largest range among the groups is unlikely under the null hypothesis, every smaller difference is judged against the same critical value. In the worked example that critical difference, the honestly significant difference (HSD), is 0.76 g/dL.

Tukey HSD = q_{α; k, df} × √(MS error ÷ n)

Dunnett critical difference = d_{α; k − 1, df} × √(2 MS error ÷ n)

q: critical value of the studentized range for k groups and the error df (3.958 for 4 groups and 20 df)
d: critical value of Dunnett's distribution for k − 1 comparisons with a control (2.540 for 3 comparisons and 20 df)
MS error: within-groups mean square from the ANOVA (0.219 in the worked example)
n: patients per group (6)

Tukey: the R documentation describes intervals “based on the Studentized range statistic, Tukey's ‘Honest Significant Difference’ method”. With unequal group sizes the Tukey-Kramer version uses the two group sizes of each pair.
Dunnett: SPSS describes it as a pairwise t test “that compares a set of treatments against a single control mean”. It does not compare the treatment groups with each other, and in exchange, with fewer comparisons in the family, its critical difference is smaller.

In the worked example the difference between oral A and control is 0.73 g/dL. Tukey's adjusted p is 0.059, and Dunnett's is 0.035. The data are the same. The answer differs because Tukey protects six comparisons and Dunnett three.

Interactive

Change the group means and watch the Tukey and Dunnett intervals

Try this: leave the worked-example means as they are and compare the two intervals for oral A. The Dunnett interval stays above zero, and the wider Tukey interval crosses zero. Then raise the number per group to 10 and watch both intervals narrow.

Made-up teaching example: rise in haemoglobin (g/dL) after 8 weeks in four groups of equal size. Means are rounded from the worked example.

Control mean0.27

Oral A mean1.00

Oral B mean1.28

IV mean2.13

SD within groups0.47

More spread → wider intervals

Patients per group

461020

0.76g/dL: smallest difference Tukey calls significant

0.69g/dL: smallest difference Dunnett calls significant

Oral A vs control: difference 0.73 g/dL. Tukey adjusted p = 0.062, Dunnett adjusted p = 0.037. Tukey protects six comparisons, Dunnett only the three against the control, so Dunnett needs a smaller difference. Degrees of freedom 20; critical values q = 3.958 and d = 2.540.

Tukey simultaneous 95% interval (six comparisons)Dunnett simultaneous 95% interval (three comparisons with control)

Adjusted p-values for every comparison
Comparison	Difference	Tukey p	Dunnett p	Bonferroni p (6 tests)
Oral A vs control	0.73	0.062	0.037	0.084
Oral B vs control	1.01	0.007	0.004	0.008
IV vs control	1.86	< 0.001	< 0.001	< 0.001
Oral B vs Oral A	0.28	0.733	not compared	1.000
IV vs Oral A	1.13	0.002	not compared	0.003
IV vs Oral B	0.85	0.025	not compared	0.031

Other procedures

Scheffé, Games-Howell and LSD: when do they fit?

Each procedure protects a different family, so each needs a different critical difference.

Worked example: the smallest difference in means that each procedure calls significant at 5%. LSD makes no adjustment. Scheffé protects every possible contrast, including averages of groups, so it needs the largest difference when only pairs are compared.

Scheffé: uses the F distribution and, in SPSS's description, “can be used to examine all possible linear combinations of group means” as well as pairs, such as the two oral groups together against the control. Critical difference here: √[(k − 1) × F_{0.05; 3, 20}] × SE = 3.049 × 0.270 = 0.823.
Games-Howell: a pairwise test for groups whose variances differ. SPSS lists it with Tamhane's T2, Dunnett's T3 and Dunnett's C under “Equal Variances Not Assumed”.
LSD: unadjusted t-tests using the pooled SD. It does not control the family-wise error rate.
Student-Newman-Keuls and Duncan: stepwise range tests that SPSS still offers. This chapter does not recommend them.

A different error rate

False discovery rate: when may some false positives be accepted?

When you test many outcomes to decide what to study next, protecting against every single false positive can leave almost nothing significant. The false discovery rate allows a planned share of false positives among the results you call significant.

The family-wise error rate asks whether a study has any false positive. The false discovery rate asks what share of the positive results are false, averaged over many studies. Controlling the FDR at 5% means that, on average, no more than 5% of the significant results are false.

Benjamini-Hochberg: largest rank i with p_(i) ≤ i × q ÷ m

Sort the p-values from smallest to largest, find the largest rank that meets this condition, and call that p-value and every smaller one significant.

q: chosen false discovery rate, often 0.05
m: number of tests

The R documentation states that the BH method controls “the expected proportion of false discoveries amongst the rejected hypotheses”, and that it is “a less stringent condition than the family-wise error rate”, so it usually calls more results significant. The Benjamini-Yekutieli variant controls the FDR under any form of dependence between the tests, and is more conservative.

When FDR fits: screening many biomarkers, genes, questionnaire items or secondary outcomes to pick candidates for a later study, reported as exploratory. When FWER fits: a confirmatory claim that one treatment works, such as the primary comparisons of a trial, where any false positive may change practice.

Interactive

Change the number of real effects and watch the family-wise error rate and the false discovery rate

Try this: set the number of real effects to 0. Every positive result is then false, so the family-wise error rate and the false discovery rate become the same number. Then set it to 10 and compare the share of real effects that Bonferroni and Benjamini-Hochberg detect.

Simulation: 400 studies, each testing 20 outcomes at α 0.05. An outcome with a real effect has a true difference of the size you choose, measured in standard errors.

Outcomes with a real effect5 of 20

Size of each real effect

2 SE3 SE4 SE

Error rates and power by method
Method	Studies with at least one false positive	False positives among positive results (average)	Real effects detected (average)
No adjustment	54.3%	13.5%	83.9%
Bonferroni	3.3%	1.1%	48.1%
Holm	3.3%	1.1%	49.1%
Benjamini-Hochberg	12.8%	3.2%	63.0%

Studies with at least one false positive (family-wise error rate)Average share of false positives among positives (false discovery rate)

Timing

Planned or post hoc: does it matter when you chose the comparison?

Yes. A comparison written in the protocol before the data were seen is a planned (a priori) comparison. A comparison chosen after looking at the results is post hoc.

SPSS explains the difference in its One-Way ANOVA documentation: contrasts “are tests set up before running the experiment, and post hoc tests are run after the experiment has been conducted”. A few planned comparisons form a small family and need less adjustment. A comparison picked because the difference looked large belongs to the family of every comparison you could have picked.

ICH E9 lists “the choice of a critical treatment contrast” as one way to avoid or reduce multiplicity. CONSORT 2025 (item 21d) asks trial reports to describe additional analyses “distinguishing prespecified from post hoc”.

Non-parametric data

After a significant Kruskal-Wallis test: Dunn's test

Dunn's test compares pairs of groups using the ranks from one ranking of all the groups together, then the p-values are adjusted for the number of comparisons.

Dunn (1964) proposed using the rank sums “from a combined ranking of k independent samples” as an alternative to making separate rankings for each pair of samples. Running a Mann-Whitney test for each pair would re-rank the data every time.

z = (mean rank_i − mean rank_j) ÷ √([N(N + 1) ÷ 12 − Σ(t³ − t) ÷ 12(N − 1)] × (1/n_i + 1/n_j))

N: total number of patients (24)
t: number of values sharing a tied rank; the tie term is 60 in the worked example

Worked example data: Kruskal-Wallis H = 16.17, df 3, p = 0.001. Mean ranks: control 4.58, oral A 10.83, oral B 13.92, IV 20.67. The denominator is √[(50 − 0.217) × 1/3] = 4.07. IV vs control: z = 16.08 ÷ 4.07 = 3.95, unadjusted p < 0.001, Holm-adjusted p < 0.001. Oral B vs control: z = 2.29, p = 0.022, Holm-adjusted p = 0.088. With Holm adjustment, only IV vs control is significant. The rank test uses less of the information in these small groups than Tukey did.

Decide

Which multiple comparison procedure fits your study?

Answer the questions in order.

Is the outcome a measurement that is roughly normal in each group?

Which procedure: all possible answers

Dunn's test with Holm or Bonferroni adjustment: Uses the ranks from the Kruskal-Wallis ranking of all groups together, then adjusts the pairwise p-values.
Tukey's HSD (Tukey-Kramer if group sizes differ): Controls the family-wise error rate for all pairwise comparisons, using the studentized range.
Games-Howell test: A pairwise test for groups with unequal variances. SPSS also offers Tamhane T2, Dunnett T3 and Dunnett C.
Dunnett's test: Compares each treatment with the control only. Fewer comparisons than Tukey, so it needs a smaller difference.
Planned comparisons with Holm (or Bonferroni) adjustment: Adjust only for the comparisons you planned. Holm gives the same protection as Bonferroni and rejects at least as many hypotheses.
Benjamini-Hochberg false discovery rate: Limits the expected share of false positives among the positive results. Label the analysis exploratory.

Comparison

Pairs of ideas that residents confuse

Family-wise error rate or false discovery rate?

Feature	Family-wise error rate	False discovery rate
Controls	Probability of one or more false positives	Expected share of false positives among significant results
Typical methods	Bonferroni, Holm, Tukey, Dunnett, Scheffé	Benjamini-Hochberg, Benjamini-Yekutieli
Suits	Confirmatory claims, a few planned comparisons	Screening many outcomes or markers
Worked example, 6 pairs at 5%	Holm: 5 significant, Bonferroni: 4	Benjamini-Hochberg: 5 significant
Label the analysis	Confirmatory if planned	Exploratory

Tukey or Dunnett?

Feature	Tukey	Dunnett
Family	Every pair of groups: k(k − 1) ÷ 2	Each treatment against one control: k − 1
Distribution	Studentized range	Dunnett's multivariate t
Critical difference, worked example	0.756 g/dL	0.686 g/dL
Oral A vs control	Adjusted p = 0.059	Adjusted p = 0.035
Compares treatments with each other	Yes	No

Bonferroni or Holm?

Feature	Bonferroni	Holm
Threshold	α ÷ m for every test	α ÷ m, α ÷ (m − 1), ... α, in order of p
Family-wise error rate	Controlled	Controlled
Significant results	Never more than Holm	At least as many as Bonferroni
Worked example	4 of 6	5 of 6
Adjusted p for oral A vs control	6 × 0.0133 = 0.080	0.027

Planned or post hoc comparisons?

Feature	Planned (a priori)	Post hoc
Chosen	In the protocol, before the data	After seeing the results
Family	Only the planned comparisons	Every comparison that could have been chosen
Adjustment	For the planned comparisons	Tukey, Scheffé, or an adjustment for all pairs
Reporting	Confirmatory	Labelled as post hoc (CONSORT 2025 item 21d)

The formula

Worked example: from ANOVA to adjusted comparisons

Made-up teaching data. Rise in haemoglobin (g/dL) at 8 weeks, 6 women in each group.

Group	Values	Mean	SD
Control (dietary advice)	0.4, −0.1, 0.4, 0.6, 0.3, 0.0	0.27	0.27
Oral A (daily oral iron)	1.1, 0.4, 1.8, 0.6, 1.9, 0.2	1.00	0.72
Oral B (alternate-day oral iron)	0.9, 1.3, 0.9, 1.2, 1.3, 2.1	1.28	0.44
IV iron	1.9, 2.3, 2.1, 2.5, 2.3, 1.7	2.13	0.29

Step 1: one-way ANOVA

SS between = 10.715 (df 3), SS within = 4.375 (df 20). MS within = 4.375 ÷ 20 = 0.219. F = 3.572 ÷ 0.219 = 16.33, p < 0.001. Levene test p = 0.06.

Step 2: the standard error of a difference

SE = √(MS within × 2 ÷ n) = √(0.219 × 2 ÷ 6) = 0.270

Step 3: every comparison, every method

Comparison	Difference	Unadjusted p	Bonferroni	Holm	BH	Tukey	Dunnett
IV − control	1.87	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001
IV − oral A	1.13	< 0.001	0.003	0.002	0.001	0.002	not compared
Oral B − control	1.02	0.001	0.007	0.005	0.002	0.006	0.003
IV − oral B	0.85	0.005	0.030	0.015	0.008	0.024	not compared
Oral A − control	0.73	0.013	0.080	0.027	0.016	0.059	0.035
Oral B − oral A	0.28	0.307	1.000	0.307	0.307	0.723	not compared

Holm thresholds for the sorted p-values: 0.0083, 0.010, 0.0125, 0.0167, 0.025, 0.05. The fifth p-value (0.0133) is below 0.025, the sixth (0.307) is not.
Tukey: q = 3.958, HSD = 3.958 × √(0.219 ÷ 6) = 3.958 × 0.191 = 0.756 g/dL. Oral A − control, 0.73, is smaller.
Dunnett: d = 2.540, critical difference = 2.540 × 0.270 = 0.686 g/dL. Oral A − control, 0.73, is larger. Simultaneous 95% intervals: oral A 0.05 to 1.42, oral B 0.33 to 1.70, IV 1.18 to 2.55 g/dL.

What changed the answer for oral A: the question. If the protocol asked only whether each treatment beats dietary advice, Dunnett is the matching test and oral A differs from control. If the protocol asked which of the four groups differ from each other, Tukey is the matching test and the oral A vs control difference is not significant. Choosing the test after seeing both answers would itself be a multiple comparison.

All results computed in Python (SciPy 1.15.3: tukey_hsd, dunnett, kruskal. statsmodels 0.14.5: multipletests).

In a real paper

Post hoc tests in two published trials

Open-access paper · CC BY

Roldan CJ, Huh B, Song J, Nieto Y, Osei J, Chai T, et al. Methylene blue for intractable pain from oral mucositis related to cancer treatment: a randomized phase 2 clinical trial. BMC Med. 2022;20(1):377.

doi:10.1186/s12916-022-02579-8 · PMID 36324139 · Free full text (PMC9632023)

The study: A single-blind randomised phase 2 trial in 60 patients with oral mucositis pain during cancer treatment. Three concentrations of methylene blue rinse plus conventional therapy were compared with conventional therapy alone (16, 15, 14 and 15 patients). The change in pain score was compared across the four arms by ANOVA, followed by pairwise comparisons.

What the authors wrote

“Tukey pairwise comparisons indicated that the differences were significant for the three arms of CTx+MB compared with the CTx alone arm (P=0.0071, P=0.0506, P=0.0114 Figs. 2 and 3).”

In plain words

After ANOVA showed that the arms differed, the authors used Tukey's test, which adjusts for all six pairs of arms. The three P-values are the adjusted P-values for each methylene blue arm against conventional therapy alone. Two are below 0.05. The middle one, 0.0506, is just above 0.05, although the sentence calls all three significant. The figure legend in the same paper says the arms were “significantly or marginally better”. The methods state that “A P-value less than 0.05 indicated statistical significance”.

Reported	Value	What it means
Pain score reduction at day 2, mean ± SD	1.69 ± 3.09 (CTx alone); 5.2 ± 2.81, 4.54 ± 2.93, 5.15 ± 2.64 (MB 0.025%, 0.05%, 0.1%)	All three methylene blue arms reduced pain by about 3 points more than conventional therapy alone
ANOVA across four arms	P = 0.0034	At least one arm differs
Tukey, each MB arm vs CTx alone	P = 0.0071, 0.0506, 0.0114	Adjusted P-values. 0.0506 is not below the stated 0.05 threshold
Tukey, between MB concentrations (pain)	P = 0.9299, 1.0000, 0.9477	No evidence that the concentrations differ from each other
Rule for running Tukey	performed if the overall group showed significant differences	Pairwise tests only after a significant ANOVA

Check these when you read it

Is every P-value called significant actually below the stated α? Here one is 0.0506.
Which family did the question need? If only each concentration against conventional therapy mattered, a comparison with control (Dunnett) matches the question. Tukey also covers the three comparisons between concentrations.
Are the group sizes large enough for the comparisons? Here 14 to 16 patients per arm in a phase 2 trial.
Are the differences reported with confidence intervals, not only P-values?

Open-access paper · CC BY

Liu-Ambrose T, Falck RS, Dao E, Best JR, Davis JC, Bennett K, et al. Effect of exercise training or complex mental and social activities on cognitive function in adults with chronic stroke: a randomized clinical trial. JAMA Netw Open. 2022;5(10):e2236510.

doi:10.1001/jamanetworkopen.2022.36510 · PMID 36227593 · Free full text (PMC9561961)

The study: A three-group randomised trial in 120 adults with chronic stroke: multicomponent exercise (EX), cognitive and social enrichment (ENRICH), or a balance and tone control (BAT), for 6 months. The research question was whether each intervention improved cognitive function compared with the control, so only two comparisons were planned.

What the authors wrote

“We performed 2 planned contrasts using the Dunnett Test, a multiple comparison procedure, to assess differences in ADAS-Cog-Plus at 6 months between: (1) EX vs BAT; and (2) ENRICH vs BAT.”

In plain words

The family was fixed in advance: each intervention against the control, two comparisons, adjusted with Dunnett's test at an overall two-sided α of .05. Exercise was not compared with enrichment. The authors also set an allocation ratio of 2:2:3 (EX:ENRICH:BAT), and state that it “accounts for the 2 planned contrasts”: the control group is used in both comparisons, so it was made larger. The superscript reference number after “Dunnett Test,” is omitted from the quotation.

Reported	Value	What it means
Randomised	EX 34, ENRICH 34, BAT 52	Larger control group, because both comparisons use it
EX vs BAT at 6 months	−0.24 (95% CI −0.43 to −0.04), P = .02	Exercise improved ADAS-Cog-Plus compared with control (a negative difference is an improvement)
ENRICH vs BAT at 6 months	−0.11 (95% CI −0.30 to 0.09), P = .27	No evidence that enrichment differed from control
EX vs BAT at 12 months	−0.08 (95% CI −0.29 to 0.12), P = .43	The difference did not persist after the intervention ended
Overall α	.05 (2-sided)	The error rate for the family of two planned comparisons

Check these when you read it

Were the comparisons named before the results were known? Here they are described as planned and the trial was registered (NCT01916486).
Does the family match the question? Two comparisons with control, so Dunnett rather than Tukey.
Did the design allow for the multiple comparisons? Here the allocation ratio did.
Are post hoc analyses labelled? The 13-item ADAS-Cog results are described as a post hoc analysis.

Software

How to run multiple comparisons

Checked against the IBM SPSS Statistics documentation, the R documentation and CRAN reference manuals, and the jamovi (jmv) reference manual in September 2026.

SPSS

Analyze > Compare Means > One-Way ANOVA. Put Hb rise in the dependent list and group as the factor.
Post Hoc, Equal Variances Assumed: tick Tukey for all pairs, or Dunnett for comparisons with a control. For Dunnett, the last category is the default control category, and you can choose the first category instead. Choose 2-sided, < Control or > Control. Bonferroni, Sidak and Scheffé are in the same list.
Post Hoc, Equal Variances Not Assumed: Tamhane's T2, Dunnett's T3, Games-Howell or Dunnett's C.
Non-parametric: Analyze > Nonparametric Tests > Independent Samples. On the Settings tab, customise the tests, choose Kruskal-Wallis 1-way ANOVA (k samples), and request multiple comparisons: all pairwise, or stepwise step-down.

R

d$group <- factor(d$group, levels = c("Control", "OralA", "OralB", "IV"))
fit <- aov(hb ~ group, data = d)
summary(fit)

TukeyHSD(fit, conf.level = 0.95)                       # column "p adj"
pairwise.t.test(d$hb, d$group, p.adjust.method = "holm")  # pooled SD by default
p.adjust(c(0.01331, 0.00122, 0.000001, 0.30657, 0.00044, 0.00506),
         method = "BH")                                # or "bonferroni", "holm"

library(multcomp)                                      # Dunnett: first level is the control
summary(glht(fit, linfct = mcp(group = "Dunnett")))
library(DescTools)
DunnettTest(hb ~ group, data = d, control = "Control")

kruskal.test(hb ~ group, data = d)
library(FSA)
dunnTest(hb ~ group, data = d, method = "holm")       # Dunn's test; "holm" is also the default

jamovi

ANOVA analysis, Post Hoc Tests: corrections No correction, Tukey (the default), Scheffé, Bonferroni and Holm. Dunnett is not in this list.
One-Way ANOVA analysis: Welch's test is the default, with Games-Howell (unequal variances) or Tukey (equal variances) post hoc tests.
One-Way ANOVA (non-parametric), the Kruskal-Wallis test: pairwise comparisons can be requested. Version 2.8.0 of the jmv package also offers Dunn pairwise comparisons.

jmv::ANOVA(formula = hb ~ group, data = d, postHoc = ~ group,
           postHocCorr = "holm")   # the default is "tukey"
jmv::anovaOneW(formula = hb ~ group, data = d, phMethod = "gamesHowell")
jmv::anovaNP(formula = hb ~ group, data = d, pairsDunn = TRUE)

Reporting

How to report multiple comparisons

Name the family, the procedure and why it matches the question. Give the adjusted p-values and simultaneous confidence intervals.

In the protocol, write which comparisons answer the research question: all pairs, each treatment with control, or a few planned contrasts.
In the methods, name the procedure (Tukey, Dunnett, Holm, Benjamini-Hochberg, Dunn with Holm) and the overall α.
Give each difference with its adjusted p-value and its 95% simultaneous confidence interval.
Say which analyses were planned and which were post hoc.
If you did not adjust, say why, for example because the analysis is exploratory.

A sentence for your thesis, using this chapter's teaching numbers:

Haemoglobin rise at 8 weeks differed between the four groups (one-way ANOVA, F(3, 20) = 16.33, p < 0.001). As planned in the protocol, each iron regimen was compared with dietary advice alone using Dunnett's test. Compared with dietary advice, the rise was greater with daily oral iron (difference 0.73 g/dL, 95% simultaneous CI 0.05 to 1.42, adjusted p = 0.035), alternate-day oral iron (1.02 g/dL, 0.33 to 1.70, p = 0.003) and intravenous iron (1.87 g/dL, 1.18 to 2.55, p < 0.001).

The SAMPL guidelines (Lang and Altman) ask authors to “indicate whether and how any allowance or adjustments were made for multiple comparisons (performing multiple hypothesis tests on the same data)”.

By specialty

Multiple comparison problems by specialty

Duration of analgesia with bupivacaine alone or with one of three adjuvants. The question is whether each adjuvant prolongs the block.

Each adjuvant against bupivacaine alone is a comparison with one control: Dunnett's test.

Common myths

Common mistaken beliefs

Myth: A significant ANOVA shows which groups differ.
It shows only that at least one mean differs. A multiple comparison procedure shows which pairs differ.
Myth: Bonferroni is the best correction because it is the strictest.
Holm controls the same family-wise error rate and rejects at least as many hypotheses. The R documentation says Bonferroni is dominated by Holm.
Myth: Tukey and Dunnett give the same answer.
In the worked example, oral A vs control has adjusted p = 0.059 with Tukey and 0.035 with Dunnett, because the families differ.
Myth: Every p-value in a paper must be adjusted.
Rothman (1990) argued that no adjustment is needed, and Perneger (1998) that Bonferroni adjustments answer a largely irrelevant question. Bender and Lange (2001) state that adjustment is required in confirmatory studies when several tests are combined in one conclusion. Exploratory analyses can be left unadjusted if they are labelled.
Myth: After Kruskal-Wallis, run a Mann-Whitney test for each pair with no correction.
Adjust for the number of pairs, or use Dunn's test, which uses the ranking of all groups together.
Myth: The LSD test controls false positives because it follows a significant ANOVA.
SPSS describes the LSD test as t-tests with no adjustment to the error rate for multiple comparisons.

Why this matters to you

Five consequences of ignoring multiple comparisons

Protocol sent back
ICH E9 expects the method of controlling type I error, or the reason for not adjusting, in the analysis plan. "Appropriate post hoc tests will be used" does not say which.
False findings in the thesis
With six unadjusted t-tests, the chance of at least one false positive is far above 5%, and a false positive becomes a conclusion.
Questions you cannot answer at the viva
"Why Tukey and not Dunnett?" needs an answer in terms of the family of comparisons your question required.
Misreading papers
A paper with 20 outcomes and three p-values just below 0.05 may have found nothing. Check whether the authors adjusted or labelled the analyses as exploratory.
Harm to patients
An ineffective regimen can enter practice because one of many unadjusted comparisons was significant by chance.

History

History of multiple comparison procedures

1949
John Tukey
Published "Comparing individual means in the analysis of variance" in Biometrics, proposing a procedure for dividing treatments into distinguishable groups after the F-test.
1953
H. Scheffé
Published "A method for judging all contrasts in the analysis of variance" in Biometrika.
1955
C. W. Dunnett
Published "A multiple comparison procedure for comparing several treatments with a control" in the Journal of the American Statistical Association.
1956
Kramer
Published "Extension of multiple range tests to group means with unequal numbers of replications" in Biometrics. Tukey's test with this modification is called the Tukey-Kramer test.
1961 and 1964
O. J. Dunn
Published "Multiple comparisons among means" (1961) and "Multiple comparisons using rank sums" (1964). The Bonferroni method is also called "Dunn's method", and the rank-sum procedure is the test now used after Kruskal-Wallis.
1979
S. Holm
Published "A simple sequentially rejective multiple test procedure" in the Scandinavian Journal of Statistics.
1990
K. J. Rothman
Argued in Epidemiology that "No adjustments are needed for multiple comparisons", because reducing type I error increases type II error.
1995
Y. Benjamini and Y. Hochberg
Published "Controlling the false discovery rate" in the Journal of the Royal Statistical Society, Series B, the procedure now called Benjamini-Hochberg.
1998
Thomas Perneger
Wrote in the BMJ that Bonferroni adjustments are "at best, unnecessary and, at worst, deleterious to sound statistical inference".

Guidelines

Multiple comparisons in guidelines

ICH E9 (1998), section 5.6

“In confirmatory analyses, any aspects of multiplicity which remain after steps of this kind have been taken should be identified in the protocol; adjustment should always be considered and the details of any adjustment procedure or an explanation of why adjustment is not thought to be necessary should be set out in the analysis plan.”

ICH E9 (1998), section 2.2.5

With multiple primary variables, “the method of controlling type I error should be given in the protocol”.

SAMPL guidelines (Lang and Altman)

“Indicate whether and how any allowance or adjustments were made for multiple comparisons (performing multiple hypothesis tests on the same data).”

CONSORT 2025, item 21d

“Methods for any additional analyses (eg, subgroup and sensitivity analyses), distinguishing prespecified from post hoc.”

Reading a paper

Five checks on multiple comparisons in any paper

After ANOVA or Kruskal-Wallis, is the post hoc procedure named?
Does the procedure match the question: all pairs (Tukey), comparisons with a control (Dunnett), a few planned contrasts (Holm)?
Are the reported p-values adjusted, and is every "significant" p-value below the stated α?
With many outcomes or subgroups, did the authors adjust, or label the analyses as exploratory?
Are planned and post hoc comparisons kept apart?

Why is it called that?

Word surgery

“Discovery”

In everyday speech

Finding something new and true.

In research

In "false discovery rate", any result called significant. A discovery can be false.

“Control”

In everyday speech

Power over something, or keeping it in check.

In research

The reference group in Dunnett's test. To control an error rate means to keep its probability at or below the chosen level.

Memory trick

All pairs: Tukey. Against a control: Dunnett. A few planned comparisons: Holm. Many outcomes to screen: Benjamini-Hochberg. Ranks: Dunn.

Family

Latin familia: the servants in a household, then the household itself

In statistics, the set of comparisons that answer one research question together.

Post hoc

Latin: "after this"

A comparison chosen after seeing the data.

Honest

Latin honestus: "honorable, respected"

Tukey's test was originally called the honestly significant difference (HSD) test.

Contrast

Latin contra "against" + stare "to stand"

A comparison of group means, such as one group against another or two groups together against a third.

Control

Medieval Latin contrarotulus: "a counter, register", a duplicate record for checking accounts

The group that treatments are compared against. The term "control group" in scientific experiments is recorded from 1952.

Exam answers

Short-note outlines

Multiple comparison (post hoc) tests after ANOVA

Problem: k groups give k(k − 1) ÷ 2 pairs. Separate t-tests at 5% raise the chance of at least one false positive (up to 1 − 0.95^m: 26.5% for 6 independent tests).
ANOVA tests whether any mean differs. Post hoc tests show which pairs differ while controlling the family-wise error rate.
Tukey HSD: all pairs, studentized range, HSD = q × √(MS error ÷ n). Tukey-Kramer for unequal n.
Dunnett: each treatment against one control, fewer comparisons, smaller critical difference.
Bonferroni: α ÷ m for each test. Holm: stepwise, same protection, at least as many rejections.
Scheffé: all possible contrasts, most conservative for pairs. Games-Howell: unequal variances. LSD: no adjustment.
Non-parametric: Dunn's test after Kruskal-Wallis, with Holm or Bonferroni adjustment.
Example: four groups of 6. Oral A vs control: Tukey p = 0.059, Dunnett p = 0.035.

Family-wise error rate and false discovery rate

Feature	Family-wise error rate	False discovery rate
Definition	P(one or more false positives)	Expected share of false positives among rejected hypotheses
Methods	Bonferroni, Holm, Tukey, Dunnett	Benjamini-Hochberg (1995), Benjamini-Yekutieli
Stringency	Stricter	Less strict, more significant results
Use	Confirmatory comparisons	Screening many outcomes, markers, genes

BH steps: sort m p-values, compare the i-th smallest with i × q ÷ m, and call significant every p-value up to the largest one that passes.

When not to adjust: exploratory analyses, labelled as such (Rothman 1990, Perneger 1998). Confirmatory studies combining several tests in one conclusion need adjustment (Bender and Lange 2001, ICH E9 section 5.6).

Planned (a priori) and post hoc comparisons

Planned: chosen in the protocol before data collection, few in number, adjusted only for the planned family. Example: each intervention against control (Liu-Ambrose et al., 2022, two contrasts with Dunnett's test).
Post hoc: chosen after the results, so the family includes every comparison that could have been chosen. Use Tukey for pairs or Scheffé for any contrast.
ICH E9: choosing a critical treatment contrast reduces multiplicity. CONSORT 2025 item 21d: distinguish prespecified from post hoc analyses.
A planned design can allow for the comparisons, for example a larger control group when several treatments are compared with it.

The one thing to remember

Decide which comparisons answer your question before you see the data, and choose the procedure that protects exactly that family.

All worked examples and interactive elements use made-up or simulated numbers for teaching. Every statistic was computed in Python (SciPy 1.15.3, statsmodels 0.14.5) and checked against the interactive elements.

Sources: Roldan CJ et al. BMC Med 2022;20(1):377 (doi:10.1186/s12916-022-02579-8, PMID 36324139). Liu-Ambrose T et al. JAMA Netw Open 2022;5(10):e2236510 (doi:10.1001/jamanetworkopen.2022.36510, PMID 36227593). Tukey JW. Comparing individual means in the analysis of variance. Biometrics 1949;5(2):99–114 (doi:10.2307/3001913, PMID 18151955). Scheffé H. Biometrika 1953;40:87–110 (doi:10.1093/biomet/40.1-2.87). Dunnett CW. J Am Stat Assoc 1955;50(272):1096–121 (doi:10.1080/01621459.1955.10501294). Kramer CY. Biometrics 1956;12(3):307 (doi:10.2307/3001469). Dunn OJ. J Am Stat Assoc 1961;56(293):52–64 (doi:10.1080/01621459.1961.10482090). Dunn OJ. Technometrics 1964;6(3):241–52 (doi:10.1080/00401706.1964.10490181). Holm S. Scand J Stat 1979;6:65–70. Rothman KJ. Epidemiology 1990;1(1):43–6 (doi:10.1097/00001648-199001000-00010, PMID 2081237). Benjamini Y, Hochberg Y. J R Stat Soc Series B 1995;57(1):289–300 (doi:10.1111/j.2517-6161.1995.tb02031.x). Benjamini Y, Yekutieli D. Ann Stat 2001;29:1165–88 (doi:10.1214/aos/1013699998). Perneger TV. BMJ 1998;316(7139):1236–8 (doi:10.1136/bmj.316.7139.1236, PMID 9553006). Bender R, Lange S. J Clin Epidemiol 2001;54(4):343–9 (doi:10.1016/s0895-4356(00)00314-0, PMID 11297884). Lee S, Lee DK. What is the proper way to apply the multiple comparison test? Korean J Anesthesiol 2018;71(5):353–60 (doi:10.4097/kja.d.18.00242, PMID 30157585; correction 2020). ICH E9 Statistical Principles for Clinical Trials, 5 February 1998. Lang T, Altman D. SAMPL Guidelines (EQUATOR Network); Int J Nurs Stud 2015;52(1):5–9. Hopewell S et al. CONSORT 2025 statement. BMJ 2025;389:e081123. IBM SPSS Statistics documentation (One-Way ANOVA, One-Way ANOVA Post Hoc Tests, Independent-Samples Nonparametric Tests). R documentation for TukeyHSD, p.adjust, pairwise.t.test and kruskal.test (stats 4.5.0). CRAN reference manuals: multcomp 1.4-32, DescTools 0.99.60, FSA 0.10.1, jmv 2.8.0. Etymologies: Online Etymology Dictionary, accessed 15 September 2026.

Further reading: Jawlik A. Statistics from A to Z: Confusing Concepts Clarified. Hoboken: Wiley; 2016. Indrayan A. Medical Biostatistics. Boca Raton: CRC Press.

Why not run a t-test for every pair of groups?

What is a family, and what is the family-wise error rate?

How do Bonferroni and Holm share out the 5%?

Edit the p-values and watch which results stay significant after each adjustment

Tukey and Dunnett: how do they compare groups after ANOVA?

Change the group means and watch the Tukey and Dunnett intervals

Scheffé, Games-Howell and LSD: when do they fit?

False discovery rate: when may some false positives be accepted?

Change the number of real effects and watch the family-wise error rate and the false discovery rate

Planned or post hoc: does it matter when you chose the comparison?

After a significant Kruskal-Wallis test: Dunn's test

Which multiple comparison procedure fits your study?

Pairs of ideas that residents confuse

Family-wise error rate or false discovery rate?

Tukey or Dunnett?

Bonferroni or Holm?

Planned or post hoc comparisons?

Worked example: from ANOVA to adjusted comparisons

Step 1: one-way ANOVA

Step 2: the standard error of a difference

Step 3: every comparison, every method

Post hoc tests in two published trials

How to run multiple comparisons

SPSS

R

jamovi

How to report multiple comparisons

Multiple comparison problems by specialty

Common mistaken beliefs

Five consequences of ignoring multiple comparisons

Protocol sent back

False findings in the thesis

Questions you cannot answer at the viva

Misreading papers

Harm to patients

History of multiple comparison procedures

John Tukey

H. Scheffé

C. W. Dunnett

Kramer

O. J. Dunn

S. Holm

K. J. Rothman

Y. Benjamini and Y. Hochberg

Thomas Perneger

Multiple comparisons in guidelines

Five checks on multiple comparisons in any paper

Word surgery

“Discovery”

“Control”

Family

Post hoc

Honest

Contrast

Control

Short-note outlines

Multiple comparison (post hoc) tests after ANOVA

Family-wise error rate and false discovery rate

Planned (a priori) and post hoc comparisons

Related Stateazy articles