If You Flip Enough Coins, One Will Land on Its Edge

The Problem First

You're reading a paper at journal club. The authors studied a new diabetes drug. Their results section reports:

• Primary endpoint (HbA1c): p = 0.08. Not significant.
• Fasting glucose: p = 0.12. Not significant.
• Post-prandial glucose: p = 0.35. Not significant.
• Body weight: p = 0.04. Significant!
• Triglycerides: p = 0.03. Significant!
• LDL cholesterol: p = 0.41. Not significant.
• Quality of life score: p = 0.02. Significant!

The conclusion reads: "The drug significantly improved body weight, triglycerides, and quality of life in patients with type 2 diabetes."

Sounds reasonable? It's a statistical crime scene.

They tested 7 outcomes. The primary endpoint failed. But they kept looking — and surprise, 3 out of 7 secondary endpoints crossed p<0.05. The paper presents these as genuine discoveries.

Here's the mathematics they're hiding from you:

They had a nearly 1-in-3 chance of finding "something significant" even if the drug does absolutely nothing. And they found 3 things. Sounds impressive until you do the maths.

This is the multiplicity problem. And it is the single most common way medical research lies to you without technically lying.

The Concept — Before the Jargon

The Loaded Dice Analogy

You suspect a die is loaded. You roll it once and get a 6. Suspicious? Not really — 1-in-6 chance.

You roll it 50 times and get ten 6s. Now you're suspicious.

But what if I gave you 20 different dice and asked you to find the loaded one? You roll each die once. By pure chance, some will show 6. You point at those and say "These are loaded!"

You haven't found loaded dice. You've found the ones that got lucky on a single roll.

That's what happens when a study tests 20 outcomes and reports the ones that crossed p<0.05. The "significant" results aren't discoveries — they're the dice that got lucky.

The Birthday Problem Version

In a room of 23 people, there's a >50% chance two share a birthday. Not because anything special is happening — just because with enough comparisons, coincidences become likely.

In a study testing 20 endpoints, there's a 64% chance at least one crosses p<0.05. Not because the drug works — just because with enough tests, false positives become likely.

Multiplicity is the birthday problem of clinical research.

Now the Formal Framework

What is "Multiplicity" Anyway? Why Not Just Say "Multiple Testing"?

Word Surgery:

• Multi- — Latin multus = "many, much"
• -plic- — Latin plicare = "to fold" (same root as "complicate" = folded together, "duplicate" = folded in two, "replicate" = folded back)
• -ity — Latin suffix turning adjective into noun (a state or condition)
• Literal meaning: "The state of being many-folded" — i.e., having many layers, many instances

Why This Name? Statisticians didn't call it "multiple testing problem" (though informally they sometimes do). They chose multiplicity — a more formal, abstract term — because the problem isn't just about running many tests. It's about the many-foldedness of the entire analysis: multiple endpoints, multiple timepoints, multiple subgroups, multiple looks at the data, multiple treatment arms. The problem is the state of having many things, not just the act of testing many things. The word captures the condition, not the activity.

The term became standard in FDA/ICH regulatory language by the 1990s, especially in ICH E9 (1998). The FDA's 2017 guidance is literally titled "Multiple Endpoints in Clinical Trials."

The "Aha" Bridge: So... "multiplicity" doesn't mean "we ran many tests." It means "our entire analysis has many folds — many places where a false positive can hide." Every fold is a hiding spot for a false positive. The more folds, the more hiding spots.

Naming Family:

• Multiple comparisons — older, narrower term (just about pairwise group comparisons)
• Multiple testing — informal synonym
• Multiplicity adjustment — the correction you apply
• Family-wise error — the inflated error rate caused by multiplicity
• Alpha inflation — what happens when you don't adjust

How Does Multiplicity Inflate Error?

Every time you perform a statistical test at alpha=0.05, you accept a 5% chance of a false positive (Type I error). That's fine for ONE test.

But clinical trials don't test one thing. They test:

• Multiple endpoints (primary, secondary, exploratory)
• Multiple timepoints (6 months, 1 year, 2 years, 5 years)
• Multiple subgroups (age, sex, disease severity, geography)
• Multiple treatments (dose groups, combinations)
• Multiple interim analyses (DSMB looks at data 3 times during the trial)

Each test is another roll of the dice. The cumulative false positive rate inflates rapidly.

What Do We Call This Inflated Error Rate?

Word Surgery:

• Family — not biological family. Here, "family" = a group of related statistical tests (a "family of hypotheses")
• Wise — Old English suffix meaning "in the manner of" or "with respect to" (like "otherwise," "likewise," "clockwise")
• Error Rate — the probability of making a wrong decision
• Literal meaning: "The error rate with respect to the entire family of tests"

Why This Name? The concept was formalised by John Tukey in the 1950s during his work on multiple comparisons in ANOVA. Before Tukey, people talked about the "per-comparison error rate" (alpha for each individual test). Tukey pointed out that what matters is the error rate across the whole family of comparisons. He needed a term to distinguish "error rate for one test" from "error rate for all tests combined." So: per-comparison vs. family-wise.

The "Aha" Bridge: So... think of it like insurance. Your per-comparison error rate is like the risk on one policy. The family-wise error rate is the risk across your entire portfolio. Even if each policy has only 5% chance of a claim, a portfolio of 100 policies will almost certainly have claims. FWER is the portfolio risk of your analysis.

Naming Family:

• Per-comparison error rate (PCER) — alpha for a single test (the contrast)
• False discovery rate (FDR) — a less strict alternative to FWER (Benjamini-Hochberg, 1995), controls the proportion of false positives among discoveries rather than the probability of any false positive
• Experiment-wise error rate — older synonym of FWER

Here's the FWER table:

At 100 tests, you are virtually guaranteed to find something "significant" even if nothing is real.

The Three Levels of Multiplicity

Level 1: Multiple endpoints in one study Testing HbA1c, fasting glucose, weight, lipids, quality of life, adverse events all at alpha=0.05 each.

Level 2: Multiple looks at accumulating data Interim analyses where a DSMB examines data at 50%, 75%, and 100% enrollment. Each look inflates Type I error.

Level 3: Multiple subgroups Was the drug better in men? Women? Young? Old? Severe? Mild? Asian? European? Each subgroup analysis is another test.

The Regulatory Dimension — How FDA Handles This

FDA's Position: Non-Negotiable

The FDA does not accept multiplicity-unadjusted secondary endpoints as confirmatory evidence. Period. ICH E9 (Statistical Principles for Clinical Trials) and the 2017 FDA Guidance on Multiple Endpoints explicitly state: the overall Type I error rate must be controlled.

The Methods (What You'll See in SAPs)

1. Bonferroni Correction — The Blunt Hammer

Word Surgery:

• Bonferroni — Carlo Emilio Bonferroni (1892-1960), Italian mathematician
• Correction — Latin corrigere = "to make straight, to set right" (cor- = together + regere = to guide/rule)
• Literal meaning: "Bonferroni's method for setting the alpha straight"

Why This Name? Bonferroni published his inequality in 1936: the probability that at least one of several events occurs is at most the sum of their individual probabilities. This is a general probability theorem, not originally about multiple testing. It was Olive Jean Dunn in 1961 who applied Bonferroni's inequality to the multiple comparisons problem and turned it into a practical correction method. So technically, it should be the "Dunn-Bonferroni correction" — but Bonferroni's name stuck because the underlying mathematical inequality is his.

The "Aha" Bridge: So... you have a total alpha budget of 0.05. Bonferroni says: divide it equally among all your tests. Like splitting a pizza equally among guests. 5 tests = each gets alpha/5 = 0.01. Simple. Fair. But if you have 20 guests, each gets a tiny slice. That's why Bonferroni is conservative — everyone gets an equal but tiny share of the alpha budget.

Naming Family:

• Dunn's test — Dunn's actual contribution (often confused with Bonferroni)
• Sidak correction — a slightly less conservative alternative using 1-(1-alpha)^(1/n) instead of alpha/n
• Bonferroni-Holm — the step-down improvement (see below)

The method: Divide alpha by the number of tests.

• 5 endpoints —> each tested at alpha = 0.05/5 = 0.01
• 20 endpoints —> each tested at alpha = 0.05/20 = 0.0025

Pros: Simple. Conservative. Universally accepted. Cons: Brutally conservative. With 20 endpoints, you need p < 0.0025 to claim significance. Real effects get missed (inflated Type II error).

When FDA likes it: Small number of co-primary endpoints (2-3).

2. Holm's Step-Down (Modified Bonferroni)

Word Surgery:

• Holm — Sture Holm, Swedish statistician who published this in 1979
• Step-down — you start from the most significant result and "step down" toward less significant ones
• Contrast with "step-up" where you start from the least significant and climb up

Why This Name? Holm noticed that Bonferroni was wasteful. If the most significant p-value passes alpha/n, then for the second test, you only have (n-1) remaining hypotheses — so test it at alpha/(n-1), not alpha/n. You're "stepping down" through the ordered p-values, and at each step, the threshold gets slightly more generous because there are fewer hypotheses left. Holm called it "a sequentially rejective procedure" — but everyone calls it "step-down" because the image of walking down stairs through p-values is intuitive.

The "Aha" Bridge: So... Bonferroni splits the pizza equally for ALL guests at the start. Holm says: "As each guest finishes and leaves, the remaining guests can have bigger slices." The first test uses alpha/n (same as Bonferroni). But once that hypothesis is rejected, the next test uses alpha/(n-1). Then alpha/(n-2). Each rejected hypothesis frees up alpha for the remaining tests. Holm is Bonferroni with a recycling policy.

Naming Family:

• Holm-Bonferroni — full name (acknowledges Bonferroni foundation)
• Hochberg's step-up — the reverse direction (below)
• Hommel's procedure — a more complex but more powerful closed testing variant

The method: Order all p-values from smallest to largest. Test the smallest against alpha/n, the next against alpha/(n-1), and so on. Stop at the first failure.

Slightly less conservative than Bonferroni, uniformly more powerful, no additional assumptions. This is the "why would you ever use plain Bonferroni" method.

3. Hochberg's Step-Up

Word Surgery:

• Hochberg — Yosef Hochberg, Israeli statistician, published 1988
• Step-up — you start from the LEAST significant result and "step up" toward the most significant

Why This Name? Hochberg reversed Holm's direction. Instead of starting at the most significant and walking down, you start at the least significant and walk up. If ANY p-value along the way passes its threshold, everything more significant also passes. The name "step-up" was chosen purely to contrast with Holm's "step-down."

The "Aha" Bridge: So... Holm walks downstairs: if you trip (fail), everything below you is blocked. Hochberg walks upstairs: if you clear any step, everything above you (more significant) is automatically cleared too. This is more generous — which is why Hochberg is more powerful than Holm. But it requires that the tests are independent or positively dependent (a stronger assumption).

Naming Family:

• Holm (step-down) — the contrast
• Benjamini-Hochberg — different Hochberg! Yoav Benjamini and Yosef Hochberg together developed the FDR-controlling procedure in 1995 — conceptually distinct from Hochberg's step-up
• Simes procedure — the mathematical foundation Hochberg's step-up builds on

Like Holm's but works in reverse — starts from the largest p-value. More powerful than Holm's but requires independence or positive dependence between endpoints.

4. Fixed-Sequence (Hierarchical) Testing — The FDA Favourite

Word Surgery:

• Fixed — Latin fixus = "fastened, immovable" — the order is set in stone BEFORE the data is seen
• Sequence — Latin sequentia = "that which follows" (sequi = to follow)
• Hierarchical — Greek hierarchia = "rule of a high priest" (hieros = sacred + archein = to rule) — originally described the ranked order of angels, then any ranked system
• Literal meaning: "Testing in an immovable order, ranked from most important to least"

Why This Name? The idea is simple: you declare the rank-order of your endpoints before looking at the data. The primary is "highest in the hierarchy." The first secondary is next. And so on. The "fixed" part is critical — if you chose the order AFTER seeing the data, you'd put the most significant first and cheat. The word "fixed" is a regulatory warning label: this order was locked before unblinding.

The "Aha" Bridge: So... think of it like a relay race. Runner 1 (primary endpoint) must finish before Runner 2 (first secondary) can start. If Runner 1 drops out (p > 0.05), the race is over — Runners 2, 3, 4 never even get to run, regardless of how fast they are. This is why the order of endpoints in the SAP is a strategic decision worth millions of dollars.

Naming Family:

• Gatekeeping — a generalisation where "gates" separate families of endpoints (below)
• Serial testing — synonym
• Fallback procedure — a variation where alpha can be redistributed when early tests fail
• Graphical approaches (Bretz et al.) — modern visual framework that generalises all fixed-sequence methods into directed alpha-flow graphs

The method: Pre-specify the order of endpoints before unblinding. Test the primary endpoint at full alpha=0.05. If (and ONLY if) it's significant, test the first secondary at alpha=0.05. If that's significant, test the next. Stop at the first failure.

Primary endpoint (HbA1c): p=0.02 --> Significant --> Proceed Secondary #1 (Weight): p=0.03 --> Significant --> Proceed Secondary #2 (Lipids): p=0.04 --> Significant --> Proceed Secondary #3 (QoL): p=0.06 --> NOT significant --> STOP Secondary #4 (BP): p=0.01 --> Cannot claim significance even though p=0.01!

The brutal rule: Once the chain breaks, everything after is exploratory — even if p=0.001. This is why the ORDER of endpoints in the SAP is a strategic decision worth millions of dollars.

When FDA loves it: Pivotal trials with clear primary and ranked secondaries. It's the most common approach in NDA submissions.

5. Gatekeeping Procedures

Word Surgery:

• Gate — Old English gaet = an opening in a wall or fence that can be opened or closed
• Keeping — maintaining, guarding
• Literal meaning: "Guarding the gate" — controlling who/what passes through

Why This Name? In multiplicity, a "gate" is a statistical barrier between families of endpoints. The primary family is behind Gate 1. Only if the primary family passes (all or some endpoints significant) does Gate 1 open, allowing the secondary family to be tested. Dmitrienko, Offen, and Westfall formalised this in the early 2000s. The metaphor is perfect: there is literally a gate between your primary and secondary analyses, and only statistical significance can open it.

The "Aha" Bridge: So... fixed-sequence testing is a single corridor with one door after another. Gatekeeping is a castle with multiple rooms behind multiple gates. Each room (family of endpoints) is locked behind a gate. You can only enter the next room if you've conquered the current one. Within each room, you might use Bonferroni or Holm to handle the endpoints inside.

Naming Family:

• Serial gatekeeping — gates in strict sequence
• Parallel gatekeeping — multiple gates that can open simultaneously
• Dmitrienko-Tamhane procedures — the mathematical formalisation
• Truncated Holm — a specific gatekeeping variant

Multiple families of endpoints with hierarchical gates. The secondary family can only be tested if the primary family passes. Within each family, Bonferroni or Holm applies.

Used in complex oncology trials with co-primary endpoints (OS and PFS) and multiple secondary endpoints.

6. Alpha-Spending Functions (For Interim Analyses)

Word Surgery:

• Alpha (alpha) — the Greek letter used for the Type I error probability (by convention since Neyman-Pearson, 1933)
• Spending — from Old English spendan = "to use up, consume"
• Literal meaning: "Using up your alpha budget over time"

Why This Name? Peter Lan and David DeMets coined the term "alpha-spending function" in 1983. They imagined alpha = 0.05 as a budget that you "spend" across multiple looks at the data. Each interim analysis "spends" some alpha. The metaphor is financial: you have 5 paise of alpha, and each interim analysis costs some paise. Spend too much early, and you have nothing left for the final analysis. The brilliance of the term is that it reframes a mathematical constraint as household budgeting — something everyone understands.

The "Aha" Bridge: So... you're a DSMB. You want to peek at the data 3 times during the trial. But each peek costs you alpha. Alpha-spending is like having Rs 5 to spend across 3 shopping trips. If you blow Rs 4 on the first trip (very loose early stopping boundary), you only have Re 1 left for the final analysis. O'Brien-Fleming spending is like being extremely frugal on early trips (spending only 5 paise) so you have almost the full Rs 5 for the final analysis.

Naming Family:

• Pocock spending — spends alpha equally at each look (aggressive early spending)
• O'Brien-Fleming spending — barely spends anything early, saves for the end (conservative early spending)
• Lan-DeMets — the flexible framework that encompasses both
• Group sequential methods — the broader class of methods for interim analyses

When a DSMB looks at data 3 times during a trial, each look consumes some of the alpha budget:

Word Surgery:

• O'Brien — Peter C. O'Brien, biostatistician at Mayo Clinic
• Fleming — Thomas R. Fleming, biostatistician at University of Washington (later famous for the Prentice criterion for surrogate endpoints and the FDA advisory committee)
• Published 1979

Why This Name? O'Brien and Fleming designed stopping boundaries that are extremely strict early in the trial and become progressively more lenient. Their key insight: if the drug is truly effective, the effect will still be there at the final analysis — so don't waste alpha on early looks. But if the effect is so overwhelming that it clears an impossibly high bar early, then it would be unethical NOT to stop. The boundaries reflect this philosophy.

The "Aha" Bridge: So... O'Brien-Fleming boundaries are like a cricket selector who won't pick a player after one good innings (too much chance involved). But if someone scores 300 in a single innings, you'd be foolish not to select them immediately. The early bar is set so high that only a truly extraordinary effect can clear it.

Naming Family:

• Pocock boundaries — equal spending at each look (less conservative early)
• Haybittle-Peto — fixed boundary of p<0.001 at all interim looks
• Wang-Tsiatis — a family that includes both O'Brien-Fleming and Pocock as special cases

O'Brien-Fleming spending:

Total alpha = 0.05. The early looks are nearly impossible to pass (preserving alpha for the final), but they allow stopping for overwhelming efficacy or futility.

Word Surgery:

• Lan — K.K. Gordon Lan, biostatistician
• DeMets — David L. DeMets, biostatistician at University of Wisconsin-Madison
• Published 1983

Why This Name? O'Brien-Fleming required you to pre-specify exactly HOW MANY interim looks you'd take and exactly WHEN. Lan and DeMets solved this by creating a continuous "spending function" — a mathematical curve that tells you how much alpha you've spent at any point during the trial, regardless of how many looks you take or when you take them. It's the flexible version of alpha-spending. They called it an "alpha-spending function" because you specify a function (a curve) rather than fixed boundaries.

The "Aha" Bridge: So... O'Brien-Fleming is like a pre-paid phone plan: you decide in advance exactly how many calls you'll make and when. Lan-DeMets is like a pay-as-you-go plan: you can make calls whenever you want, and the system tracks how much of your credit (alpha) you've used. Much more practical for real trials where unplanned interim analyses sometimes become necessary.

Naming Family:

• Alpha-spending function — the general concept they formalised
• Information fraction — the proportion of total data collected at each look (the "x-axis" of the spending function)
• Error-spending approach — generic term

Lan-DeMets is a flexible version that allows unplanned interim analyses.

Real example: The RECOVERY trial for dexamethasone in COVID-19 used an O'Brien-Fleming spending function. The interim analysis was so overwhelmingly positive (p < 0.001) that the trial stopped early. Thousands of lives were saved because the spending function allowed early stopping while protecting Type I error.

What Happens When You DON'T Control Multiplicity

Case: Subgroup Fishing in PLATO

The PLATO trial (ticagrelor vs clopidogrel in ACS) was positive overall. Then a subgroup analysis found that the benefit disappeared in North American patients. Was this real?

The trial tested dozens of subgroups. By chance alone, some would show no benefit. FDA's statistical reviewer called it a likely multiplicity artefact. But the finding triggered years of debate, additional studies, and clinical confusion.

One unadjusted subgroup analysis —> years of wasted resources and clinical uncertainty.

Case: The Duke Clinical Research Institute Lesson

A cardiovascular outcomes trial tested a new anticoagulant. Primary endpoint: negative. But among 18 pre-specified subgroups, 2 showed "significant" benefit. The sponsor highlighted these in the submission.

FDA reviewer's response (paraphrased): "With 18 subgroups at alpha=0.05, we expect 0.9 false positives. You found 2. This is not a discovery. This is arithmetic."

Drug not approved for those subgroups. The subgroup "findings" were exactly what multiplicity predicts from noise.

Key Terms You'll See in Papers — Deconstructed

Before we go branch-by-branch, let's arm you with the vocabulary.

Word Surgery:

• Pre- — Latin prae = "before"
• Specification — Latin specificare = "to mention particularly" (species = kind/type + facere = to make)
• Literal meaning: "Mentioning it particularly, before"

Why This Name? In clinical trials, "pre-specification" means writing down your analysis plan (which endpoints, which tests, which order, which corrections) BEFORE you look at the data. The word exists because its opposite — post-hoc specification — is the root of all multiplicity evil. If you decide which endpoints to report AFTER seeing which ones are significant, you're not doing science; you're doing storytelling.

The "Aha" Bridge: So... pre-specification is like placing your bet BEFORE the cricket match starts. Post-hoc analysis is like "betting" AFTER the match and claiming you "predicted" the result. Pre-specification is what makes a clinical trial different from a fishing expedition.

Naming Family:

• Pre-registered — similar concept applied to the entire trial (ClinicalTrials.gov)
• SAP (Statistical Analysis Plan) — the document where pre-specification lives
• Post-hoc — the opposite (Latin: "after this")
• A priori vs. a posteriori — the philosophical version (before vs. after experience)

Word Surgery:

• Exploratory — Latin explorare = "to search out, investigate" (originally military: ex = out + plorare = to cry out, referring to scouts who "cry out" when they find something)
• Confirmatory — Latin confirmare = "to make firm, strengthen" (con- = together + firmare = to make firm)
• Literal meaning: Exploratory = "scouting ahead, searching." Confirmatory = "making it firm, cementing it."

Why This Name? ICH E9 formally distinguishes exploratory from confirmatory analyses. An exploratory analysis generates hypotheses ("Hey, this subgroup looks interesting!"). A confirmatory analysis tests a pre-specified hypothesis with controlled error rates ("We confirm that the drug works for this endpoint at this alpha level"). The distinction exists because only confirmatory analyses can support regulatory claims. Exploratory findings are interesting; confirmatory findings are actionable.

The "Aha" Bridge: So... exploration is sending scouts into unknown territory. Confirmation is sending the army to take and hold the territory the scouts found. You don't build a fort based on a scout's first report. You send more scouts, make a plan, then march in with full force. In clinical trials: exploratory analysis = the scout. Confirmatory trial = the army.

Naming Family:

• Hypothesis-generating — synonym of exploratory
• Hypothesis-testing — synonym of confirmatory
• Pivotal trial — a confirmatory trial intended for regulatory submission
• Phase 2 (exploratory) vs. Phase 3 (confirmatory) — the trial phase mapping

Word Surgery:

• Sub- — Latin = "under, below"
• Group — from Italian/German gruppo = "a cluster, knot"
• Literal meaning: "A cluster within/below the main group"

Why This Name? A subgroup is a subset of the trial population defined by some baseline characteristic (age, sex, race, disease severity). Subgroup analysis asks: "Does the treatment effect differ in this subset?" The term is straightforward. What's NOT straightforward is the multiplicity disaster it creates.

The "Aha" Bridge: So... if you slice a pizza 18 ways (18 subgroups), some slices will, by pure chance, have more toppings than others. That doesn't mean the chef put more toppings on those slices. It means you sliced it more times. Every subgroup is another slice, and every slice is another chance for noise to masquerade as signal.

Naming Family:

• Stratified analysis — similar but involves pre-planned stratification in the randomisation
• Interaction test — the correct statistical test for subgroup effects (tests whether the treatment effect truly differs across subgroups, rather than testing within each subgroup separately)
• Forest plot — the standard visual display of subgroup results

Word Surgery:

• Interim — Latin interim = "meanwhile, in the meantime" (inter = between + im from eum = "it")
• Literal meaning: "An analysis done in the meantime" — between the start and end of the trial

Why This Name? A clinical trial collects data over months or years. An interim analysis looks at the data BEFORE the trial is finished — "in the meantime." The term distinguishes it from the "final analysis" (when all data is in). Interim analyses create multiplicity because each look at accumulating data is another test.

The "Aha" Bridge: So... an interim analysis is like checking the score at halftime. The problem? Every time you check the score, you're tempted to make decisions based on incomplete information. A team leading 2-0 at halftime might still lose 2-3. Similarly, a drug looking effective at 50% enrollment might fail at 100%. Each "check" increases the chance you'll make a premature call.

Naming Family:

• DSMB/DMC (Data Safety Monitoring Board / Data Monitoring Committee) — the independent committee that conducts interim analyses
• Futility analysis — an interim look to check if the trial should stop for futility (the drug is clearly not working)
• Efficacy stopping — stopping early because the drug clearly works
• Information fraction — what proportion of total planned data has been collected at the time of the interim look

Branch-by-Branch — Where Multiplicity Bites You

General Medicine

The trap: A meta-analysis of Vitamin D supplementation tests its effect on 12 different outcomes: mortality, cancer, CVD, diabetes, fractures, falls, depression, respiratory infections, autoimmune disease, cognition, muscle strength, quality of life.

Two come out "significant." The paper concludes Vitamin D prevents respiratory infections and falls.

The reality: 12 tests. Expected false positives by chance alone: 0.6. Finding 2 is not impressive — it's predictable. Every Vitamin D meta-analysis with dozens of outcomes will find "something." That's not Vitamin D working. That's multiplicity working.

Your patient: You start giving Vitamin D to every patient "for immunity" based on a multiplicity artefact.

Surgery

The trap: A trial comparing robotic vs laparoscopic cholecystectomy reports outcomes at 1 week, 1 month, 3 months, 6 months, and 1 year. At each timepoint they measure: pain score, cosmetic score, return to work, complication rate, operative time, hospital stay, and cost.

That's 5 x 7 = 35 comparisons. Expected false positives: 1.75.

The paper reports: "Robotic surgery showed significantly better cosmetic scores at 3 months (p=0.04) and faster return to work at 1 month (p=0.03)."

Two "significant" findings out of 35 tests is EXACTLY what you'd expect from random noise. The robot does nothing, but the paper looks like it does.

Your hospital buys a Rs 3 crore robot.

Paediatrics

The trap: Paediatric drug trials are small. Small trials have noisy estimates. Noisy estimates + multiple endpoints = high multiplicity risk.

A trial of 40 children tests a new asthma controller on: FEV1, PEF, symptom days, rescue inhaler use, exacerbation rate, and quality of life. Six endpoints at alpha=0.05 —> 26% chance of a false positive.

The paper reports "significant improvement in rescue inhaler use (p=0.04)." Everything else negative.

One out of six is not a discovery in a 40-child trial. It's what multiplicity predicts. But the drug gets added to guidelines for paediatric asthma based on this single unadjusted endpoint.

Obstetrics

The trap: Subgroup analysis. A trial of antenatal corticosteroids in late preterm birth finds no overall benefit. But in the subgroup of "women delivering within 7 days of steroid administration," the benefit is "significant."

This subgroup was one of 12 tested. No multiplicity correction. The result is almost certainly a false positive. But now every obstetrician feels compelled to time steroids to delivery — an impossible task in practice — based on a subgroup that was likely noise.

Psychiatry

The trap: The PANSS scale (Positive and Negative Syndrome Scale) for schizophrenia has 30 individual items grouped into 3 subscales, plus a total score. Many trials report the total AND each subscale AND selected individual items.

A drug fails the total PANSS. Fails the positive subscale. Fails the negative subscale. But shows "significant improvement" on item P3 (hallucinatory behaviour) and item G12 (lack of judgment).

2 out of 34 tests (total + 3 subscales + 30 items). Expected false positives: 1.7. The paper writes: "The drug showed targeted improvement in hallucinatory behaviour and judgment."

That's not a targeted effect. That's multiplicity with a marketing spin.

Community Medicine / PSM

The trap: A massive NFHS-style survey collects data on 200+ variables across 500,000 households. Researchers run bivariate associations between everything and everything.

"Significant association between type of cooking fuel and childhood stunting (p=0.03)." "Significant association between distance to health centre and contraceptive use (p=0.02)." "Significant association between mobile phone ownership and institutional delivery (p=0.04)."

With 200 variables and potentially thousands of pairwise comparisons, hundreds of these associations are expected by chance alone. But each one becomes a separate publication, a separate policy recommendation, a separate PhD thesis.

The entire epidemiological literature is riddled with unreproducible associations that are artefacts of unadjusted multiplicity in large surveys.

Orthopaedics

The trap: Implant comparison studies love to report every functional score that exists.

"We compared Implant A vs Implant B using: Harris Hip Score, WOMAC, SF-36 (8 subscales), Oxford Hip Score, EQ-5D, radiographic loosening, osteolysis, revision rate."

That's 14 outcomes. Expected false positives: 0.7.

"Implant A showed significantly better SF-36 Physical Functioning subscale (p=0.03)."

One out of 14. This is noise. But the implant company uses this in their marketing material: "Proven superior physical functioning."

The 5 Ways Not Knowing Multiplicity Destroys You

1. You believe "significant" secondary endpoints when the primary failed

Iron rule: If the primary endpoint fails and there's no pre-specified multiplicity correction, ALL secondary "significant" results are exploratory. They are hypothesis-generating, not hypothesis-confirming. They belong in the Discussion, not the Conclusion.

If you see a paper with a failed primary and excited conclusions about secondaries — the authors are either ignorant of multiplicity or deliberately misleading you.

2. You can't spot subgroup fishing

The classic move: overall result is boring or negative. But wait — in women over 65 with baseline LDL > 160 who were enrolled in European sites... p=0.02!

The more specific the subgroup, the more comparisons were implicitly made to find it, and the more likely it's a false positive.

If the subgroup wasn't pre-specified in the SAP, it's fishing. If it was pre-specified but there are 20 pre-specified subgroups with no multiplicity correction, it's fishing with a licence.

3. You don't understand why your thesis needs a single primary endpoint

Your thesis committee insists on ONE primary endpoint. You want five because "what if the primary doesn't work?"

Now you know why: with five primary endpoints and no correction, your thesis has a 23% false positive rate. Your examiner will ask: "What was your primary endpoint?" If you say "we had five," you've admitted your study has no multiplicity control and your "significant" result may be noise.

4. You misread oncology trials

Modern oncology trials are the most complex multiplicity puzzles in medicine. Co-primary endpoints (OS + PFS), multiple dose groups, multiple tumour types, multiple lines of therapy, interim analyses — all requiring elaborate gatekeeping and alpha-splitting strategies.

When Keytruda reports "significant improvement in PFS in PD-L1 >= 50% patients," that significance was achieved within a pre-specified hierarchical testing framework that preserved overall alpha=0.05 across dozens of comparisons. The SAP for these trials is often 100+ pages of multiplicity strategy.

If you don't understand this, you can't evaluate whether the significance claim is legitimate or whether it's a cherrypicked finding from an unadjusted subgroup.

5. You fall for the "look at all these significant p-values" trick

A paper reports 8 out of 12 outcomes as significant. Impressive? Not necessarily.

If the 12 outcomes are correlated (e.g., multiple measures of the same underlying construct — weight, BMI, waist circumference, body fat percentage), then one real effect can produce multiple "significant" results that are all measuring the same thing. It looks like 8 independent confirmations but it's really 1 finding reflected 8 times in a hall of mirrors.

Conversely, if 12 truly independent outcomes are tested, Bonferroni applies and you need p < 0.004 for each. Most of those 8 "significant" results would vanish.

The One Thing to Remember

Every time a paper reports a significant result, ask:

If the answer is "one pre-specified primary endpoint tested at alpha=0.05 with a pre-registered SAP" — you can trust the p-value.

If the answer is "they tested 15 things and are reporting the 2 that worked" — you're looking at statistical noise with a narrative wrapped around it.

Multiplicity is the reason good journals require pre-registration. It's the reason FDA demands fixed-sequence testing. It's the reason your thesis needs one primary endpoint. And it's the reason most "statistically significant" findings in exploratory analyses will never replicate.

The resident who understands multiplicity doesn't just read the Results section. They count the tests. They check the SAP. They ask: "Was this finding the one they were looking for, or was it the one they stumbled upon after everything else failed?"

That question — more than any formula — is what separates someone who reads papers from someone who understands them.