How do you show that two treatments are about the same?

Start here

Why does p > 0.05 not show that two treatments are the same?

A test of no difference asks whether the data are surprising if the treatments are identical. A large p-value means the data are not surprising. It does not mean the treatments are identical, because a small trial produces large p-values for large differences too.

To show that two treatments are about the same, you first decide how different they may be and still count as the same. That amount is the margin, written Δ (delta). ICH E9 defines it as “the largest difference that can be judged as being clinically acceptable”. Then you must show that the true difference is unlikely to be as large as Δ. The clinical trials chapter shows where the margin sits for superiority, non-inferiority and equivalence. This chapter shows how the test works, how the margin is chosen, and how many patients it needs.

The test

How do two one-sided tests show that two treatments are equivalent?

Turn the question around. The null hypothesis becomes 'the treatments differ by Δ or more', and the trial must reject it. Because the difference can be too large in either direction, there are two null hypotheses, and each gets its own one-sided test.

t₁ = (d + Δ) ÷ SE   ·   t₂ = (d − Δ) ÷ SE   ·   equivalence if p₁ < α and p₂ < α

d

observed difference, new minus reference

Δ

equivalence margin, fixed in the protocol

SE

standard error of the difference, the same as in an ordinary t-test

p₁

upper-tail p-value of t₁ (null hypothesis: difference ≤ −Δ)

p₂

lower-tail p-value of t₂ (null hypothesis: difference ≥ +Δ)

TOST p

the larger of p₁ and p₂

ICH E9 describes this exactly: equivalence is inferred when the whole confidence interval falls within the margins, which is equivalent to “using two simultaneous one-sided tests”, and because the two null hypotheses do not overlap, the type I error is controlled without any correction. Schuirmann compared this procedure with the older “power approach” in 1987.

Interactive

Change the difference, the spread, the sample size and the margin, and watch both one-sided tests

The ordinary t-test and TOST use the same difference and the same standard error. They ask opposite questions, so a small trial can give a large ordinary p-value and a large TOST p-value at the same time.

Confidence intervals

Why does equivalence use a 90% confidence interval and non-inferiority a 95% one?

The confidence level follows from α. A one-sided test at level α rejects exactly when one limit of the 100(1 − 2α)% two-sided interval lies beyond the null value.

In the teaching example the 90% CI is −0.14 to 0.24 and the 95% CI is −0.18 to 0.28. The 95% interval is also inside ±0.4, so the biosimilar is non-inferior as well as equivalent. A 95% interval inside the margins would satisfy an equivalence test at α 2.5% for each side, which is stricter than TOST at 5%. Whichever level you use, state it in the protocol.

Decide

Which design answers your question?

Answer the questions in order.

Comparison

Which of these ideas do residents mix up?

Non-inferiority or equivalence?

A test of no difference or two one-sided tests?

The two tests can disagree in useful ways. A very large trial can find a statistically significant difference that lies well inside the margin: significant and equivalent at the same time.

The margin

How is the non-inferiority margin chosen?

The margin has two jobs. It must be small enough that a new drug meeting it still works better than placebo, and small enough that the loss of benefit is clinically acceptable. The FDA guidance calls these two margins M1 and M2.

M1

the entire effect of the active control assumed to be present in the non-inferiority trial (FDA 2016). Ruling out a loss of M1 shows the new drug has some effect

M2

the largest clinically acceptable loss of that effect, a clinical judgement, usually smaller than M1

Fraction preserved

if M2 keeps 50% of the effect, M2 = 0.5 × M1 on the chosen scale

What the guidance documents require

• Evidence that the comparator works. ICH E10 says the margin cannot be greater than the smallest effect the active drug would reliably be expected to have compared with placebo in the setting of the planned trial. That requires historical evidence of sensitivity to drug effects: earlier trials of similar design that regularly separated the drug from placebo.
• The constancy assumption. The FDA guidance asks whether the comparator's effect in the new trial is likely to be the same as in the earlier trials. Different patients, doses, outcomes or background treatment can make the old effect estimate wrong for the new trial.
• Keeping a fraction of the effect. FDA: choosing M2 as 50% of M1 “has become usual practice for cardiovascular (CV) outcome studies”. For antibiotic trials, where effects over placebo are large, a 10–15% margin on the risk difference scale is commonly chosen.
• A worked regulatory example. In the FDA guidance, six placebo-controlled warfarin trials gave a pooled relative risk of 0.361 (95% CI 0.248 to 0.527). The upper limit 0.527 gives M1 = 1/0.527 = 1.898. M2 was set at 50% of M1 on the log hazard scale, 1.378. In SPORTIF V the 95% CI for ximelagatran was 0.91 to 2.12. The upper limit 2.12 exceeds M2, so non-inferiority to warfarin was not shown.
• Prespecified and justified. ICH E9: the margin is specified in the protocol and its choice “should be justified clinically”. It should also be smaller than the differences seen in superiority trials of the active comparator.

Analysis sets

Why do intention to treat and per protocol both matter in a non-inferiority trial?

In a superiority trial, patients who stop treatment or cross over make the groups look more alike, which works against the claim of a difference. In a non-inferiority trial the claim is that the groups are alike, so the same dilution now works in favour of the claim.

ICH E9 states that in an equivalence or non-inferiority trial the full analysis set (intention to treat) “is generally not conservative”. The per-protocol set has its own risk: its bias “may be severe” because adherence may be related to treatment and outcome. Neither analysis alone is safe, so both are reported, and ICH E9 notes that when they “lead to essentially the same conclusions, confidence in the trial results is increased”. The clinical trials chapter explains the two analysis sets.

Interactive

Move the share of patients who switch drugs and watch the intention-to-treat result move towards zero

Patients who switch to the standard drug receive its benefit but are counted in the new-drug group. The more who switch, the more alike the two groups look.

Pitfalls

What goes wrong when margins are loose or trials are chained?

A non-inferior drug may be slightly worse than its comparator. If the next drug is then compared only with that one, and the one after with the next, small accepted losses can add up.

How to reduce the risk

• Use the best available treatment as the comparator, not the most recently approved one (Odem-Davis and Fleming, 2015).
• Base the margin on a well-estimated effect of that comparator over placebo, allowing for its uncertainty and possible bias, and keep a stated fraction of it (M2).
• Distrust a loose margin. A margin close to M1, or wider than differences that clinicians would notice, can label a less effective treatment as non-inferior. Aberegg and colleagues (2018) studied 163 non-inferiority trials from the five highest impact general medical journals. Comparisons of a therapy at reduced intensity against the same therapy at full intensity reached a favourable result less often than the other comparisons (58.3% against 82.2%).
• Keep trial conduct tight. Poor adherence, crossovers and loss to follow-up all move a non-inferiority result towards the claim.

Bioequivalence

How do regulators decide that a generic tablet is bioequivalent?

A generic contains the same drug as the reference product. If both put the same amount of drug into the blood at the same rate, the clinical effect is expected to be the same, so the comparison is made on blood levels instead of on a clinical outcome.

The EMA guideline (2010) states that two products containing the same active substance are bioequivalent if they are pharmaceutically equivalent or pharmaceutical alternatives and their bioavailabilities (rate and extent) after the same molar dose lie within acceptable predefined limits. AUC, the area under the concentration-time curve, reflects the extent of exposure. Cmax and tmax are influenced by the rate of absorption.

The standard requirements (EMA 2010, with FDA 2001)

1. Design: a randomised, two-period, two-sequence, single-dose crossover, usually in healthy volunteers. The washout must bring drug concentrations below the lower limit of quantification, normally at least 5 elimination half-lives.
2. Parameters: after a single dose, AUC(0–t) (or AUC(0–72h) when relevant) and Cmax.
3. Analysis: ANOVA on log-transformed values. The confidence interval for the difference on the log scale is back-transformed to a confidence interval for the ratio. A non-parametric analysis is not acceptable (EMA).
4. Acceptance: the 90% confidence interval for the ratio must lie within 80.00–125.00%. The FDA guidance states that this 90% interval approach is equivalent to two one-sided tests at the 5% level.
5. Narrow therapeutic index drugs: the AUC range is tightened to 90.00–111.11%, and also for Cmax where Cmax matters for safety, efficacy or drug monitoring (EMA).
6. Highly variable drugs: the Cmax range may be widened, to a maximum of 69.84–143.19%, only in a replicate design where the within-subject variability of the reference Cmax is above 30%, with clinical justification. AUC stays at 80.00–125.00% (EMA).

Interactive

Change the ratio, the variability and the number of subjects, and watch the 90% confidence interval against 80–125%

The point estimate alone never decides bioequivalence. The whole 90% interval must lie inside the acceptance range, and its width depends on the within-subject variability and the number of subjects.

The formula

Worked example: equivalence, non-inferiority and bioequivalence from the same steps

Made-up teaching data. 150 patients per group. HbA1c fell by 1.10 percentage points with the biosimilar and 1.15 with the reference insulin. SD of the change 1.0 in both groups. Equivalence margin ±0.4.

Step by step: two one-sided tests

1. Difference d = −1.10 − (−1.15) = 0.05.
2. SE = SD × √(1/n₁ + 1/n₂) = 1.0 × √(2 ÷ 150) = 0.1155. Degrees of freedom = 150 + 150 − 2 = 298.
3. Test 1: t₁ = (0.05 + 0.4) ÷ 0.1155 = 3.90. Upper-tail p₁ = 0.00006.
4. Test 2: t₂ = (0.05 − 0.4) ÷ 0.1155 = −3.03. Lower-tail p₂ = 0.0013.
5. TOST p = the larger = 0.0013, below 0.05. Equivalence is shown.
6. Check with the interval: t for 298 df at 0.95 = 1.650. 90% CI = 0.05 ± 1.650 × 0.1155 = −0.14 to 0.24, inside ±0.4.
7. The ordinary t-test on the same data: t = 0.05 ÷ 0.1155 = 0.43, p = 0.67.

TOST p = 0.0013 → 90% CI −0.14 to 0.24: equivalent

The same data as a non-inferiority trial

With a margin of −0.4 and a one-sided α of 2.5%: t for 298 df at 0.975 = 1.968, 95% CI = 0.05 ± 1.968 × 0.1155 = −0.18 to 0.28. The lower limit −0.18 lies above −0.4, so the biosimilar is non-inferior. With your thesis sample of 20 per group, the 95% CI would be −0.59 to 0.69, and neither conclusion is possible.

Bioequivalence: a ratio on the log scale

Made-up teaching data: 24 subjects in a 2 × 2 crossover, ratio of geometric means for AUC 94%, within-subject CV 25%.

1. Within-subject variance on the log scale: s² = ln(1 + CV²) = ln(1.0625) = 0.0606. (In a real study s² is the residual mean square of the ANOVA. The EMA guideline gives CV = 100 × √(e^s² − 1).)
2. SE of the log difference = √(2 × 0.0606 ÷ 24) = 0.0711. Degrees of freedom = 24 − 2 = 22, t at 0.95 = 1.717.
3. ln 0.94 = −0.0619. Limits on the log scale: −0.0619 ± 1.717 × 0.0711 = −0.1839 and 0.0602.
4. Back-transform: e^−0.1839 = 0.8320 and e^0.0602 = 1.0620.

90% CI = 83.20% to 106.20%: bioequivalent

All numbers computed in Python (SciPy); the TOST p-values were checked with statsmodels (ttost_ind).

Sample size

How many patients does an equivalence or non-inferiority trial need?

The same four ingredients as any sample size: α, power, SD and a distance. Here the distance is between the margin and the difference you expect, and that distance is often small.

Equivalence, true difference 0: n per group = 2 × SD² × (z_1−α + z_1−β/2)² ÷ Δ²

Equivalence or non-inferiority, true difference δ: n per group ≈ 2 × SD² × (z_1−α + z_1−β)² ÷ (Δ − |δ|)²

z 1−α

1.645 for α 5% per side (equivalence); 1.960 for one-sided α 2.5% (non-inferiority)

z 1−β/2

1.645 for 90% power in the equivalence formula with δ = 0, because power is shared between the two tests

z 1−β

1.282 for 90% power

δ

the true difference you expect. For non-inferiority, a new treatment expected to be slightly worse reduces Δ − |δ|

Teaching example, SD 1.0, Δ 0.4, 90% power. Equivalence with δ = 0: 2 × (1.645 + 1.645)² ÷ 0.4² = 135.3 → 136 per group. With δ = 0.1: 2 × (1.645 + 1.282)² ÷ 0.3² = 190.3, and the exact TOST calculation gives 192 per group. ICH E9 warns that when power is assessed at a true difference of zero, the sample size is underestimated if the true difference is not zero.

Bioequivalence studies are sized from the within-subject CV and the expected ratio. For a 2 × 2 crossover with α 5% per side, an expected ratio of 95% and 80% power, the exact TOST calculation gives 20 subjects at CV 20%, 28 at CV 25%, 40 at CV 30% and 66 at CV 40%. Add subjects for expected dropouts.

In a real paper

How is non-inferiority, and bioequivalence, reported in real studies?

How to run it

How do you run an equivalence or non-inferiority analysis?

Most software does not label a test as TOST. You can always get the answer from a confidence interval at the right level. Dedicated functions give both one-sided p-values.

SPSS

1. Analyze > Compare Means > Independent-Samples T Test. Select the outcome as the test variable, select the group as the grouping variable, and click Define Groups to enter the two group codes.
2. Click Options. The Confidence Interval box is 95% by default and accepts values between 1 and 99. Enter 90 for an equivalence test at α 5% per side. Keep 95 for non-inferiority at one-sided α 2.5%.
3. Compare the interval for the mean difference with your margins. SPSS gives the ordinary test of no difference; its p-value is not the TOST p-value.

R

# TOSTER package: two one-sided tests
library(TOSTER)
t_TOST(x = hba1c_bio, y = hba1c_ref, eqb = 0.4, alpha = 0.05, var.equal = TRUE)

# From summary statistics (means, SDs, n)
tsum_TOST(m1 = -1.10, sd1 = 1.0, n1 = 150,
          m2 = -1.15, sd2 = 1.0, n2 = 150,
          eqb = 0.4, alpha = 0.05, var.equal = TRUE)

# Bioequivalence sample size, 2 x 2 crossover (PowerTOST package)
library(PowerTOST)
sampleN.TOST(CV = 0.25, theta0 = 0.95, targetpower = 0.8, design = "2x2")

• t_TOST uses the Welch test unless var.equal = TRUE. eqb takes one value for symmetric bounds or two values for lower and upper bounds. The TOSTER plot draws a 1 − 2α interval by default (90% at α 0.05).
• sampleN.TOST estimates the number of subjects for a target power. By default it works on the log scale with a lower limit of 0.80, an upper limit of 1/0.80 = 1.25, a one-sided α of 0.05 and an assumed ratio of 0.95.
• The equivalence package offers tost(x, y, epsilon = 0.4) for the same test on raw data.

jamovi

Install the TOSTER module from the jamovi library. It adds a TOSTER menu with TOST versions of the one-sample, paired-samples and independent-samples t-tests, and tests for correlations and two proportions.

How to report it

What should you write in the thesis or paper?

State the framework, the margin with its reason, the interval level that matches your α, the result against the margin, and both analysis sets.

A worked sentence with the teaching numbers

“Equivalence was prespecified with a margin of ±0.4 percentage points in HbA1c change, the largest difference judged clinically unimportant, and tested with two one-sided t-tests at α = 0.05 each. Mean HbA1c fell by 1.10 points with the biosimilar and 1.15 with the reference insulin (difference 0.05, 90% CI −0.14 to 0.24; TOST p = 0.001). The whole interval lay within ±0.4, so the two insulins were equivalent in the full analysis set.”

Then give the per-protocol result in the same form. For a bioequivalence study, report the ratio of geometric means with its 90% CI for each of AUC and Cmax, and the acceptance range.

By specialty

Equivalence and non-inferiority questions by specialty

Common myths

Common mistaken beliefs

• ✗Myth: p > 0.05 shows that the two treatments are equally effective.

  ✓It shows that no difference was detected. ICH E9 calls it inappropriate to conclude equivalence or non-inferiority from a non-significant test of no difference.

• ✗Myth: "Non-inferior" means "not worse".

  ✓It means not worse by the margin or more. A non-inferior treatment can be slightly worse than the comparator.

• ✗Myth: Intention to treat is always the conservative analysis.

  ✓Only in superiority trials. ICH E9: in equivalence and non-inferiority trials the full analysis set is generally not conservative. Report per protocol too.

• ✗Myth: Equivalence is always read from the 95% confidence interval.

  ✓The level follows from α. Two one-sided tests at 5% each correspond to the 90% interval. Non-inferiority at one-sided 2.5% uses the lower limit of the 95% interval.

• ✗Myth: The margin can be settled after looking at the results.

  ✓It must be specified in the protocol and justified clinically (ICH E9), and based on the comparator's effect over placebo (ICH E10, FDA 2016).

• ✗Myth: Bioequivalence means the point estimate lies between 80% and 125%.

  ✓The whole 90% confidence interval for the ratio of geometric means must lie within 80.00–125.00% (EMA 2010, FDA 2001).

Why this matters to you

Five consequences of getting equivalence and non-inferiority wrong

1. Protocol sent back

   A protocol that aims to show a cheaper or safer drug is "as good" but has no margin, no rationale and no matching sample size will not pass a research committee.

2. A thesis conclusion you cannot defend

   "No significant difference, therefore equivalent" is the first sentence an examiner will question.

3. A sample size that is far too small

   Planning as a superiority trial and concluding equivalence leaves you with a confidence interval that crosses both margins.

4. Misreading published trials

   A loose margin, a missing per-protocol analysis or a comparator never proven against placebo can make a less effective treatment look acceptable.

5. Harm to patients

   Accepting small losses of effect trial after trial, or switching to a generic without understanding what bioequivalence covers, affects treatment choices for many patients.

History

History of equivalence and non-inferiority testing

1. 1972

   W. J. Westlake

   Published "Use of confidence intervals in analysis of comparative bioavailability trials" in the Journal of Pharmaceutical Sciences.

2. 1976

   W. J. Westlake

   Proposed confidence intervals symmetrical about zero for bioequivalence trials (Biometrics).

3. 1982

   William C. Blackwelder

   In "Proving the null hypothesis" in clinical trials, argued that when a new therapy is to be shown as effective as a standard, the null hypothesis should be that the standard is more effective by at least a specified amount.

4. 1987

   Donald J. Schuirmann

   Compared the two one-sided tests procedure with the "power approach" and showed that, at α 0.05, TOST had uniformly superior properties in most cases. TOST gives the same conclusion as Westlake's 1 − 2α confidence interval.

5. 1992

   US FDA

   Bioequivalence guidance recommended that AUC and Cmax be analysed with the two one-sided tests procedure (as described in the FDA 2001 guidance).

6. 1998

   ICH E9

   Statistical Principles for Clinical Trials set out equivalence and non-inferiority margins, confidence intervals and the role of analysis sets.

7. 2000

   ICH E10

   Choice of Control Group linked the non-inferiority margin to historical evidence that the active control beats placebo (assay sensitivity).

8. 2006

   Piaggio and colleagues

   Published the CONSORT extension for reporting non-inferiority and equivalence trials in JAMA. It was updated to match CONSORT 2010 in 2012.

9. 2010

   EMA; Everson-Stewart and Emerson

   The EMA Guideline on the Investigation of Bioequivalence (Rev. 1) was adopted on 20 January 2010. The same year, a simulation study examined the conditions under which bio-creep occurs.

10. 2016

    US FDA

    Issued the final guidance Non-Inferiority Clinical Trials to Establish Effectiveness (November 2016), defining M1 and M2.

Guidelines

Equivalence, non-inferiority and bioequivalence in guidelines

Reading a paper

Five checks on any equivalence or non-inferiority trial

• Was the margin set in the protocol, stated in the units of the outcome, and justified from the comparator's effect over placebo and from clinical judgement?
• Does the confidence interval level match the α of the test (90% for TOST at 5% per side, 95% for one-sided 2.5%), and is the whole relevant limit compared with the margin?
• Are both intention-to-treat and per-protocol results reported, and do they agree?
• Has the comparator reliably beaten placebo in earlier trials with similar patients, dose and outcome?
• For bioequivalence: log-transformed AUC and Cmax, a 90% CI for the ratio of geometric means, and the whole interval within 80.00–125.00% (or the tighter range for narrow therapeutic index drugs)?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

TNMGRMU questions on equivalence and non-inferiority tests

Questions from MD, MS, DM, MCh and Ph.D. papers that this chapter helps you answer. Tap a sitting to open the university's own question paper.

1. ×1Non-inferiority trialsDM Neonatology · Aug 2021
2. ×1Bioavailability and BioequivalenceMD Pharmacology · Apr 2015
3. ×1Describe the statistical design of a Non inferiority trialsMCh Surgical Oncology · Aug 2012

Further reading

Related Stateazy articles