Will your model work on new patients?

Start here

Why does a model look better in the patients it was built on?

A regression model chooses its coefficients to fit your patients as closely as possible. Some of what it fits is real, and some is chance variation that will not repeat in the next patients.

This is overfitting. The amount by which performance in the development patients exceeds performance in new patients is the optimism. Van Calster et al. (2019) describe what an overfitted model does in new data: lower discrimination, and predicted risks that are too extreme, with high-risk patients given risks that are too high and low-risk patients risks that are too low.

Optimism = performance in the development patients (apparent) − performance in new patients

Optimism is larger when:

• there are few outcome events for the number of candidate predictors;
• predictors are chosen by looking at the same data (for example, keeping those with p < 0.05);
• continuous predictors are cut at the cut-points that happen to work best in these patients.

Interactive

Add useless predictors and watch the C-statistic rise in the development patients and fall in new patients

Every added predictor lets the model fit your 120 patients more closely, even when the predictor is a random number. New patients do not share that chance pattern.

Internal validation

How can you estimate performance in new patients using only your own data?

Internal validation uses the development data set to estimate how much of the apparent performance is optimism. There are three common designs.

Steyerberg et al. (2001) compared these designs in samples drawn from a large trial data set. Split-sample analyses gave overly pessimistic estimates of performance with large variability. Bootstrapping gave stable estimates with low bias, and they recommended it for internal validation of logistic regression models. Collins et al. (2014) described a random split of one data set as a weak and inefficient design that does not constitute external validation.

The formula

Worked example: bootstrap optimism correction

Simulated teaching data: 200 patients, 37 deaths, 10 candidate predictors (3 real, 7 noise), 3.7 events per candidate predictor. Because the data were simulated, the model could also be tested in 100,000 new patients.

Corrected C = apparent C − average over resamples of (C of the resample model in its resample − C of the same model in the original data)

Apparent C

the model fitted and tested in the same 200 patients

Resample

200 patients drawn from the 200 with replacement

Optimism of one resample

how much better that resample model looks in its own resample than in the original patients

Calculated in Python (scikit-learn 1.7.2, statsmodels 0.14.5), unpenalised logistic regression, 200 resamples. The C-statistic of the true model in the new patients was 0.785: even a model with the right predictors loses some discrimination when its coefficients are estimated from 37 events.

Measuring performance

Does the model put patients in the right order, and are its risks right?

These are two different questions. Discrimination asks whether patients who have the event get higher predicted risks. Calibration asks whether a predicted risk of 20% means that about 20 in 100 such patients have the event.

Discrimination: the C-statistic

For a yes/no outcome the C-statistic equals the area under the ROC curve (Housseine et al., 2022). TRIPOD+AI notes that it is called the c statistic or AUC for binary outcomes and the c index for time-to-event outcomes. For survival models, Harrell's C counts the comparable pairs in which the patient with the earlier event had the higher predicted risk. Somers' Dxy, reported by the R rms package, is 2 × (C − 0.5).

Calibration: intercept, slope and plot

• Calibration-in-the-large (calibration intercept): compares the average predicted risk with the overall event rate. Target 0. Negative values mean the model overestimates risk, positive values that it underestimates (Van Calster et al., 2019).
• Calibration slope: the spread of the predicted risks. Target 1. A slope below 1 means the risks are too extreme (Van Calster et al., 2019), the typical result of overfitting.
• Calibration plot: observed proportion against predicted risk, by groups or with a smoothed curve. It shows where in the risk range the model goes wrong.
• Hosmer-Lemeshow test: Van Calster et al. (2019) recommend against it: it groups patients arbitrarily, its P value does not show the type or size of miscalibration, and it has low power.

Steyerberg and Vergouwe (2014) proposed four measures for validation: calibration-in-the-large (A), calibration slope (B), discrimination with a concordance statistic (C), and clinical usefulness with decision-curve analysis (D).

External validation

Will the model work in another year or another hospital?

Internal validation estimates performance in new patients from the same population. External validation tests the finished model, unchanged, in patients who were not used to build it.

• Temporal validation: patients from the same centre in a different time period (Collins et al., 2014).
• Geographic validation: patients from other hospitals, regions or countries.
• Justice, Covinsky and Berlin (1999) separated reproducibility, accurate predictions in new patients from the same population, from transportability, accurate predictions in a different but plausibly related population. They noted that a prognostic system is often limited to a single historical period, geographic location, methodologic approach, disease spectrum or follow-up interval.

Why performance changes in a new setting

• Different event rate: the average predicted risk no longer matches the observed rate, so calibration-in-the-large is wrong. Van Calster et al. (2019) note that calibration-in-the-large is irrelevant at internal validation but highly relevant at external validation.
• Different strength of the predictors: measurement, treatment or case mix changes the effect of each predictor, and the calibration slope moves away from 1.
• Narrower case mix: if patients are more alike, they are harder to rank, and the C-statistic falls even when the model is correct.

Updating a model starts with the simplest change. Van Calster et al. (2019) suggest that updating may start with changing the intercept to correct calibration-in-the-large. Nugawela et al. (2022) did the equivalent for a Cox model by recalibrating the baseline survival in each validation cohort.

Calibration is often left out. In a review of 78 external validation articles, 53 (68%) did not report evaluating calibration, and only 11 (14%) presented a calibration plot (Collins et al., 2014).

Interactive

Take a published model to a new hospital and watch the C-statistic, the calibration slope and the calibration intercept

You change the hospital's patients, not the model. See which change affects discrimination, which affects calibration, and what updating the intercept or slope repairs.

The bootstrap

How do you get a confidence interval when there is no formula?

The confidence interval for a mean comes from a formula. For a median, a ratio of two medians, or a C-statistic after model building, the formula is missing or unreliable. The bootstrap estimates the sampling variation directly from your data.

1. Draw n patients from your n patients, with replacement.
2. Calculate the statistic (median, ratio, C-statistic) in that resample.
3. Repeat many times. IBM's SPSS documentation recommends at least 1,000 bootstrap samples for percentile and BCa intervals.
4. Percentile interval: the 2.5th and 97.5th percentiles of the bootstrap statistics give the 95% interval.
5. Bootstrap standard error: the standard deviation of the bootstrap statistics.

In the ICU data, 2,000 resamples gave a 95% percentile interval for the median of 4 to 11 days, with a bootstrap standard error of 1.60 days. For the mean (8.73 days) the percentile interval was 5.53 to 12.73 days, against 4.60 to 12.86 days from the t formula, which assumes the mean is normally distributed.

The bootstrap only resamples the patients you have. If the sample is small or unrepresentative, every resample shares that problem, and the interval can be too narrow. The bias-corrected and accelerated (BCa) interval is an adjusted version that SPSS describes as more accurate, at the cost of more computing time.

Interactive

Draw resamples one at a time, then thousands, and watch the percentile interval for the median form

Each resample is one version of your study that could have happened. The spread of their medians is the uncertainty in your median.

The jackknife

What is the jackknife, and how is it different from the bootstrap?

The jackknife leaves out one patient at a time and recalculates the statistic, giving n estimates from n patients. Nothing is random: everyone who repeats it gets the same answer.

Jackknife SE = √[ (n − 1) ÷ n × Σ (θ₍ᵢ₎ − θ̄₍.₎)² ]

θ₍ᵢ₎

the statistic calculated with patient i left out

θ̄₍.₎

the average of the n leave-one-out values

For the mean ICU stay, the jackknife SE is 1.93 days, exactly the usual SD ÷ √n = 7.46 ÷ √15 = 1.93. For the median it behaves badly.

The jackknife is also used to find influential patients: if leaving out one patient changes a coefficient a lot, that patient drives the result. SPSS saves this as DfBeta, the estimated change in a coefficient if a case is removed.

Sensitivity analysis

Would the conclusion change if an assumption were different?

Every analysis rests on assumptions about missing data, outliers, the analysis method and the outcome definition. A sensitivity analysis repeats the analysis under other plausible assumptions and asks whether the conclusion stays the same.

ICH E9(R1) (2019) defines sensitivity analysis as a series of analyses that explore whether inferences from the main estimator hold when its modelling assumptions are departed from and when the data have limitations. The analyses should focus on the same estimand and be pre-specified. The addendum gives tipping point analysis as an example: finding how far an assumption must move before the interpretation changes.

Common sensitivity analyses

Thabane et al. (2013) list questions such as whether the results change with:

• a different definition of the outcome, for example a different cut-off;
• a different method of analysis;
• a different way of handling missing data;
• protocol deviations and degree of compliance;
• ignoring correlation between repeated measurements in a patient;
• a non-normal distribution or outliers.

Choosing a sensitivity analysis

Morris, Kahan and White (2014) proposed three questions:

1. Does the proposed sensitivity analysis address the same question as the primary analysis?
2. Is it possible for it to arrive at a different conclusion?
3. If it does, is there a genuine degree of uncertainty as to which analysis will be believed?

CONSORT 2025 lists subgroup and sensitivity analyses as separate kinds of additional analysis. An analysis that ignores clustering in a cluster-randomised trial is not a useful sensitivity analysis, because there would be no uncertainty about which result to believe (Morris et al., 2014).

Limits of models

How far can you trust a model's conclusions?

A model that fits the data well is one plausible description of them, not proof that its structure is the true one. Its conclusions deserve trust in proportion to the checks it has passed.

1. Assumptions checked: linearity of continuous predictors, proportional hazards for Cox models, independence of observations.
2. No single patient decides the result: leave-one-out (jackknife) or DfBeta checks for influential observations.
3. Optimism measured: bootstrap internal validation, with the corrected C-statistic and calibration slope.
4. Conclusions unchanged under other plausible assumptions: pre-specified sensitivity analyses.
5. Performance confirmed elsewhere: temporal and geographic validation, with calibration reported.
6. Measurement comparable: Van Calster et al. (2019) point out that medical data contain measurement error, for example biomarker values that vary with assay kits, which can change calibration between settings.

A model is a prediction built from past patients. When treatment, case mix or measurement changes, its predictions need to be checked again.

Decide

Which approach does your study need?

Answer the questions in order.

Comparison

Pairs that residents confuse

Internal or external validation?

Discrimination or calibration?

Bootstrap or jackknife?

In a real paper

Internal and external validation in published studies

How to run it

Validation and bootstrap in SPSS, R and jamovi

Checked against the IBM SPSS Statistics 24 manuals, IBM's online documentation, the CRAN reference manuals and the jamovi documentation.

SPSS

1. Analyze > Regression > Binary Logistic... Select the outcome and the covariates.
2. Save: tick Probabilities to save each patient's predicted risk as a new variable.
3. Analyze > ROC Curve... Put the saved probability in Test Variable and the outcome in State Variable, and give the positive value. The output gives the area under the curve with its confidence interval. Recent versions also have a newer ROC Analysis procedure.
4. Bootstrap confidence intervals: in a procedure that supports bootstrapping (Frequencies, Explore, Binary Logistic Regression, Cox Regression and others), click Bootstrap, select Perform bootstrapping, set the Number of samples (at least 1,000 for percentile and BCa intervals), set a seed, and choose Percentile or Bias corrected accelerated (BCa).

R (rms, boot and pROC packages)

library(rms)
fit <- lrm(died ~ age + lactate + asa + shock, data = d, x = TRUE, y = TRUE)
v <- validate(fit, method = "boot", B = 200)   # default B is 40
v                  # index.orig, training, test, optimism, index.corrected
                   # rows include Dxy, R2, Intercept, Slope
0.5 + v["Dxy", "index.corrected"] / 2          # optimism-corrected C (Dxy = 2(C - 0.5))
plot(calibrate(fit, method = "boot", B = 200))  # bias-corrected calibration curve

library(pROC)
r <- roc(d$died, predict(fit, type = "fitted"))
ci.auc(r)                                       # DeLong method by default
ci.auc(r, method = "bootstrap", boot.n = 2000)

library(boot)
med <- function(x, i) median(x[i])             # statistic(data, indices)
b <- boot(icu_days, med, R = 2000)
boot.ci(b, type = c("perc", "bca"))

• validate() and calibrate() need x = TRUE, y = TRUE in the fit. For Cox models use cph() with the same arguments.
• If you selected predictors (for example by backward elimination), the selection must be repeated inside every resample; validate() does this with bw = TRUE.
• In boot() the statistic function receives the data and a vector of indices that define the resample.

jamovi

Analyses → Regression → 2 Outcomes (Binomial) fits logistic regression. The Prediction drop-down gives a classification table. For bootstrap internal validation, use the R code above.

How to report it

What to write in your thesis or paper

Follow TRIPOD+AI (2024), which supersedes TRIPOD 2015. For validation, the relevant items include:

• 10: how the study size was arrived at, and why it was sufficient.
• 12c: the type of model, all model-building steps, and the method for internal validation.
• 12e: all measures and plots used to evaluate performance (discrimination, calibration, clinical utility).
• 12f and 24: any model updating, such as recalibration, and its results.
• 23a: performance estimates with confidence intervals.
• 26: limitations, including sample size and overfitting.

Worked sentence, with the teaching numbers:

“The model was developed in 200 patients, of whom 37 died within 30 days (3.7 events per candidate predictor). The apparent C-statistic was 0.81. Internal validation with 200 bootstrap resamples, repeating all modelling steps, estimated an optimism of 0.05, giving an optimism-corrected C-statistic of 0.76. The optimism-corrected calibration slope was 0.74, indicating predictions that were too extreme, so the coefficients were shrunk before presentation. The model has not been externally validated.”

For a sensitivity analysis in a trial, state the assumption varied, whether the analysis was pre-specified, and the result of each analysis alongside the primary one (CONSORT 2025, item 21d).

By specialty

Validation and bootstrap problems by specialty

Common myths

Common mistaken beliefs

• ✗Myth: A high C-statistic in my own data shows that the score works.

  ✓That is the apparent performance, and it is optimistic. In the worked example it fell from 0.814 to 0.763 after bootstrap correction.

• ✗Myth: Splitting the data into a training half and a test half is the best validation.

  ✓A random split builds the model on fewer patients and gives unstable, pessimistic estimates (Steyerberg et al., 2001). In the teaching data, 200 splits gave C-statistics from 0.599 to 0.861.

• ✗Myth: A good C-statistic means the predicted risks are right.

  ✓Discrimination and calibration are separate. A model can rank patients well while every predicted risk is twice too high.

• ✗Myth: A non-significant Hosmer-Lemeshow test proves good calibration.

  ✓The test has low power and does not show the type or size of miscalibration. Van Calster et al. (2019) recommend a calibration plot with intercept and slope instead.

• ✗Myth: The bootstrap creates new data, so it can rescue a small study.

  ✓It resamples the same patients. It measures the uncertainty in your sample; it cannot add information that the sample does not contain.

• ✗Myth: Any extra analysis counts as a sensitivity analysis.

  ✓Morris et al. (2014) require a sensitivity analysis to address the same question as the primary analysis. CONSORT 2025 lists subgroup and sensitivity analyses separately.

Why this matters to you

Five consequences of skipping validation

1. Protocol or paper sent back

   TRIPOD+AI item 12c asks for the method of internal validation. "The model was tested on the study patients" does not meet it.

2. An inflated result in your thesis

   An apparent C-statistic and a model with 3.7 events per candidate predictor overstate how well the score will work.

3. Questions you cannot answer at the viva

   "What is the calibration slope?" and "Was it validated in other patients?" need numbers, not a P value from the Hosmer-Lemeshow test.

4. Misreading papers

   A model reported without calibration or external validation is not ready for your patients: 68% of external validation articles in one review did not report calibration.

5. Harm to patients

   A score that over-predicts risk in your population, as the UK retinopathy equations did before recalibration in India, sends low-risk patients for unnecessary tests, and one that under-predicts delays care.

History

History of resampling and model validation

1. 1949

   Maurice Quenouille

   Proposed the method later called the jackknife, in "Approximate tests of correlation in time-series", and expanded it in "Notes on bias in estimation" (1956).

2. 1974

   M. Stone

   Published "Cross-validatory choice and assessment of statistical predictions", applying cross-validation to choosing and assessing predictions.

3. 1979

   Bradley Efron

   Introduced the bootstrap in "Bootstrap methods: another look at the jackknife".

4. 1982

   Frank Harrell and colleagues

   Published "Evaluating the yield of medical tests" in JAMA. Later authors cite it as the origin of the C-index for survival data.

5. 1996

   Frank Harrell, Kerry Lee and Daniel Mark

   Described overfitted models and recommended that predictive accuracy be validated with bootstrapping or cross-validation before predictions are used in new patients.

6. 1999

   Amy Justice, Kenneth Covinsky and Jesse Berlin

   Separated reproducibility from transportability, and proposed a hierarchy of generalisability across periods, places, methods, disease spectrum and follow-up.

7. 2001

   Ewout Steyerberg and colleagues

   Showed that split-sample validation is inefficient and recommended bootstrapping for internal validation of logistic regression models.

8. 2014

   Gary Collins and colleagues

   Reviewed 78 external validation articles: 68% did not report calibration.

9. 2015

   TRIPOD group

   Published the TRIPOD statement, a 22-item checklist for reporting prediction model development and validation.

10. 2019

    Ben Van Calster and colleagues

    Published "Calibration: the Achilles heel of predictive analytics", defining calibration intercept and slope and recommending against the Hosmer-Lemeshow test.

11. 2024

    TRIPOD+AI group

    Published TRIPOD+AI, a 27-item checklist for models built with regression or machine learning, which supersedes TRIPOD 2015.

Guidelines

Validation and sensitivity analysis in guidelines

Reading a paper

Five checks on any prediction model paper

• Are the number of events and the number of candidate predictors both reported?
• Is the C-statistic apparent, optimism-corrected, or from external data?
• Is calibration reported with a plot, intercept and slope, and not only a Hosmer-Lemeshow P value?
• Was the model validated in patients from another time or place, and was it updated there?
• Were all modelling steps, including predictor selection, repeated inside the bootstrap or cross-validation?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

TNMGRMU questions on bootstrap and model validation

Questions from MD, MS, DM, MCh and Ph.D. papers that this chapter helps you answer. Tap a sitting to open the university's own question paper.

1. ×1Sensitivity AnalysisMD Community Medicine · May 2022
2. ×1Regarding randomised clinical trials, explain the following terms: a. External validity. b. Allocation concealment. c. Stratification. d. Sensitivity analysis. e. Fragility IndexDM Critical Care Medicine · Aug 2022