Game 2 · Diagnostic tests
How do you pick the best cut-off from an ROC curve?
ROC curves, area under the curve and choosing the best cut-off.
Read with: Diagnostic tests21 DNB questions
Play
Move the cut-off, read the curve
Some rounds ask you to find the cut-off with the highest Youden index, watching the point move along the ROC curve. Some ask you to estimate the area under a curve. Some show two tests and ask which is better. Every sixth round is a short question. You have three lives.
Loading the first round…
The idea
One point for every cut-off
A test that gives a number, such as a blood level or a score, has no single sensitivity and specificity. Each possible cut-off gives its own pair. The receiver operating characteristic (ROC) curve plots all of these pairs on one graph: sensitivity (the true-positive rate) on the y-axis and 1 − specificity (the false-positive rate) on the x-axis.
A cut-off above every result calls everyone negative, which is the point (0, 0). A cut-off below every result calls everyone positive, which is the point (1, 1). So every ROC curve runs from the lower left corner to the upper right corner. A test no better than chance lies on the diagonal between them. The closer a curve comes to the top left corner (0, 1), the better the test separates diseased from healthy people. Lowering the cut-off moves the point up and to the right.
Area under the curve
One number for the whole curve
The area under the curve (AUC) is 0.5 for a test on the diagonal and 1.0 for a perfect test. Hanley and McNeil (1982) showed that the AUC is the probability that a randomly chosen person with the disease has a more abnormal result than a randomly chosen person without it. This is the same quantity the Wilcoxon (Mann–Whitney) statistic estimates.
The curves in the game come from the binormal model: scores in the healthy and the diseased are each normally distributed. Then the AUC has a formula.
AUC = Φ[(m₁ − m₀) ÷ √(s₀² + s₁²)]
- m₀, s₀
- mean and SD of the scores in people without the disease
- m₁, s₁
- mean and SD of the scores in people with the disease
- Φ
- the standard normal cumulative probability
With real data and a graded result, the area is found by joining the observed points with straight lines and adding up the trapezoids under them. Report the AUC with its 95% confidence interval. When two tests are done on the same patients, their AUCs are correlated, so compare them with a paired method such as that of DeLong, DeLong and Clarke-Pearson (1988).
Myth: A high AUC means the test works well at the cut-off I use.
AUC summarises all cut-offs. In practice you use one cut-off, so look at the sensitivity and specificity at that point.
Myth: The test with the larger AUC is better at every cut-off.
Curves can cross. One test can be better where high sensitivity is needed and the other where high specificity is needed. The partial AUC compares them only over the range of false-positive rates that matters clinically.
Myth: An AUC below 0.5 means the test is useless.
It means the test ranks people the wrong way round, with lower values in the diseased. Reverse the direction of positivity and the AUC becomes 1 minus the original value.
Choosing the cut-off
The Youden index and when to move away from it
Youden index J = sensitivity + specificity − 1
At any point on the curve, J is the vertical distance from the diagonal up to the point. The cut-off with the largest J, or the point closest to the top left corner, gives equal weight to a false negative and a false positive. In the worked example of the diagnostic tests chapter, grade 4 or more had both the highest J (0.617) and the shortest distance to the corner (0.283). The two rules do not always pick the same cut-off.
| Feature | Choose a lower cut-off | Choose a higher cut-off |
|---|---|---|
| When | A missed case is worse, as in neonatal sepsis or screening | A false positive leads to harmful or costly treatment |
| On the curve | Point moves up and to the right | Point moves down and to the left |
| You gain | Sensitivity | Specificity |
Writing the answer
How to write this in a 10-mark DNB answer
- Define the ROC curve: a plot of sensitivity against 1 − specificity for every possible cut-off of a test with a continuous or graded result.
- Draw it first: the square with both axes from 0 to 1, the diagonal labelled “chance, AUC 0.5”, one good curve bowing to the top left corner, and three marked cut-offs.
- Explain the corners (0, 0) and (1, 1), and that lowering the cut-off moves the point up and to the right.
- Define the AUC: 0.5 chance, 1.0 perfect, and its meaning as the probability that a random diseased person scores higher than a random healthy one (Hanley and McNeil, 1982). Mention the 95% CI.
- Give the Youden index (J = sensitivity + specificity − 1) for choosing a cut-off, and say when you would move away from it.
- Uses: choosing a cut-off, comparing two tests over all cut-offs (paired DeLong method on the same patients). Limitations: ignores the cost of each error, curves can cross, depends on the patient spectrum.
The one thing to remember
The ROC curve shows every cut-off at once. The AUC compares tests. The cut-off you actually use is a clinical choice about which error costs more.
Past papers
DNB questions this game helps you answer
Questions from NBE theory papers for DNB, DrNB and Diploma courses, most often asked first. The tag after each question is the specialty and the year it was set.
- ×1Describe ROC curve and its clinical significance. [4]
- ×1Receiver operating curve characteristics. [3]
- ×1Define sensitivity, specificity, likelihood ratios and ROC curve in research.
- ×1ROC (Receiver Operating characteristic curve) and AUC (Area under the curve).
- ×1ROC analysis.
- ×1Receiver operating characteristic curve (ROC).
- ×1Explain the Receiver Operating Characteristic (ROC) Curve.
- ×1Receiver Operating Characteristic (ROC) curve.
- ×1Receiver and operating characteristic curve (ROC) and its implications.
- ×1Briefly discuss Receiver operating charecteristics and Kaplan-Meier analysis.
- ×12+3+5 a) Briefly describe the ROC curve.
- ×1Interpret the results of Test A, B, C from above given figure of ROC curve.
- ×1ROC curve.
- ×1Receiver Operating Characteristic (ROC) curve. 10
- ×1Explain the significance of the ROC curve for statistical analysis.
- ×1ROC Analysis
- ×1The Receiver Operating Characteristics (ROC).
- ×1ROC curve (Receiver Operating Characteristic curve).
- ×1ROC Curve.
- ×1ROC.
- ×1Briefly discuss the following: a) Measures of variance b) ROC curve c) Odd's ratio
Read next