AI for Doctors / Theory / Lesson 4

Sensitivity, specificity, PPV: the numbers that decide if you trust an alert

Same model, different ward, very different usefulness. Play with prevalence and watch the false alarms.

10 min

Every yes/no AI tool can be laid out in a 2 × 2 table, the same one you used for diagnostic tests in community medicine. Sensitivity asks: of the people with the disease, how many did the AI catch? Specificity asks: of the people without it, how many did it correctly clear? These belong to the tool.

But at the bedside you want the other direction: the AI has flagged this patient, so how likely is it that they have the disease? That is the positive predictive value, and it depends heavily on how common the disease is where you work.

Same AI, different ward
Has disease (300)
No disease (700)
AI says yes
270true positive
70false alarm
AI says no
30missed
630true negative
PPV: alert is right79%
NPV: all-clear is right95.5%
Alerts per real case1.3

Sensitivity and specificity did not move, but drop the prevalence to 1% and most alerts become false alarms. This is why an AI tool validated in a tertiary ICU can flood a screening clinic with alerts.

Too many false alarms lead to alert fatigue: staff start clicking past alerts, including the true ones. Before a tool is switched on, work out its PPV at your hospital’s prevalence, not the study’s.

Accuracy: the share of all calls that were right. It can look high just because most people are healthy: a tool that calls everyone normal is 99% “accurate” at 1% prevalence.
F1 score: a single number combining PPV (precision) and sensitivity (recall). Common in AI papers; hides the trade-off.

Watch

Why PPV collapses at low prevalence
3 min · on-screen captions, no voice

Check yourself

1. Sensitivity and specificity stay at 90%. Prevalence falls from 30% to 1%. PPV will:
2. Which number answers "the AI flagged my patient; how likely is disease?"
3. A tool reports 99% accuracy for a disease with 1% prevalence. You should:

Educational material, not clinical advice. Interactive numbers are teaching examples unless a source is named.