AI for Doctors / Theory / Lesson 9

Bias, dataset shift and how good tools fail

Why an AI that worked in Boston may not work in Bilaspur, and the human side: automation bias.

10 min

A model learns whatever patterns are in its data, including the unfair or accidental ones. If dark skin is rare in a dermatology training set, performance on dark skin will usually be worse. If one hospital’s portable X-rays are mostly from sicker patients, the model may learn “portable means sick” rather than the disease.

Trained at Hospital A, deployed at Hospital B
A: sensitivity84%
A: specificity84%
B: sensitivity84%
B: specificity84%

The model and its cut-off are frozen. Only the incoming data changed, yet one of the two numbers falls. This is dataset shift, and it is the commonest reason a published tool disappoints in a new hospital. Ask for external validation on patients like yours.

A well-known example: the Epic Sepsis Model, used in hundreds of US hospitals, was externally validated at the University of Michigan in 2021. It missed about two-thirds of sepsis cases and had an AUC of 0.63, far below the vendor’s figures (Wong et al., JAMA Internal Medicine 2021).

Automation bias

People tend to accept a machine’s answer, even when their own judgement was right. Studies in radiology and other fields have shown clinicians changing correct reads to match an incorrect AI suggestion. The fix is a habit: form your own impression first, then look at the AI.

Dataset shift (drift): the data the model sees in use differs from its training data: new scanner, new population, new coding practice, a pandemic.
Spectrum bias: a model tested only on clear-cut cases looks better than it will be on the mixed cases of real practice.
Label bias: the “truth” used for training was itself biased, such as using healthcare spending as a stand-in for health need.

Check yourself

1. A tool works at the hospital that built it but poorly at yours. The most likely cause:
2. Automation bias means:
3. The best protection before adopting a published model locally is:

Educational material, not clinical advice. Interactive numbers are teaching examples unless a source is named.