Game 17 · Describing data and agreement
Do two observers really agree?
Cohen's kappa, chance agreement, and agreement versus correlation.
Read with: Kappa and agreement, ICC and Bland-Altman7 DNB questions
Play
Read the table, estimate kappa
Two raters classify the same cases. In most rounds you see their agreement table and estimate Cohen's kappa with a slider, then see the observed agreement, chance agreement and kappa worked out. Some rounds use a rare finding, some ask for the Landis and Koch label, and some ask about agreement versus correlation. You have three lives.
Loading the first round…
The idea
Some agreement happens by chance
When two observers classify the same patients, they will agree on some patients even if neither looks carefully. Imagine two readers who each call 30% of films positive by tossing a biased coin. Both call a film positive with probability 0.30 × 0.30 = 0.09 and both call it negative with probability 0.70 × 0.70 = 0.49. They agree on 58% of films with no skill at all. This is the agreement expected by chance. It is higher when one category is common.
Cohen (1960) proposed kappa to correct the observed agreement for chance. Kappa is 1 when agreement is perfect, 0 when agreement equals what chance would produce, and below 0 when the raters agree less often than chance. Kappa measures reliability: whether observers give the same answer. It does not say whether the answer is correct.
The calculation
Three numbers from one table
κ = (pₒ − pₑ) ÷ (1 − pₑ)
- pₒ
- observed agreement: the sum of the diagonal cells ÷ n
- pₑ
- chance agreement: for each category, (row total ÷ n) × (column total ÷ n), added over the categories
- 1 − pₑ
- the largest possible agreement beyond chance
Worked example from the kappa chapter: two sonologists report gallstones on the same 100 scans and agree on 20 + 65 = 85, so pₒ = 0.85. Sonologist A reports stones in 25% and B in 30%, so pₑ = 0.25 × 0.30 + 0.75 × 0.70 = 0.60. κ = (0.85 − 0.60) ÷ (1 − 0.60) = 0.625.
Most papers quote the labels of Landis and Koch (1977): below 0 poor, 0 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, 0.81 to 1.00 almost perfect. The authors themselves called the divisions clearly arbitrary. Quote the kappa with its 95% confidence interval, and the table, not the label alone.
High agreement, low kappa
Why a rare finding gives a low kappa
Feinstein and Cicchetti (1990) described the paradox you met in the game. When one category is much more common, both raters say “no” most of the time, so chance agreement is already high. Kappa can only measure the small amount of agreement left above that. In the chapter's teaching tables, all with 90% observed agreement, kappa falls from 0.80 at 50% prevalence to below 0 at 95% prevalence.
Byrt, Bishop and Carlin (1993) recommended reporting measures of prevalence and bias with kappa. For a 2 × 2 table with cells a, b, c, d:
- prevalence index
- (a − d) ÷ n: far from 0 when one category is common
- bias index
- (b − c) ÷ n: far from 0 when one rater calls positive more often
- PABAK
- prevalence-adjusted bias-adjusted kappa = 2pₒ − 1
PABAK is not a replacement for kappa. It shows what kappa would be with 50% prevalence and no bias.
Which statistic
Categories, ordered grades or measurements
| Feature | Correlation (Pearson r) | Agreement |
|---|---|---|
| Asks | Do the two readings rise and fall together? | Do they give the same value in the same patient? |
| One rater always reads higher | r is unchanged | Agreement is poor |
| Wide range of patients | Makes r high almost automatically | Limits of agreement do not depend on the range |
- Two categories, or several unordered ones: Cohen's kappa.
- Ordered grades (none, mild, moderate, severe): weighted kappa (Cohen, 1968), which gives partial credit to near misses. Choose linear or quadratic weights before the analysis. Fleiss and Cohen (1973) showed that quadratic weighted kappa is equivalent to an intraclass correlation coefficient.
- Measurements on a scale (mm, g/dL, a total score): the intraclass correlation coefficient (ICC) and Bland-Altman limits of agreement. See the ICC and Bland-Altman chapter.
Myth: A high correlation between two observers shows they agree.
A method that always reads 1 g/dL too high has a perfect correlation and does not agree.
Myth: 90% agreement means the observers are reliable.
If the finding is rare, chance alone may give close to 90%. Look at kappa as well.
Myth: A low kappa always means the observers are unreliable.
With a rare finding, kappa can be low although agreement is high. Report observed agreement, the prevalence index and PABAK beside it.
Writing the answer
How to write this in a 10-mark DNB answer
- Define kappa as a measure of agreement between two observers on a categorical result, corrected for agreement expected by chance (Cohen, 1960).
- Draw the 2 × 2 agreement table first, rater A in rows and rater B in columns, with the marginal totals.
- Write the formula and work one example: pₒ, then pₑ from the margins, then κ.
- Give the meaning of the values (1 = perfect, 0 = chance, below 0 = less than chance) and the Landis and Koch labels, saying they are arbitrary benchmarks.
- Add the limitations: kappa depends on prevalence and bias (the paradoxes), so report observed agreement and PABAK too.
- End with the variants: weighted kappa for ordered grades, Fleiss' kappa for more than two raters, and the ICC or Bland-Altman for measurements. In a pathology question, give the example of two pathologists grading the same slides.
The one thing to remember
Kappa is the agreement two observers achieved beyond chance, as a share of the most they could have achieved beyond chance.
Past papers
DNB questions this game helps you answer
Questions from NBE theory papers for DNB, DrNB and Diploma courses, most often asked first. The tag after each question is the specialty and the year it was set.
- ×1Bland Altman plot.
- ×1Cohen's kappa statistics and its use as a research tool in Pathology.
- ×1Define reliability and validity.
- ×1What is the significance of measuring reliability and validity.
- ×1Explain validity and reliability of diagnostic test.
- ×1Kappa measure (in terms of statistical analysis of medical data).
- ×1What are the different types of reliability and validity measures? How do you establish a good reliability and validity of a rating scale in psychiatry?
Read next