Do two methods or raters give the same measurement?

Start here

Two methods correlate at 0.99. Do they give the same reading?

Correlation asks whether two measurements rise and fall together. Agreement asks whether they give the same value in the same patient. A method that always reads 1 g/dL too high has a perfect correlation and does not agree.

Bland and Altman (Lancet, 1986) gave the reasons why the correlation coefficient does not show agreement:

1. r measures the strength of a linear relation. Perfect agreement needs the points on the line of equality. Perfect correlation needs them on any straight line.
2. Changing the scale of one method (doubling it, or adding a constant) does not change r but does change agreement.
3. r depends on the range of values in the sample. A study that recruits a wide range of patients will almost always find a high r.
4. The significance test of r only shows that two methods designed to measure the same thing are related, which is expected.

Two statistics answer the agreement question: the intraclass correlation coefficient (ICC), a unit-free index, and Bland-Altman analysis, which gives the size of the differences in the units of measurement. This chapter covers both. Pearson r itself is explained in the correlation and regression chapter.

Intraclass correlation

What does an intraclass correlation coefficient measure?

Think of every reading as the patient's true value plus measurement error. The readings vary for two reasons: patients really differ from each other, and the measurements contain error. The ICC is the share of the total variance that comes from real differences between patients.

ICC = variance between patients ÷ (variance between patients + error variance)

variance between patients

how much the true values differ from patient to patient

error variance

variation between raters or methods, and random error, depending on the ICC form

range

0 to 1 in theory. Estimates from data can fall below 0 when there is almost no variance between patients

Koo and Li (2016) describe reliability in the same way, as the ratio of true variance to true variance plus error variance. In practice the variances come from an analysis of variance (ANOVA) table: one row per patient, one column per rater or method. The mean square between patients (MS_R), the mean square between raters (MS_C) and the residual mean square (MS_E) go into the formulas in the worked example.

Because the ICC is a share of variance, it depends on how different the patients are from each other. The same instrument gives a lower ICC in a sample of similar patients. Bland and Altman (1990) concluded that the ICC is not appropriate for assessing whether two methods of measurement can be used interchangeably.

Interactive

Change the spread of patients and watch the ICC change while the error stays the same

The limits of agreement describe the differences within each patient, so they do not depend on the range of patients. The ICC does.

Decide

Which ICC: one-way or two-way, agreement or consistency, single or average?

Software offers several ICCs, and they give different numbers from the same data. Koo and Li (2016) list 10 forms. Three questions about your study decide which one you need.

1. The model: who measures each patient?

• One-way random effects: each patient is measured by a different set of raters, so a rater cannot be linked to a column of data. Differences between raters, interactions and random error are all counted as error.
• Two-way random effects: the same raters measure every patient, and they are a random sample of the raters who could use the method. The result can be generalised to similar raters.
• Two-way mixed effects: the same raters measure every patient, and these are the only raters of interest. The conclusion is confined to them.

2. The type: must the values be identical?

3. The unit: one reading or the mean of several?

Each model gives an ICC for a single rating and one for the mean of k ratings. Choose the unit you will use in practice. If each patient will be measured once by one rater, report the single-measure ICC. The average-measure ICC is higher, because averaging reduces error.

Two naming systems for the same coefficients

MS_W is the within-patient mean square of the one-way ANOVA. n is the number of patients and k the number of raters. Each row also has an average-measure form, ICC(1,k), ICC(2,k), ICC(3,k): 5 models × 2 units = the 10 forms of McGraw and Wong. The 6 forms of Shrout and Fleiss are a subset of them. The random and mixed models share formulas but differ in what the result may be generalised to.

Interactive

Give one resident a systematic offset and watch the six ICC forms separate

Consistency forms ignore a rater who always reads higher. Absolute agreement forms count it as error. Averaging over raters raises every form.

Interpretation

Is an ICC of 0.80 good?

Koo and Li (2016) suggested bands for reliability studies, based on the 95% confidence interval of the ICC and not on the point estimate alone.

• Use the confidence interval. In the worked example below the absolute agreement ICC is 0.97, but its 95% CI runs from 0.17 to 0.99. By these bands the reliability could be anything from poor to excellent. Ten patients are too few.
• The bands assume a suitable study. Koo and Li note that a low ICC can reflect a lack of variability among subjects, few subjects or few raters. As a rule of thumb they suggest at least 30 heterogeneous subjects and at least 3 raters.
• The cut-offs are conventions. An ICC tells you nothing about the size of the error in mmHg or g/dL. A clinically important error can coexist with a “good” ICC when the patients vary widely.
• State the form. Values from different forms are not interchangeable. Koo and Li recommend reporting the software, the model, the type and the definition (single or average) used.

Agreement in units

How far apart can the two readings be in one patient?

Bland and Altman (1986) replaced the correlation coefficient with a plot and two numbers: the average difference between the methods (bias) and the range within which most differences fall (limits of agreement).

1. Before the study, decide how large a difference would change a clinical decision. Bland and Altman wrote that this should ideally be defined in advance, both to interpret the comparison and to choose the sample size.
2. For each patient calculate the difference (new − old) and the mean of the two readings.
3. Plot the difference against the mean. Do not plot it against either method alone: the difference is related to each single method, a statistical artefact.
4. Bias = mean of the differences. It shows whether one method reads higher on average.
5. 95% limits of agreement = bias ± 1.96 × SD of the differences. About 95% of differences are expected between them if the differences are roughly normal. Check this with a histogram of the differences. The measurements themselves need not be normal.
6. Look at the plot for a trend (differences that change with the size of the measurement) and for outliers.
7. Compare the limits with the difference decided in step 1. If both limits lie inside it, the methods can be used interchangeably. The decision is clinical, not statistical.

Precision of the limits. The limits are estimates, and a second sample would give different limits. In the 1986 paper the standard error of the bias is √(s² ÷ n) and the standard error of each limit is about √(3s² ÷ n), where s is the SD of the differences. Confidence intervals then use t with n − 1 degrees of freedom. Small studies have wide intervals around both limits.

Several readings per patient. The basic calculation assumes one pair of readings per patient. Bland and Altman (1999) extended the limits of agreement to data with repeated measurements, so that many pairs from the same patient are not treated as independent.

Interactive

Add bias and error to a new method and watch r, the ICC and the Bland-Altman plot

Constant bias moves the cloud on the Bland-Altman plot and lowers the agreement ICC, but leaves r and the consistency ICC unchanged. Random error widens the limits and lowers every coefficient. Proportional bias tilts the cloud.

Proportional bias

What if the differences grow as the measurement gets larger?

The basic limits assume the differences do not change in any systematic way across the range. When they widen or tilt, a single pair of horizontal limits is misleading.

• Look first. A funnel shape or a sloping cloud on the Bland-Altman plot is the main check.
• Regress the differences on the means. A slope clearly different from 0 suggests proportional bias. In the worked example the slope is −0.03 g/dL per g/dL (p = 0.52), so there is no evidence of it.
• If the differences are proportional to the mean, take logarithms of both measurements and repeat the analysis. The antilogs of the limits are ratios: in the 1986 paper one echocardiographic method gave values between 0.80 and 1.27 times the other, that is 20% below to 27% above.
• If a log transformation does not remove the trend, Bland and Altman (1999) described a more general regression approach for differences that are related to the size of the measurement.

Repeatability

How much does one method disagree with itself?

Two methods cannot agree better than each one agrees with itself. If the old method gives different readings when repeated, even a perfect new method will not agree with it.

Measure the same patients twice with the same method and plot the differences against the means, as before. The mean difference should be close to 0. If it is not, the first measurement is affecting the second, or the measurement is changing the quantity.

coefficient of repeatability = 2 × SD of the differences between repeated measurements

Bland and Altman (1986) used this definition, adopted by the British Standards Institution: about 95% of differences between two readings on the same patient are expected to be smaller than this value. When the mean difference can be taken as 0, the SD of the differences is √(Σd² ÷ n).

Teaching example (made up): the meter was used a second time in the same 10 patients: 8.2, 8.6, 9.8, 10.7, 11.3, 12.0, 12.8, 13.5, 13.9 and 14.7 g/dL. The differences from the first reading are ±0.1 to ±0.3 g/dL and add to 0.

Σd² = 0.42 → SD of differences = √(0.42 ÷ 10) = 0.205 → coefficient of repeatability = 2 × 0.205 = 0.41 g/dL

within-subject SD (s_w)

SD of the differences ÷ √2 = 0.145 g/dL: the typical error of a single reading

coefficient of variation (CV)

s_w ÷ overall mean × 100 = 0.145 ÷ 11.55 × 100 = 1.3%: the error as a percentage of the value

A single within-subject SD is valid only when the error does not change with the size of the measurement (Bland and Altman, BMJ 1996). When it grows with the mean, analyse the logarithms or express the error as a percentage. Repeatability of a questionnaire score (test-retest reliability) is covered in the reliability and validity of scales chapter.

Other methods

Lin's concordance correlation coefficient, Deming and Passing-Bablok regression

These appear in method comparison papers, especially from laboratories. Each answers a narrower question than Bland-Altman analysis.

Lin's concordance correlation coefficient

Lin (1989) proposed a coefficient that measures how closely paired readings fall on the 45-degree line through the origin, the line of equality. It combines precision (how tightly the points cluster around their own line, which is Pearson r) with accuracy (how far that line is from the line of equality).

CCC = 2 s_xy ÷ [s_x² + s_y² + (mean_x − mean_y)²]

s_xy is the covariance of the two methods and s² their variances. In the worked example CCC = 0.96. Like the ICC it has no units and depends on the range of patients.

Regression for method comparison

• Ordinary least squares assumes the x-axis method has no measurement error. Cornbleet and Gochman (1979) showed that its slope is incorrect when the x values are imprecise, with a significant error once the SD of measuring a single x value exceeds 0.2 times the SD of the x data. Of the methods they compared, Deming's was the most useful.
• Deming regression allows measurement error in both methods. It needs the ratio of the two error variances. The mcr package in R uses a ratio of 1 unless you set another.
• Passing-Bablok regression (1983) makes no special assumptions about the distribution of the samples or the measurement errors, and its result does not depend on which method is put on the x-axis.
• Reading the result: a slope of 1 with an intercept of 0 means the methods give the same values. A slope different from 1 points to a proportional difference, an intercept different from 0 to a constant difference. Judge both with their confidence intervals.

Worked example (10 patients, meter on the y-axis): least squares slope 0.97, intercept 0.91. Deming (error ratio 1) slope 0.97, intercept 0.83. Passing-Bablok slope 0.96, intercept 0.97. With 10 patients the confidence intervals would be wide. Bland and Altman (1986) regarded regression, like correlation, as inappropriate for judging agreement, so report the difference plot as well.

Comparison

Pearson r, the ICC and the limits of agreement: what each one tells you

The formula

Worked example: a bedside haemoglobin meter against the laboratory

Made-up teaching data. Haemoglobin (g/dL) in 10 patients, measured by the laboratory analyser and by a bedside meter. A difference of ±0.5 g/dL was set in advance as acceptable. Every number was computed in Python (NumPy, SciPy 1.15 and pingouin 0.6).

Step 1. Bias and limits of agreement

1. Differences (meter − laboratory): 0.8, 0.3, 0.6, 0.9, 0.2, 0.7, 0.4, 0.8, 0.1, 0.5. Their sum is 5.3.
2. Bias = 5.3 ÷ 10 = 0.53 g/dL. SD of the differences = 0.275 g/dL.
3. Limits of agreement = 0.53 ± 1.96 × 0.275 = 0.53 ± 0.54, that is −0.01 to 1.07 g/dL.
4. Paired t-test on the differences: t = 6.09, df 9, p < 0.001. The meter reads higher on average.
5. Regression of the differences on the means: slope −0.03, p = 0.52. No proportional bias is visible.
6. The upper limit (1.07) is outside ±0.5 g/dL. The meter cannot replace the laboratory for decisions that depend on 0.5 g/dL.

limits of agreement = bias ± 1.96 × SD_differences = 0.53 ± 1.96 × 0.275 = −0.01 to +1.07 g/dL

Step 2. The ANOVA table (10 patients × 2 methods)

Step 3. The ICCs (n = 10, k = 2)

ICC(A,1) = (10.547 − 0.0378) ÷ [10.547 + 0.0378 + 2 × (1.4045 − 0.0378) ÷ 10] = 10.509 ÷ 10.858 = 0.968

ICC(C,1) = (10.547 − 0.0378) ÷ (10.547 + 0.0378) = 0.993 · ICC(1,1) = (10.547 − 0.1745) ÷ (10.547 + 0.1745) = 0.967

Every coefficient is above 0.95, and the meter still fails the clinical requirement. This is why a method comparison is judged on the limits of agreement in units, with the ICC as a secondary summary.

In a real paper

ICC and limits of agreement in published studies

How to run it

Running an ICC and a Bland-Altman analysis

Enter the data with one row per patient and one column per rater or method. Menu paths and function arguments below were checked against the current IBM SPSS documentation, the CRAN reference manuals and the jamovi module pages.

SPSS

1. Analyze > Scale > Reliability Analysis... Move the rater or method columns into Items.
2. Click Statistics and tick Intraclass correlation coefficient.
3. Choose the Model: Two-Way Mixed, Two-Way Random or One-Way Random. Choose the Type: Consistency or Absolute Agreement. The confidence interval defaults to 95% and the test value to 0.
4. Set the model and type yourself. An IBM technical article written for SPSS release 8.0 (Nichols, 1998) states that without further choices SPSS uses the two-way mixed model and, for two-way models, consistency.
5. The output gives an ICC for a single rating and one for the mean of the k ratings. Report the one that matches how the measurement will be used.
6. For a Bland-Altman plot, compute the difference and the mean of each pair as new variables, plot the difference against the mean, and add the bias and the limits of agreement as reference lines.

R

# ratings: data frame, one row per patient, one column per rater or method
library(irr)
icc(ratings, model = "twoway", type = "agreement", unit = "single")
# irr defaults are model = "oneway", type = "consistency", unit = "single": set all three

library(psych)
ICC(ratings)                       # the six Shrout and Fleiss forms with confidence limits

library(blandr)
blandr.statistics(lab, meter)      # bias and limits of agreement with confidence intervals
blandr.draw(lab, meter)            # Bland-Altman plot

library(DescTools)
CCC(lab, meter, ci = "z-transform", conf.level = 0.95)   # Lin's concordance correlation coefficient

library(mcr)
mcreg(lab, meter, method.reg = "Deming")   # Deming regression, error.ratio = 1 by default
mcreg(lab, meter, method.reg = "PaBa")     # Passing-Bablok regression (the default method.reg)

jamovi

• Bland-Altman analysis: install the blandr module (Bland-Altman method comparison) from the jamovi library.
• ICC and concordance: the seolmatrix module, whose analyses include intraclass correlation and concordance correlation. The SimplyAgree module offers agreement and reliability analyses for nested designs.

How to report it

What to write in the thesis or paper

Report enough for a reader to repeat the analysis and to judge the result against their own clinical limit.

1. The acceptable difference and where it came from, stated before the results.
2. The number of patients, raters and replicate measurements, and how patients were sampled (their range matters for the ICC).
3. Bias with its 95% CI, and the 95% limits of agreement (ideally with their CIs), in units.
4. The Bland-Altman plot, and whether a trend, a transformation or repeated measurements were handled.
5. For any ICC: the software, the model (one-way, two-way random, two-way mixed), the type (absolute agreement or consistency), single or average measures, and the 95% CI.
6. The clinical conclusion: are the limits inside the acceptable difference?

Worked sentence, using the teaching example:

“Haemoglobin was measured with the bedside meter and the laboratory analyser in 10 patients. A difference of ±0.5 g/dL was defined as acceptable before data collection. The meter read higher than the laboratory by a mean of 0.53 g/dL (SD 0.28), with 95% limits of agreement from −0.01 to 1.07 g/dL. There was no trend in the differences across the range of haemoglobin. The upper limit exceeded the acceptable difference. The ICC (two-way random effects, absolute agreement, single measurement; pingouin 0.6) was 0.97 (95% CI 0.17 to 0.99).”

Reporting items for reliability and agreement studies are listed in the Guidelines for Reporting Reliability and Agreement Studies (GRRAS) in the guidelines section.

By specialty

Agreement studies by specialty

Common myths

Common mistaken beliefs

• ✗Myth: r = 0.95 shows that the new method can replace the old one.

  ✓Correlation measures a linear relation, not agreement. A method that always reads 1 g/dL high has r = 1 (Bland and Altman, 1986).

• ✗Myth: A high ICC means the methods are interchangeable.

  ✓The ICC rises with the range of patients. In an emergency department study the glucose ICC was 0.967 while the limits of agreement ran from −67.69 to +80.07 mg/dL.

• ✗Myth: No significant difference on a paired t-test means the methods agree.

  ✓The t-test looks only at the mean difference. Large differences in both directions can cancel out, giving a small bias and wide limits of agreement.

• ✗Myth: The limits of agreement show whether agreement is acceptable.

  ✓They show how large the differences are. Whether that is acceptable is a clinical decision, ideally fixed before the study.

• ✗Myth: The ICC in the SPSS output is the ICC.

  ✓There are 10 forms, and they give different values from the same data. Report the model, type and single or average measures.

• ✗Myth: An ICC above 0.75 is good in any study.

  ✓Koo and Li base their bands on the 95% CI, and a low or high ICC also reflects the spread of the subjects, the number of subjects and the number of raters.

Why this matters to you

Five consequences of measuring agreement the wrong way

1. Protocol sent back

   A method comparison protocol without an acceptable difference and without a planned Bland-Altman analysis gives the committee no way to judge success.

2. A wrong thesis conclusion

   Reporting r or a high ICC as agreement can lead you to recommend a device that differs from the reference by clinically important amounts.

3. Questions you cannot answer at the viva

   "Which ICC did you use, and why?" needs the model, the type and single or average measures. "The default in SPSS" is not an answer.

4. Misreading papers

   An ICC reported without its form, its confidence interval or the range of patients cannot be compared with another study.

5. Harm to patients

   A bedside value that is 1 g/dL too high can delay a transfusion. Limits of agreement in units show whether such differences occur.

History

History of agreement statistics for measurements

1. 1925

   Ronald A. Fisher

   Statistical Methods for Research Workers included a chapter on intraclass correlations and the analysis of variance.

2. 1966

   J. J. Bartko

   Published "The intraclass correlation coefficient as a measure of reliability" in Psychological Reports.

3. 1979

   Patrick Shrout and Joseph Fleiss

   Gave guidelines for choosing among six forms of the ICC for reliability studies in which n targets are rated by k judges.

4. 1979

   P. J. Cornbleet and N. Gochman

   Showed that least-squares slopes are incorrect when the x-axis method is measured with imprecision, and found Deming's method the most useful of those compared.

5. 1983

   Passing and Bablok

   Proposed a linear regression procedure for method comparison in clinical chemistry with no special assumptions about the distribution of samples and measurement errors.

6. 1986

   J. Martin Bland and Douglas Altman

   Wrote in the Lancet that correlation is misleading for method comparison, and described the plot of differences against means and the limits of agreement.

7. 1989

   Lawrence Lin

   Proposed the concordance correlation coefficient: the correlation between two readings that fall on the 45-degree line through the origin.

8. 1996

   Kenneth McGraw and S. P. Wong

   Described confidence intervals and tests for ICCs from one-way and two-way random and mixed-effect ANOVA models.

9. 1999

   Bland and Altman

   Extended the limits of agreement to differences that change with the size of the measurement (a regression approach) and to repeated measurements.

10. 2011

    Jan Kottner and colleagues

    Proposed the Guidelines for Reporting Reliability and Agreement Studies (GRRAS): 15 issues to address.

11. 2016

    Terry Koo and Mae Li

    Published a guideline for selecting and reporting ICCs, with interpretation bands based on the 95% confidence interval.

Guidelines

Reporting agreement and reliability

Reading a paper

Five checks on any agreement study

• Was the acceptable difference stated before the results, with a clinical reason?
• Are bias and limits of agreement given in units, with a Bland-Altman plot, and not only r or an ICC?
• Is the ICC form stated (model, type, single or average) with a 95% confidence interval?
• Did the patients cover the range seen in practice, and were several readings per patient handled correctly?
• Were the raters or methods independent of each other (blinded to the other reading)?

Why is it called that?

Word surgery

Exam answers

Short-note outlines

Further reading

Related Stateazy articles