Can you trust a questionnaire score?

Start here

What is the difference between an indicator, an index and a score?

Fatigue, quality of life and disability cannot be measured with one instrument reading. You measure them by asking several questions and combining the answers into a number.

Building a scoring system

1. Define the construct. Write in one or two sentences what the score should measure, in which patients, and for what purpose (diagnosis, severity, change over time).
2. Write the items. From the literature, patients and clinicians. Each item asks about one thing.
3. Choose the response format. Yes or no, or ordered categories such as 0 to 4.
4. Decide the weights. Equal weights (every item 0 to 4, then add) are the usual choice. Unequal weights need a reason, for example points taken from the coefficients of a regression model.
5. Add the items into a total, and decide how to handle missing answers before you collect the data.
6. Test the score: content validity, a pretest with patients, then reliability and validity in a new sample.

Does every score need Cronbach's alpha? No. COSMIN distinguishes two kinds of instrument. In a reflective model the construct shows itself in the items, so the items should correlate (fatigue makes a patient tired, slow and unable to concentrate). In a formative model the items are not supposed to be correlated (a checklist of risk factors that together form a risk score). Analyses of internal structure, including alpha, apply only to reflective scales.

Two types of Likert data

Is one Likert item a scale?

No. A Likert item is one statement with ordered answers. A Likert scale is the sum or mean of several such items that measure the same construct.

Opinion differs on single Likert items: some authors treat Likert data as interval, others as ordinal. Norman (2010) reviewed studies going back to the 1930s and concluded that parametric methods can be used with Likert scale data without giving wrong answers. When you analyse single items, say which approach you used.

Comparison

What is the difference between a reliable and a valid questionnaire?

Reliable means the questionnaire gives the same result when nothing has changed. Valid means it measures the construct it claims to measure. A scale can be reliable and still measure the wrong thing.

Reliability

In what ways can a questionnaire be reliable?

Three questions, three kinds of study. Do the items agree with each other? Does the patient get the same score next week? Do two raters give the same score?

Internal consistency

How does Cronbach's alpha work?

If all items measure fatigue, a patient who scores high on one item tends to score high on the others. Their scores then vary together, and the total varies more than the items would if they were unrelated.

α = k ÷ (k − 1) × [1 − (sum of the item variances) ÷ (variance of the total score)]

k

number of items

item variance

the variance of each item across patients

variance of the total

the variance of the patients’ total scores. When items correlate, this is much larger than the sum of the item variances

The same idea written with correlations gives the standardised alpha, which SPSS reports next to Cronbach's alpha. If the two differ, the items differ in spread or in how strongly they reflect the construct:

standardised α = k × r̄ ÷ [1 + (k − 1) × r̄]

r̄

the mean correlation between all pairs of items

Related coefficients

• Kuder-Richardson 20 (KR-20): for yes or no items, Cronbach's alpha is equivalent to KR-20.
• Split-half reliability: the items are divided into two halves and the two half-scores are correlated. Because each half is shorter, the Spearman-Brown formula estimates the reliability of the full-length test: r_full = 2r ÷ (1 + r). Alpha is the mean of all possible split-half reliabilities, corrected for test length.
• Corrected item-total correlation (item-rest correlation): the correlation of each item with the sum of the other items. Items below 0.30 are candidates for revision or deletion (Boateng 2018).
• Alpha if item deleted: the alpha of the scale without that item. If it is higher than the scale's alpha, the item adds less than it takes away.

Interactive

Add items, change their correlation and delete an unrelated item, and watch alpha change

A common misreading

Does a high alpha prove that the items measure one thing?

No. Internal consistency is a necessary but not sufficient condition for unidimensionality. A questionnaire that mixes two different topics can still have a high alpha, especially if it is long.

Tavakol and Dennick (2011) note that a multidimensional test does not necessarily have a lower alpha than a unidimensional one, and that alpha should be calculated for each concept, not for a whole test that covers several. The COSMIN Risk of Bias checklist (Mokkink 2018) states that alpha can be interpreted as a measure of internal consistency only when the scale or subscale is unidimensional, and COSMIN recommends checking structural validity with factor analysis first. Factor analysis is explained in the multivariate methods chapter.

Interactive

Mix two unrelated topics in one questionnaire and watch alpha stay high

Comparison

What does McDonald's omega add?

Alpha assumes that every item measures the construct equally strongly (the tau-equivalent model). Omega uses the factor loading of each item, so items may differ in how strongly they reflect the construct.

ω = (Σλ)² ÷ [(Σλ)² + Σθ]

λ (lambda)

factor loading of each item on the single factor (standardised)

θ (theta)

error variance of each item, 1 − λ² for standardised items

Teaching example. Five standardised items with loadings 0.8, 0.8, 0.7, 0.6 and 0.3: Σλ = 3.2 and Σθ = 2.78, so ω = 3.2² ÷ (3.2² + 2.78) = 10.24 ÷ 13.02 = 0.79. Alpha calculated from the correlations these loadings produce is 0.77. With five equal loadings of 0.7, alpha and omega are the same, 0.83.

McNeish (2018) and Dunn and colleagues (2014) review the problems with alpha and recommend omega and related coefficients. Neither coefficient replaces checking the dimensions with factor analysis.

Reliability over time and between raters

Does the score stay the same when the patient has not changed?

Give the questionnaire twice to patients whose condition should not change in between, and compare the two scores. This is test-retest reliability. The same comparison between two raters is inter-rater reliability.

The interval must be short enough that the condition does not change, since real change lowers test-retest reliability (Boateng 2018), and long enough that patients do not simply remember their previous answers. The KOOS study below used 1 to 2 weeks.

Teaching example: why a correlation is not enough

Made-up totals for 8 stable patients on day 0 (12, 8, 15, 5, 10, 14, 7, 11) and day 14 (13, 8, 14, 6, 11, 15, 6, 12). Pearson r = 0.97. The ICC for absolute agreement, two-way random effects, single measurement, is 0.96 (95% CI 0.85 to 0.99).

Now suppose every day-14 score is 3 points higher, because the questionnaire was read out differently. Pearson r stays at 0.97. The ICC for absolute agreement falls to 0.66 (95% CI −0.03 to 0.94), because the patients no longer get the same score. The ICC forms, and the Bland-Altman limits of agreement, are covered in the ICC and Bland-Altman chapter. For categories and grades, use kappa or weighted kappa (kappa chapter).

Reading an ICC: Koo and Li (2016) suggest, based on the 95% CI of the estimate, values below 0.5 as poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good and above 0.90 excellent. There are 10 forms of ICC, so the form must be stated.

Decide

Which reliability statistic do you need?

Answer the questions about your study.

Validity

Does the questionnaire measure what it claims to measure?

Validity is not shown by one statistic. It is built from several kinds of evidence, collected before and after the questionnaire is given to patients.

Face validity

The degree to which respondents or end users judge that the items are appropriate to the construct and the purpose of the assessment. It is assessed with the target population, for example in cognitive interviews, and COSMIN counts it as part of content validity.

Content validity and the content validity index (CVI)

Experts rate how relevant each item is to the construct, on a 4-point scale from 1 (not relevant) to 4 (highly relevant). A rating of 3 or 4 counts as relevant.

I-CVI = number of experts rating the item 3 or 4 ÷ number of experts

S-CVI/Ave

the mean of the I-CVIs of all items

S-CVI/UA

the proportion of items rated 3 or 4 by every expert (universal agreement)

Polit and Beck (2006) recommend, for excellent content validity, an I-CVI of 1.00 with 3 to 5 experts or at least 0.78 with 6 to 10 experts, and an S-CVI/Ave of 0.90 or higher. The two scale-level indices can give different values, so state which one you report.

Interactive

Change the experts' ratings and watch the item and scale content validity indices change

Validity

How do you show criterion and construct validity?

Criterion validity compares the questionnaire with a gold standard. Construct validity checks whether the scores relate to other measures, and differ between groups, in the ways you predicted before collecting the data.

Criterion validity

• Concurrent: the questionnaire and the gold standard are measured at the same time or shortly after, such as a depression questionnaire against a psychiatrist's interview. Report a correlation, or sensitivity, specificity and the AUC.
• Predictive: the score predicts a later outcome, such as a frailty score and readmission within a year.
• COSMIN rates criterion validity as sufficient when the correlation with the gold standard, or the AUC, is 0.70 or more.

Construct validity

• Structural validity: factor analysis shows whether the items form the dimensions (subscales) the authors intended.
• Convergent validity: scores correlate strongly with a measure of a similar construct. COSMIN's generic hypothesis is a correlation of 0.50 or more.
• Discriminant validity: scores correlate weakly with measures of unrelated constructs (COSMIN: below 0.30). Related but dissimilar constructs should correlate 0.30 to 0.50.
• Known-groups validity: scores differ between groups that should differ.
• COSMIN rates hypotheses testing as sufficient when at least 75% of the results agree with the hypotheses stated in advance.

Interactive

Lower the reliability of the scale and watch its correlation with the criterion fall

Measurement error in either measure lowers the correlation you observe. This is why a questionnaire with poor reliability cannot show strong criterion or convergent validity.

observed r = true r × √(reliability of the scale × reliability of the criterion)

Rearranged from Spearman's correction for attenuation, in which the disattenuated correlation equals the observed correlation divided by the square root of the product of the two reliabilities.

Cross-cultural adaptation

Can you translate a questionnaire and use it straight away?

No. A translation can change the meaning of an item, and an item that suits one culture may not suit another. The questionnaire is adapted in stages, and its measurement properties are tested again in the new population.

• Two independent forward translations are combined into one version, which is translated back into the original language without sight of the original, so that differences in meaning become visible.
• An expert committee reviews all versions and reports, and produces the pre-final version.
• The pre-final version is tested in 30 to 40 people from the target population, to check that the items are understood and acceptable.
• COSMIN treats the translation process itself as not a measurement property. The pilot test after translation counts as part of content validity (comprehensibility), and cross-cultural validity is assessed by testing whether items behave the same way in different groups (measurement invariance, differential item functioning).
• Then test internal consistency, reliability and validity in the new population, as for a new questionnaire.

The formula

Worked example: alpha, item statistics and the item to drop

Made-up teaching data: 10 patients answer the resident's five fatigue items, each scored 0 to 4.

1. Item variances: tired 1.82, rest 1.29, focus 2.23, effort 1.16, appetite 1.11. Sum = 7.61.
2. Totals: 8, 3, 12, 15, 10, 11, 8, 14, 18, 12. Variance of the totals = 17.66.
3. α = 5 ÷ 4 × (1 − 7.61 ÷ 17.66) = 1.25 × 0.569 = 0.71.
4. Standardised alpha, from the mean inter-item correlation of 0.31, is 0.69. It differs from 0.71 because the items differ in spread (focus 2.23, effort 1.16).

5. The appetite item correlates −0.08 with the rest of the scale, below 0.30, and alpha rises to 0.83 without it. Appetite is affected by dialysis for reasons other than fatigue, so the item probably belongs to a different construct.
6. Four items: sum of item variances 6.50, variance of the totals 17.21. α = 4 ÷ 3 × (1 − 6.50 ÷ 17.21) = 0.83, 95% CI 0.55 to 0.95.
7. The confidence interval is wide because only 10 patients answered. Deleting an item on data from 10 patients is a decision for a pilot, to be confirmed in the main sample.

α (5 items) = 0.71 → delete appetite → α (4 items) = 0.83

Computed in Python with pingouin 0.6.1 and NumPy.

In a real paper

Reliability and validity reported in two published questionnaire studies

Software

How to run it

Menu paths and function names below are taken from the current IBM SPSS, CRAN and jamovi documentation.

IBM SPSS Statistics

Internal consistency
Analyze > Scale > Reliability Analysis...
  Items: move the scale items (not the total)
  Model: Alpha   (Omega is also listed in the version 31 documentation)
  Statistics...: Descriptives for > Scale if item deleted
                 Inter-Item > Correlations

Test-retest or inter-rater reliability
Analyze > Scale > Reliability Analysis...
  Items: score at time 1, score at time 2 (or rater 1, rater 2)
  Statistics...: Intraclass correlation coefficient
                 Model: Two-Way Random   Type: Absolute Agreement

For yes or no items, the alpha that SPSS reports is equivalent to KR-20. “Scale if item deleted” gives the corrected item-total correlation and alpha if item deleted. The ICC options are Two-Way Mixed, Two-Way Random and One-Way Random, with Consistency or Absolute Agreement.

R

library(psych)
a <- alpha(items)          # items: data frame, one column per item
a$total$raw_alpha          # Cronbach's alpha (std.alpha = standardised)
a$alpha.drop               # alpha if each item is dropped
a$item.stats$r.drop        # correlation of each item with the other items

ICC(retest)                # retest: one column per occasion or rater
                           # ICC2 = two-way random, absolute agreement, single

library(jmv)               # the jamovi analyses in R
reliability(data, vars = c("tired", "rest", "focus", "effort"),
            omegaScale = TRUE, alphaItems = TRUE, omegaItems = TRUE,
            itemRestCor = TRUE)

jamovi

Analyses tab: Factor → Reliability Analysis. Move the items across. The options give Cronbach's α and McDonald's ω for the scale, and for each item the α and ω if the item is dropped and the item-rest correlation. Reverse-scored items can be listed so they are recoded before the analysis.

Reporting

How to report it

Report each measurement property with the statistic, its confidence interval, the sample and the form of the analysis.

1. How the items were developed or translated, with the guideline followed.
2. Content validity: the number of experts, the rating scale, the I-CVIs and which S-CVI you report.
3. Structural validity: the factor analysis and the number of factors retained.
4. Internal consistency: alpha (and omega) for each unidimensional scale or subscale, with the 95% CI, item-total correlations and any items removed, with the reason.
5. Reliability: the software and the ICC model, type and definition (Koo and Li 2016), the 95% CI, the interval between occasions and why patients were considered stable.
6. Validity: the hypotheses stated in advance and the proportion confirmed, or the correlation or AUC against the criterion.

A sentence you can adapt (teaching numbers):

The five draft items were answered by 10 patients on haemodialysis. The appetite item had a corrected item-total correlation of −0.08 and was removed, which raised Cronbach's α from 0.71 to 0.83 (95% CI 0.55 to 0.95) for the remaining four items. In 8 stable patients who completed the scale twice, 14 days apart, test-retest reliability was ICC 0.96 (95% CI 0.85 to 0.99), two-way random effects, absolute agreement, single measurement. Analyses were done in Python (pingouin 0.6.1).

Reporting guidance: GRRAS (Guidelines for Reporting Reliability and Agreement Studies, Kottner 2011) proposes 15 issues to address when reliability and agreement are reported. The COSMIN Risk of Bias checklist (Mokkink 2018) lists the design requirements and preferred statistical methods for each measurement property, and the COSMIN criteria (Prinsen 2018) define sufficient results.

By specialty

Questionnaire reliability and validity by specialty

Common myths

Common mistaken beliefs

• ✗Myth: An alpha of 0.95 means an excellent questionnaire.

  ✓An alpha above 0.90 may mean that items repeat the same question, and a maximum of 0.90 has been recommended (Tavakol and Dennick 2011).

• ✗Myth: A high alpha shows that the items measure one thing.

  ✓A multidimensional test does not necessarily have a lower alpha. Check the dimensions with factor analysis first, then report alpha for each subscale.

• ✗Myth: Alpha is a fixed property of the questionnaire.

  ✓Streiner (2003) lists this as a myth. Alpha depends on the sample, so report it for your own patients.

• ✗Myth: Alpha always lies between 0 and 1.

  ✓Streiner (2003) also lists this as a myth. Alpha can be negative, for example when a reverse-worded item has not been recoded.

• ✗Myth: A high correlation between test and retest proves reliability.

  ✓In the teaching example, adding 3 points to every retest score left Pearson r at 0.97 while the ICC for absolute agreement fell from 0.96 to 0.66.

• ✗Myth: A translated questionnaire keeps the validity of the original.

  ✓It needs cross-cultural adaptation, a pretest, and new testing of reliability and validity in the target population.

Why this matters to you

Five consequences of using a questionnaire without testing it

1. Protocol sent back

   A research committee will ask how a self-made or translated questionnaire was developed and tested before it can measure your primary outcome.

2. A thesis result nobody can interpret

   If the score is unreliable, real differences between groups are hidden by measurement error and correlations with other measures are lower than they should be.

3. Questions you cannot answer at the viva

   "The questionnaire was validated, alpha 0.71" does not answer "validated against what?".

4. Misreading papers

   A paper that reports only alpha has shown internal consistency, not validity, and a very high alpha may reflect repeated items.

5. Wrong decisions for patients

   A screening score that has not been compared with a diagnostic standard can label healthy people as ill, or miss people who need treatment.

History

History of reliability and validity measurement

1. 1904

   C. Spearman

   Published "The proof and measurement of association between two things", the source of the correction for attenuation: a correlation lowered by measurement error, corrected using the reliabilities of the two measures.

2. 1910

   Spearman and Brown

   In two papers in the same issue of the British Journal of Psychology, gave the Spearman-Brown formula, which shows how reliability increases as a test is lengthened.

3. 1932

   Rensis Likert

   Introduced the Likert-type scale in a monograph on the method of summated ratings for measuring attitudes.

4. 1937

   Kuder and Richardson

   Published "The theory of the estimation of test reliability" in Psychometrika. Their KR-20 coefficient is the form of alpha for yes or no items.

5. 1951

   Lee Cronbach

   Published "Coefficient alpha and the internal structure of tests" in Psychometrika, to measure the internal consistency of a test or scale.

6. 1955

   L. J. Cronbach and P. E. Meehl

   Published "Construct validity in psychological tests".

7. 1959

   D. T. Campbell and D. W. Fiske

   Described convergent and discriminant validation with the multitrait-multimethod matrix.

8. 1986

   M. R. Lynn

   Published "Determination and quantification of content validity". Polit and Beck (2006) recommend its criteria for item-level CVIs.

9. 2000

   Beaton, Bombardier, Guillemin and Ferraz

   Published guidelines for the cross-cultural adaptation of self-report measures.

10. 2010

    COSMIN group (Lidwine Mokkink and colleagues)

    Reached international consensus, in a Delphi study with 43 experts, on the taxonomy, terminology and definitions of measurement properties.

11. 2011

    Jan Kottner and colleagues

    Proposed the Guidelines for Reporting Reliability and Agreement Studies (GRRAS), 15 issues to address.

Guidelines

Guidance on questionnaire measurement properties

Reading a paper

Six checks on a questionnaire study

• Was the questionnaire developed or translated by a stated method, with expert review and a pretest in the target population?
• Was the factor structure checked before alpha was reported, and is alpha given for each subscale?
• Is an alpha above 0.90 explained, or could items be repeating each other?
• For test-retest or inter-rater reliability, is the ICC form stated, with a 95% CI and the interval between occasions?
• Were hypotheses for construct validity stated before the data were collected, and how many were confirmed?
• Do the numbers in the abstract, results and discussion agree, and does each reliability name match the statistic used?

Why is it called that?

Word surgery

Exam answers

Model answer outlines

Past papers

TNMGRMU questions on reliability and validity of scales

Questions from MD, MS, DM, MCh and Ph.D. papers that this chapter helps you answer. Tap a sitting to open the university's own question paper.

1. ×1Validation of questionnaireMD Community Medicine · Oct 2023