When predictors overlap, which one matters?

Start here

What is multicollinearity?

Multicollinearity means that one predictor can be predicted well from the other predictors in the model. The data then carry little information about the effect of that predictor with the others held constant.

In the multiple regression chapter, the age coefficient was estimated from the part of age that duration of diabetes did not explain. When a predictor is almost fully explained by the others, that part is small and its coefficient is imprecise.

• Collinear predictors lie close to one straight line when plotted against each other. Multicollinearity covers a predictor that is close to a combination of several others. The term was first introduced by Ragnar Frisch in 1934.
• Perfect collinearity: one predictor is an exact linear combination of others, such as systolic BP, diastolic BP and pulse pressure. The coefficients cannot be estimated. SPSS will not enter a variable whose tolerance is below its default of 0.0001.
• It is a property of the predictors in your sample, not of the outcome. A study that recruits patients with a wide range of both predictors has less of it.

Consequences

What does multicollinearity do to the results?

It makes the coefficients of the overlapping predictors unstable and imprecise. The model as a whole, and its predictions, are affected much less.

• Inflated standard errors: the SE of the BMI coefficient is 0.36 with age and BMI in the model, and 1.85 with weight and waist added, 5.2 times larger.
• Wide confidence intervals: BMI 2.11 (95% CI 1.39 to 2.83) becomes 2.07 (−1.66 to 5.81).
• Coefficients that change sign: weight alone +0.54 mmHg per kg (0.24 to 0.84). With age and BMI added, −0.35 (−0.86 to 0.16).
• A significant model with no significant predictor: weight, BMI and waist together F(3, 45) = 12.33, p < 0.001, while their individual p values are 0.24, 0.27 and 0.54.
• Coefficients that change when a patient or a predictor is added or removed.
• Fit and predictions barely change: adjusted R² is 0.532 with all four predictors and 0.529 with age and BMI alone.

Kim (2019) summarises the mechanism: the variance of a coefficient is proportional to its VIF. “The larger σh2 produces unreliable probability values and confidence intervals of the regression coefficients.”

Interactive

Raise the correlation between two predictors and watch the standard errors and sign flips grow

Detection

How do you detect multicollinearity?

Start with the correlations between predictors, then calculate a variance inflation factor (VIF) for every predictor. Pairwise correlations alone can miss it.

VIF_j = 1 ÷ (1 − R²_j)    tolerance_j = 1 − R²_j = 1 ÷ VIF_j

R²ⱼ

R² from regressing predictor j on all the other predictors (the outcome is not used)

VIF

how many times larger the variance of coefficient j is than it would be if predictor j were unrelated to the others

√VIF

how many times larger the standard error is: VIF 4 doubles it, VIF 9 triples it

1. Correlation matrix: look for pairs above about 0.8. It misses a predictor explained by several others together. In the teaching data BMI correlates only 0.06 with height, but with all five predictors (age, weight, BMI, waist, height) in the model its VIF is 234.
2. VIF and tolerance: full teaching model: age 1.02, weight 5.31, BMI 26.80, waist 18.34 (tolerance for BMI 0.037). Kim (2019): “Multicollinearity is present when the VIF is higher than 5 to 10”. A VIF above 10 has been attributed to Marquardt (1970) as the level not to exceed.
3. Condition index and condition number: the square root of the largest eigenvalue of the predictors' correlation matrix divided by each eigenvalue. The largest is the condition number. Kim (2019) gives 10 to 30 as the threshold. Teaching model: 11.1. SPSS reports condition indices from the eigenvalues of the scaled and uncentered cross-products matrix, a different calculation that gives different values, so check which version your software reports.
4. Variance decomposition proportions: they identify which predictors overlap. Kim (2019): when two or more proportions above 0.8 to 0.9 belong to the same condition index above 10 to 30, those predictors are multicollinear.
5. Warning signs in the output: large SEs, a coefficient with the opposite sign to its unadjusted value or to clinical knowledge, and coefficients that change a lot when one predictor is added or removed.

Interactive

Choose the predictors and watch each VIF, standard error and sign

What to do

When predictors overlap, which one matters?

The data cannot answer that question. The answer has to come from the study question and clinical knowledge, and it should be decided before you look at the p values.

1. Check whether it matters for your question. Collinearity inflates only the SEs of the overlapping predictors. In the teaching data, age had VIF 1.02 and an SE of 0.157 with all three body-size measures in the model, against 0.155 with BMI alone. If the exposure you care about has a low VIF, overlap among adjustment variables does not harm its estimate.
2. Choose one measure when several measure the same thing. With age and BMI only: BMI 2.11 mmHg per kg/m² (1.39 to 2.83), every VIF 1.00, adjusted R² 0.529.
3. Combine them into one score decided in advance. The mean of the z-scores of weight, BMI and waist gave 9.63 mmHg per SD of the score (6.12 to 13.14), adjusted R² 0.505.
4. Centre continuous variables before forming product or squared terms (next section).
5. Keep a real confounder even if its VIF is high. Removing it lowers the VIF and biases the exposure estimate.
6. Collect more information: a larger sample, or patients with a wider range of both predictors, gives smaller SEs.
7. Penalised methods such as ridge regression (Hoerl and Kennard, 1970) accept a little bias in exchange for smaller variance. Use them with a statistician.
8. Do not drop whichever predictor has the largest p value. That choice is made by chance variation in the sample, and a different sample would drop a different one.

Centring

Why do interaction terms give huge VIFs, and what does centring change?

A product term is built from its components, so it correlates with them. Subtracting a central value from the continuous variable before multiplying removes most of that correlation.

• What changes: the main-effect coefficients. With raw BMI the sedentary coefficient is the difference at BMI 0 (−2.16, p = 0.050). Centred, it is the difference at BMI 27 (0.85, p < 0.001).
• What does not change: the interaction coefficient, its p value, R² and the predictions. Iacobucci and colleagues (2017): centring before forming the product term “can clarify the regression coefficients” while the overall model fit R² remains undisturbed.
• What centring cannot fix: two different predictors that measure the same thing. Subtracting a constant does not change the correlation between weight and BMI.

Confounding, mediation, collinearity

Is the overlapping predictor a confounder, a mediator, or just a similar measure?

The same correlation between two predictors can have three different meanings. Each needs a different decision, and the data alone cannot tell them apart.

Schisterman, Cole and Platt (2009) define overadjustment bias as “control for an intermediate variable (or a descending proxy for an intermediate variable) on a causal path from exposure to outcome.” A confounder is often correlated with the exposure, so a well-chosen confounder can raise the VIF of the exposure. The wider confidence interval that results is reported, not removed by dropping the confounder.

Variable selection

How should you choose which predictors go into the model?

Choose them before the analysis, from the study question and what is known about the causes of the outcome. Automatic selection by p value gives models that look better than they are.

Start from background knowledge

Heinze, Wallisch and Dunkler (2018): “Modeling should start with defendable assumptions on the roles of IVs that can be based on background knowledge”, from previous studies, expert knowledge or common sense. Write the list of predictors, with the reason for each, in the protocol.

What stepwise methods do

• Forward selection: the predictor with the smallest p value enters first, then the next, while each p value is below the entry level.
• Backward elimination: all predictors enter, then the weakest is removed one at a time while its p value is above the removal level.
• Stepwise: forward selection that also re-checks and removes predictors already entered.

Why stepwise selection misleads

• The p values are wrong. The IBM SPSS documentation states: “The significance values in your output are based on fitting a single model. Therefore, the significance values are generally invalid when a stepwise method (stepwise, forward, or backward) is used.”
• It selects noise. In the simulation below, 645 of 1,000 studies with no real predictor selected at least one.
• The model is unstable and its coefficients are biased. Heinze and colleagues: variable selection “can compromise stability of a final model, unbiasedness of regression coefficients, and validity of p-values or confidence intervals.”
• It overfits. Babyak (2004) lists automated variable selection, pretesting of candidate predictors and dichotomization of continuous variables as practices that “pose a considerable risk for spurious findings in models.”
• It chooses between collinear predictors by chance. Of two predictors that measure one thing, whichever has the slightly smaller p value in this sample stays.

If you must select: Heinze and colleagues note that most statisticians prefer backward elimination over forward selection, “especially when collinearity is present”. Keep predictors that are needed on clinical grounds whatever their p value, report the method and its thresholds, and label the result exploratory.

Interactive

Run forward selection on pure noise and watch which predictors it picks

Overfitting

How many predictors can your data support?

Every predictor term uses up information. With too many terms for the number of patients (or events), the model describes chance features of this sample and predicts new patients badly.

• Count parameters, not variables. A factor with five categories uses four terms, an interaction one more, and a squared term one more. Riley and colleagues (2019) state sample size in “predictor parameters”.
• Linear regression: Austin and Steyerberg (2015) found that about two subjects per variable gave accurate coefficients, standard errors and confidence intervals, but that a much higher number was needed to estimate R² without bias. Adjusted R² behaved well.
• Logistic regression: Peduzzi and colleagues (1996) found biased coefficients with fewer than 10 events per variable. Vittinghoff and McCulloch (2007) found situations in which fewer were acceptable, and van Smeden and colleagues (2016) concluded: “The current evidence supporting EPV rules for binary logistic regression is weak.”
• Prediction models: Riley and colleagues (2019) replaced rules of thumb with criteria: a global shrinkage factor of 0.9 or more, a difference of 0.05 or less between apparent and adjusted R², and precise estimates of the residual SD and the mean. In their example, 25 predictor parameters needed at least 918 subjects, 36.7 per parameter.
• Parsimony: use the fewest terms that answer the question. A smaller prespecified model is easier to interpret and more likely to hold in new patients.

Model form

How do you choose the form of the regression?

Decide the form from the type of outcome, clinical knowledge and the residual plots, and write it in the protocol.

1. Outcome type decides the family: linear for a measurement, logistic for yes/no, Cox for time to an event, Poisson for counts.
2. Each continuous predictor: a straight line if the residuals show no curve. Otherwise a transformation (such as the logarithm), a squared term, or a spline.
3. Keep continuous predictors continuous. Babyak (2004) includes dichotomization of continuous variables among the practices that risk spurious findings.
4. Interactions only where there is a prior reason, planned in advance, with continuous components centred.
5. Do not try many forms and keep the one with the smallest p value or the highest R². That is selection on the same data, with the same problems as stepwise methods.

Residual plots are explained, with an interactive element, in the multiple regression chapter.

Decide

A predictor has a high VIF: what should you do?

Answer the questions in order. Each answer links to the section that explains it.

Comparison

Pairs that residents confuse

The formula

Worked example: VIF, tolerance and condition number

Teaching data, made up: systolic BP in 50 adults on age, weight, BMI and waist. Every number was computed with statsmodels in Python.

1. Regress BMI on the other predictors (age, weight, waist), ignoring SBP: R²_BMI = 0.963.
2. VIF = 1 ÷ (1 − 0.963) = 26.8 (26.80 from the unrounded R²). Tolerance = 1 ÷ 26.8 = 0.037.
3. √26.8 = 5.18: collinearity alone makes the SE of the BMI coefficient about 5 times larger. Observed: SE 1.85 in the full model against 0.36 with age and BMI only, a ratio of 5.16.
4. The other VIFs: age 1.02, weight 5.31, waist 18.34.
5. Eigenvalues of the correlation matrix of the four predictors: 2.804, 1.000, 0.173, 0.023. Condition number = √(2.804 ÷ 0.0226) = 11.1.
6. Body-size block: F(3, 45) = 12.33, p < 0.001, although no body-size coefficient has p < 0.05.

import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

X = sm.add_constant(d[["age", "weight", "bmi", "waist"]])
vif = [variance_inflation_factor(X.values, i) for i in range(1, X.shape[1])]
print(vif)    # 1.02, 5.31, 26.80, 18.34

In a real paper

Collinearity checks and variable selection in published studies

How to run it

Checking collinearity and selecting predictors in SPSS, R and jamovi

Ask for the collinearity statistics every time you fit a multiple regression, and compare nested models with an F test rather than with automatic selection.

IBM SPSS Statistics

1. Analyze > Regression > Linear..., with the dependent and independent variables as usual.
2. Statistics: tick Collinearity diagnostics. SPSS then shows variance inflation factors and tolerances for each variable, with the eigenvalues of the scaled and uncentered cross-products matrix, condition indices and variance-decomposition proportions.
3. To test a group of overlapping predictors, put them in a second block (Next) and tick R squared change.
4. Method: Enter fits the planned model. Stepwise, Remove, Backward and Forward are selection methods. IBM states that significance values are generally invalid after stepwise, forward or backward selection.
5. If a selection method is used, Options sets the Stepping Method Criteria: with Use Probability of F, a variable enters if its significance is below the Entry value and is removed if it is above the Removal value. The default tolerance is 0.0001.

R

fit <- lm(sbp ~ age + weight + bmi + waist, data = d)

car::vif(fit)                           # VIFs, and generalized VIFs for terms with more than 1 df (package car)
performance::check_collinearity(fit)    # VIF and tolerance with confidence intervals (package performance)

fit_bmi <- lm(sbp ~ age + bmi, data = d)
anova(fit_bmi, fit)                     # F test: do weight and waist add anything once BMI is in?

d$bmi_c <- d$bmi - 27                   # centre before forming a product term

step(fit, direction = "backward")       # selection by AIC: report the result as exploratory

jamovi

1. Analyses → Regression → Linear Regression, with the dependent variable and covariates as usual.
2. Assumption Checks: tick Collinearity statistics for the VIF and tolerance of each predictor.
3. Model Builder: put the overlapping predictors in a second block with “+ Add New Block”. jamovi then shows a Model Comparisons table for the blocks.

How to report it

What to write about collinearity and model building

Say how the predictors were chosen and why, how many parameters the model had for the sample size, how collinearity was checked and what was found, and what you did about it.

• Methods: the list of predictors and the reason for each, decided before analysis. How continuous predictors were modelled, and any planned interactions and centring. The collinearity checks (VIF, with the threshold). If any selection was done, the method and its entry and removal criteria.
• Results: the largest VIF, the predictors involved, and any change made. For selected models, the candidate list and the final model, labelled exploratory.

If a selection procedure was used, add: the candidate predictors, the method (for example backward elimination), the removal p value, which predictors were kept regardless, and that the p values of the final model do not allow for the selection.

By specialty

Overlapping predictors by specialty

Common myths

Common mistaken beliefs

• ✗Myth: A high correlation between two predictors ruins the whole model.

  ✓It inflates the standard errors of the overlapping predictors only. In the teaching data, age had VIF 1.02 and almost the same SE with or without the three body-size measures (0.157 and 0.155).

• ✗Myth: A correlation matrix is enough to detect multicollinearity.

  ✓A predictor can be almost fully explained by two others without a high correlation with either. BMI correlated 0.06 with height, yet with all five predictors in the model its VIF was 234.

• ✗Myth: A predictor that is not significant has no effect and can be removed.

  ✓With collinear predictors, a large p value can mean the data cannot separate them. BMI had p = 0.27 in the full model and p < 0.001 once weight and waist were removed.

• ✗Myth: Stepwise selection finds the important predictors.

  ✓In 1,000 simulated studies with no real predictor, forward selection at p < 0.05 selected at least one in 645. IBM states that significance values are generally invalid after stepwise selection.

• ✗Myth: Ten events per variable is a proven rule.

  ✓Peduzzi and colleagues (1996) found problems below 10 events per variable in logistic regression. Later studies relaxed the rule (Vittinghoff and McCulloch, 2007) and judged the evidence for it weak (van Smeden and colleagues, 2016).

• ✗Myth: Centring removes multicollinearity.

  ✓Centring lowers the correlation between a variable and its product or squared term. It does not change the interaction coefficient or R², and it cannot separate two different measures of the same thing.

Why this matters to you

Five consequences of ignoring overlapping predictors

1. A result you cannot explain at the viva

   A negative coefficient for weight that you report as "weight lowers blood pressure" when the data simply cannot separate weight from BMI.

2. A real effect reported as absent

   BMI called "not significant" (p = 0.27) because weight and waist were entered with it, although it was clearly associated with SBP on its own and with age.

3. A thesis model that fails in new patients

   Twenty candidate predictors and 40 patients, chosen by stepwise selection, give an impressive R² that disappears in the next sample.

4. A protocol that invites rejection

   "All significant variables in univariate analysis will be entered into a stepwise model" gives no reasons, no list and no plan for overlapping predictors.

5. Misreading papers

   Accepting a claim that a biomarker is "independent" when its VIF is above the authors’ own cut-off and its confidence interval crosses zero.

History

History of multicollinearity and model building

1. 1934

   Ragnar Frisch

   Published Statistical Confluence Analysis by Means of Complete Regression Systems (Oslo). The term multicollinearity was first introduced in this work.

2. 1970

   Arthur Hoerl and Robert Kennard

   Published “Ridge regression: biased estimation for nonorthogonal problems” in Technometrics.

3. 1970

   Donald Marquardt

   Published “Generalized inverses, ridge regression, biased linear estimation, and nonlinear estimation”, the paper cited for the variance inflation factor and for the suggestion that a VIF should not exceed 10.

4. 1996

   Peter Peduzzi and colleagues

   A simulation study found biased logistic regression coefficients with fewer than 10 events per predictor variable.

5. 2004

   Michael Babyak

   Explained overfitting for clinicians, and showed that automated variable selection, pretesting of predictors and dichotomization of continuous variables risk spurious findings.

6. 2007

   Eric Vittinghoff and Charles McCulloch

   Found situations in which fewer than 10 events per variable gave acceptable results, and concluded that the rule can be relaxed.

7. 2016

   Maarten van Smeden and colleagues

   Concluded that the evidence supporting events-per-variable rules for binary logistic regression is weak.

8. 2016

   Dawn Iacobucci and colleagues

   Showed that mean centring reduces the correlation between a product term and its components without changing the overall model fit.

9. 2018

   Georg Heinze, Christine Wallisch and Daniela Dunkler

   Reviewed variable selection methods and recommended starting from background knowledge, with stability checks when selection is used.

10. 2019

    Jong Hae Kim

    Summarised diagnosis of multicollinearity for clinicians: VIF above 5 to 10, condition indices above 10 to 30, and variance decomposition proportions.

Guidelines

Collinearity and model building in guidelines

Reading a paper

Five checks on collinearity and model building in a paper

• Were the predictors chosen before the analysis for stated reasons, or by stepwise or univariate screening?
• Are VIFs (or another collinearity check) reported, with the threshold used, and do the authors act on values above it?
• Does any coefficient have the opposite sign to its unadjusted value, or a much wider CI than expected?
• Is the number of patients, or events, large for the number of predictor parameters, and was the model validated?
• Was a variable on the causal path adjusted for as if it were a confounder?

Why is it called that?

Word surgery

Exam answers

Short-note outlines

Further reading

Related Stateazy articles