QTA 9: Regression Diagnostics

The opposite mistake is milder but not free. An extraneous variable, also called superfluous or irrelevant, sits in the fitted equation without belonging in the true process. Its population coefficient is zero and OLS estimates it consistently, so in a large sample the reported coefficient drifts toward zero and the genuine drivers stay unbiased. Precision is what suffers.

The adjusted R-squared makes the cost visible, because it charges the model for every variable it uses.

Adding a variable raises k, which raises g and pushes the adjusted measure down. A larger model nearly always produces a smaller residual sum of squares, and that improvement can offset the penalty. When the true coefficient is zero the offset never arrives: the residual sum of squares stays where it was while g keeps growing, so the adjusted R-squared falls and the standard error widens.

Why correlated regressors make it worse

The loss of precision is not fixed in size. It grows with the correlation between the extraneous variable and the variables that matter, because a correlated intruder competes for the same variation and the estimator can no longer attribute it cleanly. Correlations among financial series are large, so one unnecessary column can push a comfortably significant parameter below the usual thresholds.

Set the two errors side by side. Omitting a relevant variable moves the surviving coefficients away from their population values, and the damage does not shrink as the sample grows. Including an irrelevant one leaves them centred correctly but scatters them more widely.

Because the two errors bite in different places, deciding how many variables to carry is a trade-off. Bias measures how far an estimator sits from its target on average. Variance measures how widely it scatters around its own mean. Being incomparable directly, they are weighed through mean squared error.

Large specifications that sweep in every plausible driver are unlikely to omit anything, so their coefficients are close to unbiased. They pay in variance, since attributing an effect to one variable is hard when correlated variables compete for the same movement. Small specifications reverse the position: tighter estimates, but any driver left outside pulls them off target.

General-to-specific model selection

The first procedure starts wide and prunes. It begins with a large specification holding every variable theory or intuition suggests could matter. If any coefficient misses significance at the chosen test size, the variable with the smallest absolute t-statistic is removed as empirically irrelevant, the model is re-estimated, and the two steps repeat until every survivor clears the bar. Test size matters more here than in ordinary testing, because the test runs many times over. Common choices sit between 1% and 0.1%, keeping only variables whose absolute t-statistics reach 2.57 or 3.29.

Cross-validation with m folds

The second procedure judges candidates by how well they predict data they were not fitted on, and it needs a candidate set first. When the pool is small, every combination can be examined: ten candidate variables give 2¹⁰ = 1,024 specifications, and the count doubles with each further variable, so exhaustive search stops being feasible quickly.

Cross-validation splits the sample at random into m blocks of equal size, with five and ten the usual choices for m. Parameters are estimated on m – 1 blocks and residuals computed by applying them to the block held out, which is why they are called out-of-sample residuals. Rotating that block through all m positions gives every observation one turn outside the fitting sample. The m – 1 fitting blocks are the training set, the held-out block the validation set.

Squaring those residuals and adding them across all m rounds gives one number per candidate, and the smallest total wins. That embeds the bias-variance trade-off in the rule: a specification missing a driver predicts badly because its coefficients are wrong, one stuffed with irrelevant variables because they are noisy.

Source: original illustration.

Model B wins: Model A is too small and carries bias into its predictions, while Model C adds more estimation noise than it removes.

Once a specification has been chosen, the assumptions used to justify the estimator need checking: whether the functional form is adequate, whether the parameters stay constant, and whether the errors have constant variance. That last property is homoskedasticity, one of the five assumptions behind the asymptotic distribution of the OLS estimator, and it requires the conditional variance of the shock to be the same whatever the regressors do.

When it fails, the errors are heteroskedastic. In models built on financial data the assumption is often false, and one common pattern has the variance of the shock widening as a regressor moves away from its mean, so observations near either end of the range carry noisier shocks than those in the middle.

What heteroskedasticity does and does not break

The estimator itself survives. Heteroskedasticity leaves the OLS coefficients unbiased and consistent, so they still point at the right population values. What changes is their asymptotic distribution, which makes the usual standard error formula wrong and every t-statistic and F-statistic built on it unreliable. Observations furthest from the mean of the regressor carry the most information about the slope, and when those come with the noisiest shocks the slope is harder to pin down. Rejecting homoskedasticity calls for a repair to the standard errors, not to the coefficients.

Heteroskedasticity is a property of quantities nobody observes, so it is diagnosed through the residuals. White proposed a two-stage procedure. The first stage fits the model by OLS and saves the residuals; the second asks whether the size of those residuals can be predicted from the explanatory variables, which is what constant variance rules out.

The second stage regresses the squared residuals on a constant, each explanatory variable, the square of each, and every cross-product between different variables. With two regressors in the first stage, five terms sit beyond the intercept.

The null sets every slope there to zero, leaving the squared residuals unexplained by anything on the right-hand side. The test statistic multiplies sample size by the R-squared from that regression, and under the null it follows a chi-squared distribution with one degree of freedom per restriction.

Large values point away from the null, so homoskedasticity is rejected and the standard errors need repairing. The Breusch-Pagan test is the same idea with the cross-products left out, often preferred when the first-stage model carries many variables.

Run on industry portfolios with a market model as the first stage, the test found the coefficient on the market return mostly insignificant, so the level of the market does not move the shock variance much. The coefficient on the squared market return was significant for most portfolios, and several series rejected homoskedasticity.

Three routes are open once the data are known to be heteroskedastic, differing in how much precision they recover.

Repair the standard errors and leave the estimator alone

The simplest route keeps the OLS coefficients as they are and swaps the covariance estimator. White’s estimator, which some software packages label Eiker-White, modifies the usual formula to allow for the shape of the heteroskedasticity, and the standard errors it produces are called heteroskedasticity-robust. Parameters are then tested with ordinary t-tests using the robust standard error. F-tests are harder to adjust, so testing several restrictions at once calls for care, though software normally applies the right modification automatically.

Robust standard errors are almost always larger than conventional ones, making t-statistics smaller and p-values larger. The change is not cosmetic. In one industry portfolio, a joint test that the size and value coefficients are both zero was rejected by the conventional F-test but returned a p-value of 20% under the robust version. Ignoring heteroskedasticity does not make estimates look worse than they are; it makes them look better.

Transform the data

The second route changes the variable being modelled. Heteroskedasticity turns up often in strictly positive series such as trading volume or firm revenues, where the spread scales with the level. Modelling the natural logarithm of the dependent variable can reduce or remove it, and often describes the data better. Dividing the dependent variable by another strictly positive variable does the same job, which is what turns dividends per share into a dividend yield, frequently homoskedastic even when dividends per share are not.

Weighted least squares

The third route is the most involved and the most powerful. Weighted least squares applies weights before estimating anything. Suppose the shock variance for observation i is proportional to the square of a known positive variable w_i. Dividing the equation through by w_i leaves an error with constant variance.

OLS is then applied to the transformed variables. That regression carries no explicit intercept, since the reciprocal of the weight takes the slot, but alpha is still read as the intercept of the original relationship. The weights are hardly ever known in advance, so the workaround runs three steps: fit by OLS, estimate the weights from the residuals, re-fit the transformed model. That is feasible weighted least squares, or feasible generalised least squares.

Explanatory variables in any real regression carry some correlation, and that alone is not a problem. Multicollinearity names the case where it has gone too far, so that one or more regressors can be largely explained by the others. With two variables the diagnosis is direct: regress either on the other, and a very high R-squared marks the pair as multicollinear.

Perfect collinearity is stronger and quite different: one variable is explained by the others exactly, so R-squared is 1 and the coefficients cannot be computed at all, the estimator being unable to separate variables moving in lockstep. Multicollinearity leaves the arithmetic working and only the answers imprecise. The rule of thumb is that an R-squared above 90% creates serious difficulties at sample sizes in the hundreds, and 90% corresponds to a correlation of 95%.

Nothing in the assumptions behind linear regression rules multicollinearity out, so it is no technical obstacle to estimation or testing. The difficulty is one of interpretation, and it is common in finance and risk management, because many candidate regressors respond to the same news.

Multicollinear data leave a distinctive pattern: coefficients that are jointly significant on the F-statistic while individual t-statistics stay small, often below 1.96 in magnitude. The group clearly explains the dependent variable, but no individual test can say which member deserves the credit.

Measuring it with the variance inflation factor

The standard diagnostic regresses each variable on all the others in turn and reads off the R-squared.

Values above 10 count as excessive, that threshold being where 90% of the variation in the variable is explained by its companions. Two responses are available: do nothing, since multicollinearity breaks no assumption, or drop the offending variables. Model selection procedures drop them anyway, because their coefficients are poorly estimated and removing them barely changes the fitted values.

Residual plots are the first line of specification checking, and they are quick. A well specified model leaves residuals unrelated to any included explanatory variable and modest in size, typically within plus or minus four times s, where s squared estimates the shock variance.

The basic version puts the residual on the vertical axis against an explanatory variable on the horizontal axis. Curvature points to a missing nonlinear term, a fan or bowtie shape to heteroskedasticity, isolated points to outliers. A variant plots the standardised residual, each residual divided by s, so a deviation reads directly in estimated standard deviations. Residuals against fitted values give one picture for a model with many regressors, and residuals in arrival order reveal drift. Their strength is coverage rather than precision: several kinds of defect show up at once, but none is quantified.

Outliers and Cook’s distance

An outlier here is defined by its consequences rather than its position: an observation whose removal would move the coefficients substantially. Cook’s distance turns that into a number by refitting without observation j and measuring how far the fitted values move.

An observation that distorts nothing is an inlier, with a small distance. A value of D_j above 1 is the usual signal that observation j counts as an outlier.

The chapter works through fifteen trials whose last observation looks suspicious. Fitting on all fifteen gives Y = 1.38 + 1.04X, and on the first fourteen Y = 1.15 + 0.69X, so both parameters move.

Source: chapter data containing a potential outlier, with fitted values from both regressions.

The same diagnostic on real return data shows how much one month can matter. In a three-factor regression of an industry portfolio excess return on the market, size and value factors, one observation returned a distance above 1, and dropping it moved the size factor coefficient from -0.30 to -0.18. Against a standard error of 0.054, that is 2.2 standard errors, while the other coefficients barely moved.

Detecting an outlier does not settle what to do about it. The first question is whether the data point is valid, since a recording error can be corrected. When it is genuine, and especially with return data, deleting it understates risk, because the move happened once and can happen again.

Given how many ways a regression can go wrong, it is worth stating what OLS delivers when nothing has. Under five assumptions it is the best linear unbiased estimator, or BLUE, and each word does work.

The estimator is linear because the intercept and the slope are both weighted sums of the observed dependent variable.

Weights in a linear estimator must not depend on the dependent variable, though they may depend on anything else, and here they come from the explanatory variable alone. Best means smallest variance within that class. Any other linear unbiased estimator can be written as the OLS estimator plus a term uncorrelated with it, so its variance is the sum of two variances.

That is minimised when the second variance is zero, which happens only when the competing estimator is OLS itself. This is the Gauss-Markov theorem.

The conditions and the caveats

The result rests on the assumptions established for the linear regression model: the relationship is linear in the parameters, the observations are well behaved draws from the population, the errors have conditional mean zero given the explanatory variables, no regressor is a perfect linear combination of the others, the errors are homoskedastic, and extreme observations are ruled out by finite fourth moments. Two caveats follow. Homoskedasticity carries more weight than the rest, and when the error variance changes with the regressors a weighted least squares estimator can beat OLS. The claim also says nothing about nonlinear estimators such as maximum likelihood, which achieve the smallest asymptotic variance among consistent estimators while being biased in finite samples.

What normal errors add

If the errors are independent, identically distributed and normal, the guarantee strengthens: OLS becomes the best unbiased estimator, smallest variance among all unbiased estimators, linear and nonlinear alike. The usual check is the Jarque-Bera statistic, which tests the two properties normal variables must have, zero skewness and kurtosis of 3.

Normality is not a precondition for using OLS. What the errors genuinely need is conditional mean zero and freedom from outliers. Without it the intercept and slope still estimate their population values accurately, keeping consistency and asymptotic normality. Normality buys a stronger optimality claim, not the right to run the regression.