QTA 7: Linear Regression
Two random variables move together, and the question is how much of one moves with the other. Regression is the workhorse answer, the most widely used technique for measuring and testing relationships between random variables, and in finance it does three jobs constantly: it measures how sensitive a portfolio is to a common risk factor, it produces the hedge ratio that shrinks an unwanted exposure, and it separates a fund manager who adds value from one riding a benchmark. Every regression model links a dependent variable, written Y, to one or more explanatory variables, written X. This chapter works entirely with the version carrying a single explanatory variable, known as bivariate regression or simple regression. Its appeal is practical: the estimators have closed forms, the parameters mean something a practitioner can state in a sentence, and the specification bends to cover a wide range of shapes.
The model and its three pieces
Linear regression assumes a linear link between an explanatory variable and a dependent variable.
The slope, written beta and also called the regression coefficient, measures how sensitive Y is to a change in X. The intercept, written alpha, is a constant. The third piece, epsilon, goes by several names: the shock, the innovation, the error or the disturbance. It carries whatever part of Y the explanatory variable cannot account for, and its mean is assumed to be zero.
The shock is the reason statistical analysis is needed at all. Strip it out and every pair of observations would sit on one straight line, so two points and some algebra would recover both parameters exactly. The two variables are also not treated the same way, since causality is assumed to run from X to Y and not back, a sharp contrast with correlation, where both variables enter symmetrically.
| Explained variable | Explanatory variable | Shock |
|---|---|---|
| Left-hand-side variable | Right-hand-side variable | Innovation |
| Dependent variable | Independent variable | Noise |
| Regressand | Regressor | Error |
| Disturbance |
Source: nomenclature varies across finance, economics, engineering and the natural sciences.
Reading the intercept
The obvious reading of the intercept is the value Y takes when X equals zero. That reading is only worth anything when zero is a value the explanatory variable could plausibly reach. Suppose the average maturity of a firm’s corporate bond offerings is regressed on its market capitalisation. Market capitalisation is always positive, so calling the intercept the average maturity issued by a firm worth USD 0 describes nothing real, and a negative intercept would predict negative maturity, which cannot happen. Where zero is out of reach, treat the intercept as the constant that forces the fitted line through the point given by the sample mean of X and the sample mean of Y.
Explanatory variables may be continuous, discrete, or built out of other variables, so a product of two variables can serve as a third. What the method will not tolerate is a violation of three restrictions, and every specification has to be checked against all three.
The three restrictions
First, the relationship must be linear in the unknown coefficients rather than in the variables, so each term on the right-hand side is one unknown coefficient multiplied by one explanatory variable. Second, the error has to enter additively, which rules out specifications where the size of the error is scaled by the data, such as a model carrying a term in the product of the explanatory variable and the shock. Third, every explanatory variable has to be observable, so missing data blocks direct application, although values can be imputed first and the completed dataset then handed to the regression.
A specification that raises the explanatory variable to an unknown power fails the first restriction on both counts, since the term carries two unknown parameters and the exponent does not multiply anything.
More variables buy more shapes
Adding explanatory variables widens the range of relationships that can be captured, because the effect of any one is measured while the others are held fixed. Assessing a hedge fund manager commonly calls for between seven and 11 explanatory variables.
Nothing in that model requires the k explanatory variables to be independent of each other, and one may be a known nonlinear function of another. Setting the first explanatory variable equal to X and the second equal to X squared gives a model linear in the parameters yet curved in X, because a modest rise in X changes Y by an amount depending on the level of X as well as on the size of the increase. Polynomials approximate many smooth nonlinear functions well, so including powers covers a great deal of ground. What is not permitted is a variable that is a linear transformation of another already in the model, since the correlation between the two would then be exactly 1 or exactly -1.
Transformations rescue awkward specifications
Some models fail the three tests as written but pass once both sides are transformed. Take a positive dependent variable generated multiplicatively by a positive shock.
After the transformation the coefficient reports how a change in the transformed explanatory variable moves the transformed dependent variable, which is not the effect of X on Y. Two cases are worth memorising. When both variables have been replaced by their logarithms, the specification is called a log-log model and the slope is the elasticity of Y with respect to X, reading directly as the percentage change in Y produced by a 1% change in X. When only the dependent variable has been logged, the slope is the percentage change in Y produced by a one-unit change in X.
Qualitative information does not arrive as a number, and a dummy variable is how it gets into a regression. A dummy is almost always binary, taking the value 1 when an observation has some quality and 0 when it does not. Sector membership is the standard illustration: a transportation dummy is 1 for a commercial airline or a bus operator and 0 for firms whose main business lies elsewhere. Country of origin for a bond works the same way, and so do seasonal effects. Dummies are also built out of continuous variables, as with a market direction dummy recording 1 whenever the market return is negative.
Shifting the intercept and shifting the slope
Adding a dummy on its own moves the intercept up or down when the quality is present, and the variable is then called an intercept dummy.
Multiplying the dummy by an explanatory variable is what changes the steepness. An empirically relevant extension of CAPM lets the sensitivity of a firm’s return depend on which way the market moved. Define X as the excess market return and D as a dummy set to 1 when that excess return is negative, so the dummy marks a loss.
This is an ordinary linear regression, because both the excess market return and the product of that return with the dummy are observable, and each is attached to the dependent variable by a single coefficient. The second variable is called an interaction dummy variable. What it buys is two models written as one equation: when the dummy is 0 the slope is the first coefficient on its own, and when the dummy is 1 the slope becomes the sum of the two coefficients. Each branch is linear in X, which makes the fitted relationship piecewise linear, but as soon as the second coefficient differs from zero the overall relationship between Y and X is nonlinear. Because the dummy alters steepness here, it is known as a slope dummy variable. Nothing stops both types appearing in the same equation.
The same construction answers a common exam setup. Suppose CAPM holds on every day except those carrying a Federal Open Markets Committee announcement, and the beta differs on those days. Add a dummy equal to 1 on announcement days, interact it with the market return, and test whether the coefficient on that interaction is zero. If both the alpha and the beta are suspected of changing, use the intercept dummy and the slope dummy together.
A bivariate model carries three parameters: the intercept, the slope, and the variance of the shock. Suppose n paired observations are available and every pair obeys the same linear relationship. Ordinary least squares picks the intercept and slope making the fitted line sit as close to the cloud of points as possible, where closeness is the sum of the squared vertical gaps between the observed values and the line. Squaring stops a gap above the line cancelling one below it.
The quantity being made as small as possible is the residual sum of squares. Solving the problem is a calculus exercise: differentiate with respect to each parameter in turn, set both derivatives to zero, and rearrange. The answer is a pair of closed-form expressions needing nothing more than sample means.
The slope estimator only makes sense when the denominator is strictly positive, which requires the explanatory variable to vary around its own mean. If every observation of X took the same value, the points would stack on one vertical line and no finite slope would fit them. Violations are easy to spot in a dataset.
Three ways of writing the same slope
Multiplying top and bottom of the slope formula by one over n turns both sums into sample moments, so the estimator becomes the ratio of the sample covariance between the two variables to the sample variance of the explanatory variable.
The second form is worth holding on to. The sample correlation alone fixes the sign of the slope, and it contributes to the size as well, while the two sample standard deviations scale that correlation into the units the regression is measured in. One consequence gets tested often: the sample correlation equals zero if and only if the estimated slope equals zero, so linear regression is a convenient formal test of whether two variables are correlated at all.
The two estimates generate a fitted value for every observation, and the leftover part of each observed value is its residual.
Two properties come free with the minimisation and are not assumptions about the world: the residuals always sum to zero, and their sample correlation with the observed explanatory variable is always exactly zero. Model residuals stay distinct from the population disturbances they estimate, which are never observed.
Reading the two estimates out loud
The slope estimate is a rate: an increase of one unit in the explanatory variable is associated on average with a change of that many units in the dependent variable, so where both variables are returns a slope above 1 says the dependent series amplifies the explanatory series. The intercept estimate is a level: the average value of the dependent variable when the explanatory variable equals zero, subject to the caveat that zero has to be reachable.
Eight quarters of excess returns are recorded for a sector fund and for the equity market, both in percent, with the market as the explanatory variable.
| Quarter | Market x | Fund y | Fitted value | Residual |
|---|---|---|---|---|
| 1 | -6 | -5.75 | -6.75 | 1.00 |
| 2 | -4 | -5.75 | -4.25 | -1.50 |
| 3 | -2 | -1.25 | -1.75 | 0.50 |
| 4 | 0 | -0.25 | 0.75 | -1.00 |
| 5 | 2 | 4.25 | 3.25 | 1.00 |
| 6 | 4 | 5.25 | 5.75 | -0.50 |
| 7 | 6 | 9.75 | 8.25 | 1.50 |
| 8 | 8 | 9.75 | 10.75 | -1.00 |
Deriving the estimators needs only the one easy condition already met, that the explanatory variable has positive variance. Interpreting what comes out needs five further assumptions, and the first is the most important and the hardest to check.
The requirement is that the explanatory variable carries no information about where the shock sits. Equivalently, the expected product of the shock with any well defined function of X is zero, which covers both the function returning X itself and the function returning a constant. Two consequences follow: the correlation between the shock and the explanatory variable is zero, so the innovations are uncorrelated with the regressor, and the unconditional mean of the shock is zero.
No direct test exists, because the estimated residuals carry that same zero correlation by construction, so looking at them settles nothing. Judging the assumption means thinking carefully about the process that generated the data, and four familiar processes break it.
Sample selection bias and survivorship bias
Selection bias appears when observations go unrecorded because of the value the dependent variable took. Model changes in house prices from completed transactions and the sample is already distorted, because owners sitting on negative equity tend not to sell, which strips out the properties whose prices fell furthest. Studying firm performance raises the same problem: the firms still trading performed well enough to keep trading, weaker firms delist, and a model fitted to survivors describes survivors rather than firms in general. The usual remedy is a database that reports a final return for every firm, delisted ones included.
Simultaneity bias
When both variables are settled at the same moment by the same forces, modelling one as a function of the other is not meaningful, because the supposed cause is itself responding. Quantity transacted and transaction price is the classic pair, since moving the price changes the quantity traded and the quantity traded feeds back into the price.
Omitted variables
Leaving out a variable that helps determine the dependent variable biases the coefficients on the variables kept, and it can manufacture the appearance of a relationship where none exists.
Attenuation bias
Explanatory variables are treated as non-stochastic, and measurement error breaks that. When the variable used is the true variable plus independent noise, the estimated slope is smaller in magnitude than the true slope, because the noise weakens the measured link and the estimates become inconsistent. Balance-sheet variables taken from quarterly statements are a common source in finance. The effect is persistent enough to have been called the iron law of econometrics, the suggestion being that measurement error is so widespread that estimated coefficients systematically understate how strong relationships really are.
Four assumptions remain, and unlike the conditional mean condition they can usually be checked with a plot or a test.
The data are realisations from iid random variables
In an experiment the explanatory variable is set by the person running it. Financial data offer no such control, and the explanatory variable is normally thrown off by the same process that throws off the dependent variable. CAPM is the obvious case: the market return and the portfolio return are both outcomes of investors folding new information into prices, so CAPM models the mean of the excess portfolio return conditional on the excess market return. The formal assumption is that the observed pairs are independent and identically distributed draws from their joint distribution, which lets the two members of a pair be generated together while ruling out correlation across observations. Sequentially produced time series data violate it. OLS can still be used, but the method for computing the standard errors of the estimators has to be changed.
Constant variance of the shocks
Alongside the requirement that the variance of X is strictly greater than zero, which is what makes the slope well defined, the variance of the shock is finite and does not move with the explanatory variable.
Picture the density of the shock drawn at several points along the fitted line. Under homoskedasticity that density has the same width everywhere. When the spread widens systematically as the explanatory variable grows, the shocks are called heteroskedastic.
No outliers
The chance of a large outlier in the explanatory variable has to be small, stated formally as the requirement that X has finite fourth moments. Since OLS minimises squared errors, a single extreme point contributes hugely to the objective and drags the line towards itself, so estimates can sit far from the true parameters. In one illustration a correctly fitted slope of 1.33 becomes 1.42 when a moderate outlier is added to the shock, and 2.82 when the outlier is extreme, so the slope doubles despite the 49 other points all sitting close to the true line. An outlier in the explanatory variable does the same job in reverse: a correct slope of 1.08 falls to 0.94 with a moderate outlier and to 0.39 with an extreme one, a reduction of more than 50%. The simplest defence is to plot the explanatory variable and the fitted residuals and look for extreme observations first.
With the five assumptions in place, two results follow, and everything done later with regression output rests on them both.
Unbiased in any sample, consistent in large ones
Both estimators are unbiased, so the expected value of the estimated intercept is the true intercept and the expected value of the estimated slope is the true slope. Unbiasedness is a finite sample property, holding for any number of observations rather than only for large datasets. Alongside it sits consistency: as the sample grows without limit, each estimator converges in probability to the parameter it targets.
Jointly normal in large samples
The second result is that the two estimators are jointly normally distributed. A central limit theorem applies, and the asymptotic distribution of the slope is written in the scaled form below.
Two moments drive the precision of the slope. The variance of the estimator rises with the variance of the shocks, since noisier data make the underlying rate harder to pin down, and it falls as the variance of the explanatory variable rises: observations of X strung out across a wide range identify the steepness of a line far better than observations bunched together.
The intercept has its own asymptotic distribution, depending on the same two variances plus the squared mean of the explanatory variable.
When the explanatory variable has mean zero the extra term disappears and the intercept is doing nothing more than estimating the mean of the dependent variable.
In practice the theorem is used as an approximation, and the estimated slope is treated as a normal random variable centred on the true slope.
Because the two estimators are jointly bivariate normal rather than merely normal one at a time, hypotheses involving both parameters together can be tested. Usually the slope is the parameter of interest and the intercept is carried only to stop the specification being misspecified, which would contaminate the slope.
The asymptotic variance of the slope contains two population quantities that nobody observes, so both are replaced by sample counterparts. The variance of the shocks becomes the estimated variance of the residuals.
The square root of that quantity is called the standard error of the regression, the standard deviation of the residuals in the units of the dependent variable, and it summarises how far the typical observation sits from the fitted line. The variance of the explanatory variable becomes its sample analogue. Since n times that sample variance is exactly the sum of squared deviations from the denominator of the slope estimator, the two forms below are one expression written twice.
The intercept needs the same treatment plus the term carrying the mean of the explanatory variable.
Everything needed here can be pulled from the covariance matrix of the two variables together with the sample size, so a regression can be reconstructed from summary statistics alone.
A CAPM regression of the excess return on Coca-Cola against the excess market return is fitted on 20 years of monthly data, giving 240 observations. The intercept is 0.71 and the slope is 1.37. The estimated variance of the shocks is 20.38, the estimated variance of the explanatory variable is 19.82, and its estimated mean is 0.71.
Because the estimators are normally distributed in large samples, testing a claim about a regression parameter is no different from testing a claim about the mean of any random variable obeying a central limit theorem. Six steps: state the null hypothesis and the alternative, choose the size of the test, estimate the parameter, estimate its standard error, form the test statistic, and compare it against the critical value or its p-value against the size.
The statistic measures the gap between the estimate and the value the null asserts, in units of the standard error. When the null is true it has an asymptotic standard normal distribution, so critical values are read off that distribution, and for a two-sided alternative the null is rejected whenever the absolute value of the statistic exceeds the critical value for the chosen size. A 5% test splits its size across the two tails, leaving 2.5% in each, and the critical value that does this is 1.96.
One null hypothesis is reported so often that it has its own name. Setting the hypothesised value to zero turns the test statistic into the t-statistic of the coefficient, which every statistical package prints beside the estimate. It tests the null that the parameter is zero against the alternative that it is not, and a value above 1.96 in absolute terms says the parameter is statistically different from zero. Many practitioners use 2 as a rule of thumb.
Continue with the Coca-Cola regression, where the slope is 1.37 with a standard error of 0.0655 and the intercept is 0.71 with a standard error of 0.2951.
Three numbers appear in regression output beside every coefficient, and they are three views of one calculation. Any of them settles a two-sided test, and they never disagree. The p-value converts the test statistic into a probability, the chance of seeing a statistic at least as extreme as the one obtained if the null is true. For a two-sided test it is the area in the right tail beyond the absolute value of the statistic, doubled.
A p-value below 5% rejects the null at a size of 5%. The confidence interval flips the same information around: an interval built at confidence one minus c collects the set of null values that a test of size c would fail to reject, and building one means stepping out from the estimate by the relevant critical value in each direction.
So a coefficient is statistically different from zero when the t-statistic exceeds the critical value, or the p-value falls below the size, or zero lies outside the interval. Because the interval is symmetric around the estimate, the traffic runs both ways: given an interval and its confidence level, the estimate and the standard error follow, and from those the t-statistic and the p-value.
Traded option volume on the S&P 500, puts and calls together, is regressed on a measure of overnight news. The software reports only a 90% confidence interval for the slope of [0.32, 1.89].
Coefficients say how the dependent variable responds. R squared says how much of it the model accounts for, reporting the share of the total variation in the dependent variable that the explanatory variable explains. It is bounded between 0 and 1.
Reversing that relationship is the route from a reported R squared back to the correlation coefficient: take the square root and attach the sign of the estimated slope, since the two always share a sign. Two consequences are worth carrying into an exam. Regressing Y on X and regressing X on Y produce the same R squared, because the squared correlation does not care which variable was put on the left. And a slope of exactly zero forces an R squared of zero, since that only happens when the sample covariance is zero.
The measure also links the variance of the model error to the variance of the dependent variable, and it becomes far more informative once several explanatory variables compete for the same variation.
Stock XYZ is regressed on the market. The standard deviation of its returns is 20%, that of the market returns is 15%, and the estimated beta is 0.75.
Measuring exposure to a risk factor
CAPM ties a portfolio excess return to the market excess return, both measured over the risk-free rate.
Both coefficients matter. The slope fixes how much of the market risk premium flows through to the portfolio. The intercept measures abnormal return, whatever the portfolio delivers beyond what its market exposure alone would justify. Fitted to value-weighted sector portfolios on monthly data between 1989 and 2018, with the whole US equity market as the market portfolio and the one-month Treasury bill rate as the risk-free proxy, the model gives alphas close to zero and not statistically different from zero in 8 of the 9 portfolios.
Estimating an optimal hedge ratio
A long-short equity fund aims to profit from differences between firms rather than from the market itself, yet its long and short books rarely offset market exposure completely. Regressing the return on the portfolio being hedged against the return on the hedging instrument gives a slope that is the hedge ratio minimising the variance of the residual risk, and an intercept that is the expected return on the hedged position.
| Sector | Mean return | Hedged return | Hedge ratio | SD unhedged | SD hedged | R squared |
|---|---|---|---|---|---|---|
| Consumer | 6.235 (2.30) | 2.907 (1.30) | 0.483 (10.81) | 12.0 | 9.8 | 0.335 |
| Energy | 10.188 (2.11) | 3.907 (1.01) | 0.913 (11.85) | 21.3 | 16.8 | 0.377 |
| Financial | 7.480 (1.50) | -0.770 (-0.25) | 1.199 (19.15) | 22.0 | 13.7 | 0.613 |
| Health | 8.298 (2.67) | 3.129 (1.61) | 0.751 (19.33) | 13.7 | 8.5 | 0.617 |
| Materials | 9.442 (1.98) | 1.403 (0.48) | 1.168 (20.07) | 21.0 | 12.7 | 0.634 |
| Technology | 7.713 (1.46) | -1.682 (-0.58) | 1.365 (23.63) | 23.3 | 12.6 | 0.706 |
| Utilities | 7.473 (2.22) | 4.481 (1.45) | 0.435 (7.04) | 14.9 | 13.5 | 0.176 |
Source: monthly data from 1999 until 2018, annualised returns, t-statistics in parentheses.
Read across a row and the trade-off is plain. Every unhedged mean return is positive and five of the seven are statistically different from zero, since their t-statistics exceed 1.96. Not one hedged return is statistically different from zero, and two turn negative: hedging has removed risk and taken the compensation for that risk with it. The standard deviation falls in every case, and falls hardest where R squared is highest, because a high R squared means the hedging instrument explains most of what moved the portfolio. The hedged standard deviation is the standard error of the regression, since the hedged return is the residual. The intercept has been multiplied by 12 to state it annually, and rescaling of that kind leaves the t-statistic untouched.
Evaluating a fund manager
Fund managers are judged against a style benchmark so that the ordinary performance of their category is stripped out first. Regressing the fund return on the benchmark return puts the question into the intercept, which is positive for a manager who genuinely outperforms. The slope checks whether the benchmark was the right one: a suitable benchmark produces a slope near 1 and a large R squared, the first saying it carries similar risks to the fund and the second saying it tracks the fund closely. A poor benchmark leaves the model misspecified and the slope mismeasured, and compensation for risk then gets misread as manager skill. One caution applies to any study built on funds that still exist: the sample is made of survivors, so the estimated alphas flatter.