QM 10 Applications of Simple Linear Regression in Finance

Simple linear regression measures how one variable moves with another, quantifies the direction and strength of that relationship, and lets an analyst predict one variable from the other. It is used constantly in finance to link economic indicators, earnings, and asset returns. Here we cover univariate regression: the dependence of a single dependent variable on a single independent variable.

The dependent variable, denoted Y, is the one whose variation the model tries to explain. The independent variable, denoted X, is the one whose variation is used to do the explaining. The assumed underlying relationship is written as follows.

The population values b₀ and b₁ stay hidden from us; a sample only yields estimates of them, which we mark with a hat. Ordinary least squares, or OLS, chooses the estimates that make the fitted line come as close as possible to the points, where closeness is measured by the sum of the squared vertical gaps between each observation and the line. Regression fitted this way is often called OLS regression, and the phrase is that Y is regressed on X.

At any observation i, the line supplies a fitted value, and the residual is the vertical gap separating the observed point from that fitted value.

OLS drives down the total of the squared residuals, a quantity named the sum of squares error, or SSE. Two useful properties follow from this criterion: the residuals sum to zero, and the fitted line always passes through the point of sample means, so when X equals its mean the predicted Y equals the mean of Y.

Solving for the slope and intercept

Minimizing SSE gives closed-form estimates. The slope divides the covariance of X with Y by the variance of X, capturing the portion of movement in X that it shares with Y. The intercept then falls out of the mean relationship.

Indicator variables

Many financial questions involve a qualitative change, such as whether behaviour differed before and after a policy reform. Such a variable, known either as an indicator or a dummy, equals 1 while a condition is met and 0 when it is not. In a regression on a single indicator, the intercept is the mean of Y when the indicator is 0, and the slope is the change in that mean when the indicator switches to 1. For example, a fitted model of inflation on a policy-change indicator, Y hat equals 3.44 plus 2.95 times SHIFT, says average inflation was 3.44 percent before the change and 6.39 percent after it, a rise of about three percentage points.

Least squares always produces a line, but valid inference from that line rests on four assumptions. Because the true errors cannot be observed, each assumption is checked by studying the residuals, which are the estimated errors that the regression leaves behind.

1. Linearity. The true relationship between Y and X is linear in the parameters. If it is actually curved, a straight-line fit is biased, and the residuals plotted against X show a systematic shape, often a U pattern, rather than random scatter.
2. Homoskedasticity. The variance of the errors is constant across all observations. When the spread of the residuals widens or narrows across the data, for instance across a low-volatility and a high-volatility regime, the errors are heteroskedastic and this assumption is violated.
3. Independence. The paired observations are independent, so the residuals are uncorrelated across observations. Correlated residuals are called autocorrelation, and they typically appear as a cyclical or drifting pattern when residuals are plotted in observation order. Autocorrelation can bias the coefficient estimates and invalidate the inferences drawn from them.
4. Normality. The errors are normally distributed. This applies to the errors, not to X or Y themselves. In large samples the central limit theorem lets us relax this requirement, since the test statistics remain approximately valid even when the residuals depart somewhat from normality.

A single skill ties these together: reading residual plots. Residuals that scatter randomly around zero, with roughly constant spread and no distribution skew, are the sign of a model whose assumptions hold. A visible pattern is a warning that one of the four assumptions has failed and that the estimates and their standard errors may not be trustworthy.

Minimizing SSE finds the best line, but it does not say how well that line fits. To judge fit, the total variation in Y is split into a part the regression explains and a part it does not. The total variation is the sum of squares total (SST), the explained variation is the sum of squares regression (SSR), and the unexplained variation is the sum of squares error (SSE).

These three pieces add up exactly, which is the identity that makes the fit measures work.

Coefficient of determination

The coefficient of determination, R squared, is the share of the total variation in Y that the regression explains. In a simple linear regression it also equals the square of the correlation between X and Y. It is a descriptive measure of fit, not a statistical test.

The F-test and the ANOVA table

The F-statistic tests whether the slope differs from zero, against the alternative that it is not zero. It is the ratio of the mean square regression (MSR) to the mean square error (MSE), each being a sum of squares divided by its degrees of freedom. For a simple linear regression the regression has one degree of freedom, so MSR equals SSR, and the error has n minus 2 degrees of freedom because two parameters are estimated.

A large F-statistic is evidence against the null hypothesis, because it means the explained variance is large relative to the unexplained variance. Measured in the original units of Y, the standard error of the estimate (written s_e or SEE) comes from taking the square root of MSE. A smaller SEE means the observations sit closer to the line.

In a simple linear regression the F-test judges the overall model, while a t-test judges an individual coefficient against a hypothesized value. Because only two parameters are estimated, the t-statistic is distributed as a t with n minus 2 degrees of freedom. For the slope, the test compares the estimate with a hypothesized value B₁, most often zero, scaled by the standard error of the slope.

The intercept is tested the same way, dividing its estimate minus the hypothesized value by its own standard error. Greater spread in the independent variable shrinks the standard error of the slope and so raises the calculated t-statistic. If the t-statistic falls outside the critical values, the null hypothesis is rejected; if it falls within them, it is not.

The significance level, alpha, sets the tolerance for a Type I error, which is rejecting a true null. A common choice is 0.05. Lowering alpha reduces the chance of a Type I error but raises the chance of a Type II error, failing to reject a false null. The p-value is the smallest significance level at which the null could be rejected, so a smaller p-value gives more confidence that the coefficient is real.

Once the coefficients are estimated, a forecast of Y for a chosen value of X is read straight off the fitted line. But because the line does not fit perfectly, a point forecast should be surrounded by an interval that reflects the uncertainty.

The uncertainty is summarized by the standard error of the forecast, which combines the standard error of the estimate with two penalties: one for the finite sample size, and one that grows as the forecast point moves away from the mean of X. Predictions far from the centre of the data are therefore less precise.

The prediction interval places a band around the point forecast using the appropriate critical t-value. It shrinks with a smaller standard error of the estimate, a larger sample, greater spread in X, and a forecast point closer to the mean of X.

Functional forms for non-linear data

Not every relationship is linear, and forcing a straight line onto curved data misestimates the relationship. Often a transformation of Y, of X, or of both turns a non-linear relationship into one that OLS can fit. Three log-based forms are common.

• Log-lin model: the natural log of Y is regressed on X. A one-unit rise in X produces a constant percentage change in Y, which suits data that grow or decay exponentially.
• Lin-log model: Y is regressed on the natural log of X. A one percent rise in X produces a constant absolute change in Y, which suits diminishing marginal effects, such as saturating advertising returns.
• Log-log model: the natural log of Y is regressed on the natural log of X. The slope gives the elasticity of Y against X, so a one percent rise in X produces a fixed percentage move in Y.

The right form is chosen by comparing goodness-of-fit measures and, just as importantly, by inspecting the residuals for patterns. Theory should lead, though: letting the data alone pick the form risks a model that fits well yet means little. When Y is in log form, a forecast must be converted back with the exponential. For instance, a log-lin fit ln Y equals 1.9958 plus 0.0301X gives, at X equal to 2.5, a value of ln Y equal to 2.07105, so predicted Y is the antilog, e raised to 2.07105, or about 7.937.

The capital asset pricing model, or CAPM, links an asset’s expected return to a single source of systematic risk, its beta. Beta measures how sensitive the asset is to the market. The model says the expected return equals the risk-free rate plus beta times the market risk premium, the amount by which the market is expected to beat the risk-free rate.

Expected returns cannot be observed directly, so beta is estimated empirically by regressing the asset’s excess return on the market’s excess return, where each excess return is the raw return minus the risk-free rate. The estimated slope is the beta.

The empirical evidence on CAPM is mixed, yet it stays in heavy use for managing portfolios, evaluating performance, estimating the cost of capital, and valuing projects, because it is simple and captures the intuitive link between systematic risk and expected return.