QM 10 Applications of Simple Linear Regression in Finance

Fitting a line by least squares, measuring its fit, testing its coefficients, forecasting, and estimating beta. Check-yourself items are study aids.

Simple linear regression measures how one variable moves with another, quantifies the direction and strength of that relationship, and lets an analyst predict one variable from the other. It is used constantly in finance to link economic indicators, earnings, and asset returns. Here we cover univariate regression: the dependence of a single dependent variable on a single independent variable.

The dependent variable, denoted Y, is the one whose variation the model tries to explain. The independent variable, denoted X, is the one whose variation is used to do the explaining. The assumed underlying relationship is written as follows.

b₀ is the intercept, which sets the level of Y at an X of zero. b₁ is the slope, telling how far Y shifts when X rises by a unit. The error term ε absorbs whatever the line omits, and n counts the paired observations.

The population values b₀ and b₁ stay hidden from us; a sample only yields estimates of them, which we mark with a hat. Ordinary least squares, or OLS, chooses the estimates that make the fitted line come as close as possible to the points, where closeness is measured by the sum of the squared vertical gaps between each observation and the line. Regression fitted this way is often called OLS regression, and the phrase is that Y is regressed on X.

At any observation i, the line supplies a fitted value, and the residual is the vertical gap separating the observed point from that fitted value.

OLS drives down the total of the squared residuals, a quantity named the sum of squares error, or SSE. Two useful properties follow from this criterion: the residuals sum to zero, and the fitted line always passes through the point of sample means, so when X equals its mean the predicted Y equals the mean of Y.

Solving for the slope and intercept

Minimizing SSE gives closed-form estimates. The slope divides the covariance of X with Y by the variance of X, capturing the portion of movement in X that it shares with Y. The intercept then falls out of the mean relationship.

Figure 1: A fitted regression line through scattered data

Each point is one paired observation. The least squares line is the one that makes the sum of the squared vertical gaps (one residual is marked) as small as possible.

Example 1 · Worked

An analyst studies 20 annual observations from a developing economy, regressing the annual inflation rate, Y (in percent), on annual real GDP growth, X (in percent). The summary calculations from the data are below.

Inputs for the least squares estimates

Quantity	Value
Mean of X	2.63
Mean of Y	4.92
Covariance of X and Y	3.27
Variance of X	2.58
Sum of cross products, Σ(Y − mean Y)(X − mean X)	62.13
Sum of squares of X, Σ(X − mean X)²	49.10

Estimate the slope and the intercept, and state the fitted equation.

Solution. The slope is the cross-product sum divided by the sum of squares of X, which is 62.13 divided by 49.10, or 1.27. The intercept is the mean of Y minus the slope times the mean of X, which is 4.92 minus 1.27 times 2.63, or 1.59. The fitted equation is therefore Y hat equals 1.59 plus 1.27X. It says that when real GDP growth is zero, expected inflation is 1.59 percent, and each additional one percentage point of growth raises expected inflation by 1.27 percentage points. The positive slope is consistent with demand-pull inflation, but regression measures association only and does not by itself establish causation.

Indicator variables

Many financial questions involve a qualitative change, such as whether behaviour differed before and after a policy reform. Such a variable, known either as an indicator or a dummy, equals 1 while a condition is met and 0 when it is not. In a regression on a single indicator, the intercept is the mean of Y when the indicator is 0, and the slope is the change in that mean when the indicator switches to 1. For example, a fitted model of inflation on a policy-change indicator, Y hat equals 3.44 plus 2.95 times SHIFT, says average inflation was 3.44 percent before the change and 6.39 percent after it, a rise of about three percentage points.

Check yourself

A fitted model relates return on equity to advertising spending as a share of sales: ROE hat equals 4 percent plus 1.8 times (advertising divided by sales). If advertising is 10 percent of sales, what is predicted ROE, and what is the residual for a firm that actually reported 21 percent ROE at 8 percent advertising?

At 10 percent advertising, predicted ROE is 4 percent plus 1.8 times 10 percent, which is 22 percent. At 8 percent advertising, predicted ROE is 4 percent plus 1.8 times 8 percent, which is 18.4 percent, so the residual is 21 percent minus 18.4 percent, or 2.6 percent.

Least squares always produces a line, but valid inference from that line rests on four assumptions. Because the true errors cannot be observed, each assumption is checked by studying the residuals, which are the estimated errors that the regression leaves behind.

Linearity. The true relationship between Y and X is linear in the parameters. If it is actually curved, a straight-line fit is biased, and the residuals plotted against X show a systematic shape, often a U pattern, rather than random scatter.
Homoskedasticity. The variance of the errors is constant across all observations. When the spread of the residuals widens or narrows across the data, for instance across a low-volatility and a high-volatility regime, the errors are heteroskedastic and this assumption is violated.
Independence. The paired observations are independent, so the residuals are uncorrelated across observations. Correlated residuals are called autocorrelation, and they typically appear as a cyclical or drifting pattern when residuals are plotted in observation order. Autocorrelation can bias the coefficient estimates and invalidate the inferences drawn from them.
Normality. The errors are normally distributed. This applies to the errors, not to X or Y themselves. In large samples the central limit theorem lets us relax this requirement, since the test statistics remain approximately valid even when the residuals depart somewhat from normality.

A single skill ties these together: reading residual plots. Residuals that scatter randomly around zero, with roughly constant spread and no distribution skew, are the sign of a model whose assumptions hold. A visible pattern is a warning that one of the four assumptions has failed and that the estimates and their standard errors may not be trustworthy.

Why a good fit is not the same as a valid model

OLS guarantees the smallest possible SSE among all straight lines, but that says nothing about whether a straight line was the right choice. A model can have a respectable coefficient of determination and still violate linearity or homoskedasticity, in which case the reported standard errors, t-statistics, and p-values are misleading. This is why residual diagnostics come before, not after, trusting the numbers a regression reports.

Check yourself

Which assumption holds that residual variance stays identical across every observation?

Homoskedasticity. Constant residual variance across observations is exactly this assumption; differing variance is called heteroskedasticity.

When the linearity assumption is violated, what is the tell-tale sign in the residual plot?

The residuals show a clear pattern when plotted against X, rather than scattering randomly around zero.

Minimizing SSE finds the best line, but it does not say how well that line fits. To judge fit, the total variation in Y is split into a part the regression explains and a part it does not. The total variation is the sum of squares total (SST), the explained variation is the sum of squares regression (SSR), and the unexplained variation is the sum of squares error (SSE).

These three pieces add up exactly, which is the identity that makes the fit measures work.

Coefficient of determination

The coefficient of determination, R squared, is the share of the total variation in Y that the regression explains. In a simple linear regression it also equals the square of the correlation between X and Y. It is a descriptive measure of fit, not a statistical test.

Example 2 · Worked

For the 20-observation inflation-versus-GDP regression, the variation calculations give a sum of squares total of 122.37, a sum of squares regression of 78.62, and a sum of squares error of 43.75.

Confirm the variation identity and compute the coefficient of determination.

Solution. The identity holds: 78.62 plus 43.75 equals 122.37, so SSR plus SSE equals SST. The coefficient of determination is SSR divided by SST, which is 78.62 divided by 122.37, or 0.6425, about 64 percent. The model explains roughly 64 percent of the variation in inflation, while the remaining 36 percent, the ratio of SSE to SST, is left unexplained.

The F-test and the ANOVA table

The F-statistic tests whether the slope differs from zero, against the alternative that it is not zero. It is the ratio of the mean square regression (MSR) to the mean square error (MSE), each being a sum of squares divided by its degrees of freedom. For a simple linear regression the regression has one degree of freedom, so MSR equals SSR, and the error has n minus 2 degrees of freedom because two parameters are estimated.

A large F-statistic is evidence against the null hypothesis, because it means the explained variance is large relative to the unexplained variance. Measured in the original units of Y, the standard error of the estimate (written s_e or SEE) comes from taking the square root of MSE. A smaller SEE means the observations sit closer to the line.

Example 3 · Worked

The analysis of variance for the same regression is summarized below, with n equal to 20 so the error carries 18 degrees of freedom.

ANOVA table for the inflation regression

Source	Sum of squares	Degrees of freedom	Mean square	F-statistic
Regression	78.62	1	78.62	32.35
Error	43.75	18	2.43	–
Total	122.37	19	–	–

Work out the F-statistic together with the standard error of the estimate, and interpret both.

Solution. The mean square regression is 78.62 and the mean square error is 43.75 divided by 18, which is 2.43. The F-statistic is 78.62 divided by 2.43, or about 32.35, with a p-value near 0.000021, so the slope departs significantly from zero and the null hypothesis is turned down with high confidence. The SEE is the square root of 2.43, or about 1.56 percent. The large F-statistic says X has a meaningful effect on Y, and the small standard error says the fitted line tracks the data closely.

Check yourself

A regression reports SSR of 65.3 and SSE of 43.4. What share of the variation in Y does the model explain?

About 60.1 percent. Total variation is 65.3 plus 43.4, which is 108.7, and the explained share is 65.3 divided by 108.7, or 0.601.

In a simple linear regression the F-test judges the overall model, while a t-test judges an individual coefficient against a hypothesized value. Because only two parameters are estimated, the t-statistic is distributed as a t with n minus 2 degrees of freedom. For the slope, the test compares the estimate with a hypothesized value B₁, most often zero, scaled by the standard error of the slope.

The intercept is tested the same way, dividing its estimate minus the hypothesized value by its own standard error. Greater spread in the independent variable shrinks the standard error of the slope and so raises the calculated t-statistic. If the t-statistic falls outside the critical values, the null hypothesis is rejected; if it falls within them, it is not.

The significance level, alpha, sets the tolerance for a Type I error, which is rejecting a true null. A common choice is 0.05. Lowering alpha reduces the chance of a Type I error but raises the chance of a Type II error, failing to reject a false null. The p-value is the smallest significance level at which the null could be rejected, so a smaller p-value gives more confidence that the coefficient is real.

Example 4 · Worked

An analyst regresses the actual Swedish consumer inflation rate on the consensus forecast made by professional forecasters, using 60 observations. If the forecasts are unbiased, the intercept should be 0.0 and the slope should be 1.0. The regression output is below.

Regression output: actual inflation on the forecast

	Coefficient	Standard error	t-statistic
Intercept	0.0001	0.0002	0.5000
Forecast (slope)	0.9830	0.0155	63.4194

R squared is 0.9859, the standard error of the estimate is 0.0009, and n is 60.

At the 5 percent level, test whether the slope equals 1.0 and whether the intercept equals 0.0. The critical t-values are plus or minus 2.0017.

Solution. With 60 observations there are 58 degrees of freedom. For the slope, the test statistic is 0.9830 minus 1.000, divided by 0.0155, which is −1.0968. That falls inside plus or minus 2.0017, so the null of a slope equal to one is not rejected. For the intercept, the statistic is 0.0001 minus 0.0000, divided by 0.0002, which is 0.5000, again inside the bounds, so the null of an intercept equal to zero is not rejected. Both results together say the forecasts look unbiased. Note that the t-statistics printed in the output, such as 63.4194 for the slope, test the coefficient against zero, which is a different null from the one tested here.

Check yourself

A regression of return volatility on asset size reports a slope of −4.1589 with a standard error of 1.8718, giving a t-statistic of −2.2219. What does the sign tell you about the scatterplot, and is the relationship different from zero at the 5 percent level if the critical value is about 2.0?

The negative slope means the fitted line is downward sloping, so larger assets are associated with lower volatility. The t-statistic of −2.2219 lies outside plus or minus 2.0, so the slope is significantly different from zero at the 5 percent level.

Once the coefficients are estimated, a forecast of Y for a chosen value of X is read straight off the fitted line. But because the line does not fit perfectly, a point forecast should be surrounded by an interval that reflects the uncertainty.

The uncertainty is summarized by the standard error of the forecast, which combines the standard error of the estimate with two penalties: one for the finite sample size, and one that grows as the forecast point moves away from the mean of X. Predictions far from the centre of the data are therefore less precise.

The prediction interval places a band around the point forecast using the appropriate critical t-value. It shrinks with a smaller standard error of the estimate, a larger sample, greater spread in X, and a forecast point closer to the mean of X.

Example 5 · Worked

A regression uses 36 monthly returns to relate the return of an index of electric-vehicle makers (the dependent variable) to the price return of lithium carbonate (the independent variable). The intercept is 0.0095 and the slope is 0.2354. The sum of squares error is 0.0715, and next month the lithium carbonate price return is forecast at −0.01, or −1 percent.

Compute the standard error of the estimate and the predicted index return for next month.

Solution. The standard error of the estimate equals the square root of MSE, so it is the square root of 0.0715 divided by (36 minus 2). That is the square root of 0.0715 divided by 34, or about 0.0459. The predicted index return is 0.0095 plus 0.2354 times −0.01, which is 0.0095 minus 0.002354, or about 0.0071, roughly a 0.71 percent return. A common error is to apply the slope to a positive 0.01, which would wrongly give 0.0119.

Functional forms for non-linear data

Not every relationship is linear, and forcing a straight line onto curved data misestimates the relationship. Often a transformation of Y, of X, or of both turns a non-linear relationship into one that OLS can fit. Three log-based forms are common.

Log-lin model: the natural log of Y is regressed on X. A one-unit rise in X produces a constant percentage change in Y, which suits data that grow or decay exponentially.
Lin-log model: Y is regressed on the natural log of X. A one percent rise in X produces a constant absolute change in Y, which suits diminishing marginal effects, such as saturating advertising returns.
Log-log model: the natural log of Y is regressed on the natural log of X. The slope gives the elasticity of Y against X, so a one percent rise in X produces a fixed percentage move in Y.

The right form is chosen by comparing goodness-of-fit measures and, just as importantly, by inspecting the residuals for patterns. Theory should lead, though: letting the data alone pick the form risks a model that fits well yet means little. When Y is in log form, a forecast must be converted back with the exponential. For instance, a log-lin fit ln Y equals 1.9958 plus 0.0301X gives, at X equal to 2.5, a value of ln Y equal to 2.07105, so predicted Y is the antilog, e raised to 2.07105, or about 7.937.

Check yourself

Which functional form suits a test of whether raising X by one unit produces a fixed percentage move in Y?

A log-lin model, where the natural log of Y is regressed on X. Adding one unit to X then translates into a steady percentage shift in Y.

Two factors that make the standard error of the forecast smaller are a larger sample and a forecast point close to the mean of X. Name a third.

A smaller standard error of the estimate, which reflects a better-fitting model. Greater variation in the independent variable also helps, since it sharpens the slope estimate.

The capital asset pricing model, or CAPM, links an asset’s expected return to a single source of systematic risk, its beta. Beta measures how sensitive the asset is to the market. The model says the expected return equals the risk-free rate plus beta times the market risk premium, the amount by which the market is expected to beat the risk-free rate.

CAPM is a one-factor model. It assumes all non-systematic, idiosyncratic risk can be diversified away, so only market risk is priced.

Expected returns cannot be observed directly, so beta is estimated empirically by regressing the asset’s excess return on the market’s excess return, where each excess return is the raw return minus the risk-free rate. The estimated slope is the beta.

Z_i,t is the asset excess return and Z_m,t is the market excess return. The slope estimate b₁ is the CAPM beta.

Example 6 · Worked

Beta is estimated for LVMH from two years of weekly returns, using the CAC 40 Index as the market proxy and the yield on three-month French government securities as the risk-free rate. The regression output is below.

ANOVA and coefficients for the LVMH beta regression

Source	Degrees of freedom	Sum of squares	Mean square
Regression	1	0.108622	0.108622
Residual	102	0.041994	0.000412
Total	104	0.150616	–

	Coefficient	Standard error	t-statistic
Intercept	0.001	0.002	0.525
Excess return (beta)	1.3684	0.084	16.243

1. State the estimated beta and the share of variation the model explains.

Solution. The beta is the slope coefficient, 1.3684, so LVMH excess returns move about 1.37 times as much as the market excess return. The coefficient of determination is SSR divided by SST, which is 0.108622 divided by 0.150616, or about 0.721. About 72.1 percent of the variation in LVMH excess returns is explained by the market.

2. Test whether beta differs from zero at the 5 percent level, using a critical value of 1.96.

Solution. The t-statistic of 16.243 is far above 1.96, so the null that beta equals zero is rejected. The beta is strongly significant.

3. Build the 95 percent confidence interval for beta.

Solution. The interval is the estimate plus or minus the critical value times the standard error: 1.3684 plus or minus 1.96 times 0.084, which is 1.3684 plus or minus 0.1646. The bounds are 1.2038 and 1.5330, so there is 95 percent confidence that the true beta lies in that range.

4. If the market risk premium rises by 0.5 percent while the risk-free rate is unchanged, what is the expected move in the asset’s excess return?

Solution. The change is beta times the change in the premium: 1.3684 times 0.5 percent, which is about 0.684 percent.

The empirical evidence on CAPM is mixed, yet it stays in heavy use for managing portfolios, evaluating performance, estimating the cost of capital, and valuing projects, because it is simple and captures the intuitive link between systematic risk and expected return.

Check yourself

A CAPM regression for FIRN AG reports an intercept of 0.0077 and a slope (beta) of 0.8260 on the market risk premium. If the market risk premium is 5 percent, what excess return does the model predict?

About 4.9 percent. The prediction is 0.0077 plus 0.8260 times 0.05, which is 0.0077 plus 0.0413, or 0.0490.

Consider two assets, one with a beta of 0.5 and another with a beta of 1.5. In a falling market, which should decline less?

The one with beta 0.5. A lower beta means less sensitivity to market moves, so it is expected to fall less when the market falls (and to rise less when the market rises).

Overview

Linear Regression and Least Squares Estimation

Solving for the slope and intercept

Indicator variables

Assumptions and Residual Diagnostics

Goodness of Fit: R-Squared, the F-Test, and the Standard Error of the Estimate

Coefficient of determination

The F-test and the ANOVA table

Hypothesis Tests on the Regression Coefficients

Prediction and Functional Forms

Functional forms for non-linear data

The Capital Asset Pricing Model