Eduzan

Eduzan / 01 Quantitative Methods

QM 4 – Extensions of Multiple Regression

Worked examples are fully visible. Check-yourself items are study aids you can reveal one at a time.

A multiple regression can be misleading for two quite separate reasons. The first is that one of the underlying assumptions has been violated, so the whole estimation procedure is compromised. The second, and the subject of this section, is far more local: the model may be sound in principle, yet a handful of observations, sometimes a single one, may be dragging the fitted coefficients away from the pattern the rest of the data displays. An observation of that kind is called an influential observation, defined as one whose presence in the sample materially changes the regression results.

Influence is not a single phenomenon. It arrives through two distinct doors, and it is worth keeping them apart because the diagnostic that catches one will not catch the other.

A high-leverage point is an observation with an extreme value of an independent variable. It sits far from the centre of the X data.
An outlier is an observation with an extreme value of the dependent variable. Its Y value sits far from the value the fitted model predicts for it, so it produces a large residual.

Both kinds of observation are unusual relative to the bulk of the sample, but they announce themselves differently. Consider a scatter of points showing a mild positive relation between X and Y. A high-leverage point breaks the pattern along the horizontal axis: its X value is far larger, or far smaller, than any other X in the sample. Fit the regression twice, once including that point and once excluding it, and the two fitted lines will separate. The reason is mechanical. Ordinary least squares minimises the sum of squared vertical deviations, and a point positioned far out along the X axis acts as a long lever arm, so a small vertical displacement at that point produces a large rotation of the fitted line.

Figure 1: A high-leverage point pivots the fitted line

The isolated observation has an extreme X value, so it exerts a long lever arm on the least squares fit. Deleting it flattens the line and changes the estimated slope.

An outlier breaks the pattern along the vertical axis instead. Its X value may be perfectly ordinary, but its Y value sits far above or far below the fitted line, producing a residual much larger in absolute size than the residuals of its neighbours. Including it pulls the fitted line toward it, again altering the slope estimates and the goodness-of-fit statistics.

Figure 2: An outlier pulls the fitted line toward itself

The circled observation has an ordinary X value but an extreme Y value. The vertical guide measures the gap between the observed Y and the value predicted by the line fitted without the outlier, and it is the size of that gap that flags the point. Including the observation lifts the whole fitted line toward it.

Unusual is not the same as influential

Neither a high-leverage point nor an outlier is automatically a problem, and this is the distinction candidates most often blur. A high-leverage point may sit far out along the X axis and yet lie almost exactly on the regression line that the remaining observations produce. In that case it confirms the relation rather than distorting it, and deleting it would change very little. Trouble arises only when the extreme observation is also distant from the fitted line. Then the least squares criterion tilts the line toward it, and both the slope coefficients and the goodness-of-fit measures move.

The operational definition follows from this. An observation is influential when removing it from the sample produces a substantial change in the estimated regression function. Leverage and residual size are the symptoms; the change in the estimated function on deletion is the diagnosis.

Why a scatterplot is not enough

With one independent variable, a scatterplot settles the matter by eye. With several independent variables there is no single picture to look at. An observation can be entirely unremarkable on every variable taken one at a time and still occupy an extreme position in the joint space of all the regressors. Multiple regression therefore needs numerical measures that quantify how far an observation sits from the centre of the data, and how badly the model misses it. The next two sections develop exactly those two measures: leverage for the independent variables, and the studentized residual for the dependent variable.

Nothing in this calculation needs to be done by hand. Every mainstream statistical package reports leverage for each observation as part of the regression diagnostics. What the analyst supplies is the judgement about what to do once a point is flagged.

Example 1 · Worked

A senior specialty retail analyst, working from the broad themes of health consciousness and an ageing population, asks a junior analyst to initiate coverage of nutritional supplement retailers. The junior analyst assembles a cross-sectional dataset of 15 such retailers and regresses operating profit margin (OPM) on two variables: the count of unique products carried (PROD), covering items such as vitamins, probiotics, antioxidants and joint supplements, and the share of sales made online (ONLINE).

Panel A: OPM regression, full sample of 15 retailers

Variable	Coefficient	Standard error	t-Statistic	P-value
Intercept	7.03	13.60	0.52	0.61
PROD	0.77	0.28	2.70	0.02
ONLINE	−0.29	0.28	−1.02	0.33

The leverage measure is then computed for each of the 15 companies. Observation 7 records a leverage of 0.80; no other observation comes close. The regression is re-run with that company removed.

Panel C: OPM regression after removing one observation

Variable	Coefficient	Standard error	t-Statistic	P-value
Intercept	9.81	24.80	0.40	0.70
PROD	0.74	0.33	2.25	0.05
ONLINE	−0.36	0.61	−0.59	0.57

1. Using the full-sample results, which independent variables have slope coefficients different from zero at the 5 percent level of significance?

Solution. Compare each P-value with 0.05. PROD has a P-value of 0.02, or 2 percent, which is below 0.05, so its slope coefficient is different from zero at the 5 percent level. ONLINE has a P-value of 0.33, far above 0.05, so its coefficient cannot be distinguished from zero. The intercept, with a P-value of 0.61, is likewise indistinguishable from zero. Only PROD is significant.

2. Which observation or observations may be influential on the leverage criterion, and why?

Solution. Apply the rule of thumb with k = 2 independent variables and n = 15 observations:

3 × [(2 + 1) ÷ 15] = 3 × (3/15) = 9/15 = 0.60.

Any observation with leverage above 0.60 is flagged. Observation 7 alone qualifies, with a leverage of 0.80, so it is the single potentially influential high-leverage point in this sample.

3. How do the regression results change once that observation is dropped, and does the conclusion survive?

Solution. The intercept rises from 7.03 to 9.81, and the evidence on PROD weakens: its P-value moves from 0.02 to 0.05, its coefficient falls slightly from 0.77 to 0.74, and its standard error widens from 0.28 to 0.33. The ONLINE coefficient becomes more negative, −0.36 against −0.29, with its standard error more than doubling to 0.61, so it remains insignificant. The observation therefore does exert influence, but the substantive conclusion is unchanged: at the 5 percent level PROD still has a slope coefficient different from zero. The finding is weakened by the deletion, not overturned by it.

Notice what question 3 illustrates. Detecting a high-leverage point is only the opening move. The analytical work is comparing the two sets of estimates and asking whether any conclusion that matters actually flips. Here it does not, so the deletion is informative but not decisive.

Leverage says nothing about the dependent variable, so it cannot find an outlier. For that the preferred tool is the studentized residual, and specifically the studentized deleted residual, which is built on a deliberately awkward idea: judge each observation using a model that has never seen it.

How the measure is constructed

The logic runs in three steps.

Estimate the regression on the full sample of n observations. Then delete observations one at a time and re-estimate the model each time on the remaining (n − 1) observations.
For observation i, compare the value of Y actually observed with the value predicted by the model that was fitted without observation i. The difference is the deleted residual.
Scale that deleted residual by the estimated standard deviation of the residuals. The scaled quantity is the studentized deleted residual.

The deleted-residual form on the left; the equivalent form on the right uses only quantities from the original full-sample regression.

The two expressions are equal. In the notation used above, e_i^* is the residual with observation i deleted, s_e* is the standard deviation of all the residuals, k is the number of independent variables, MSE_(i) is the mean squared error of the model estimated with observation i deleted, and h_ii is the leverage of observation i. The right-hand expression matters practically because it means the software does not have to run n separate regressions to produce the measure.

Look at the denominator of the right-hand form for a moment. The term (1 − h_ii) shrinks as leverage rises, which shrinks the denominator and inflates the studentized residual. The measure therefore already carries some information about position in the X space, which is one reason it is such an effective detector.

The two thresholds

Two separate rules are applied to the studentized residual, and they answer different questions.

Rules of thumb for studentized residuals

Condition	Conclusion
Absolute value of t_i* above 3	Treat the observation as an outlier
Absolute value of t_i* above the critical t-value with (n − k − 2) degrees of freedom at the chosen significance level	Treat the outlier as potentially influential

The degrees of freedom deserve attention: (n − k − 2), not the (n − k − 1) familiar from ordinary coefficient tests, because one further observation has been removed from the estimation.

The two rules are not alternatives, and they are not equally sharp. The first is a blunt, sample-independent screen: a studentized residual beyond three in absolute value is far enough into the tail of a t-distribution that the observation deserves a look whatever the sample size. The second is a proper test, and because the critical value falls as degrees of freedom rise, it becomes stricter in small samples and more permissive in large ones. In a sample of 15 the critical value sits above 2.2, while in a sample of 100 it drops close to 1.99. An observation can clear the second hurdle without clearing the first, which is the common case in small samples, and the sensible reading is that the model misses it by more than sampling variation comfortably explains.

To make this concrete, return to the 15 nutritional supplement retailers. With n = 15 and k = 2, the degrees of freedom are 15 − 2 − 2 = 11, and the two-sided critical value at the 5 percent level is 2.2010. Plotting the studentized residuals for the 15 companies shows that Observations 2 and 3 have absolute values above that critical value, so both should be flagged for closer examination. Note how different this answer is from the leverage answer, which singled out Observation 7. The two diagnostics are looking at different things and there is no reason for them to agree.

What each measure detects

Measure	Extreme value in	Range or procedure	Flag when
Leverage	Independent variable	Bounded between 0 and 1	Leverage exceeds 3(k + 1)/n
Studentized residual	Dependent variable	Compare the calculated absolute t-statistic with the critical t-value	Calculated absolute t-statistic exceeds the critical t-value

Both belong in the toolkit, together with an influence plot that shows leverage and studentized residual on the same picture, and together with the revised regression output after a candidate observation has been removed. Used as a set they reveal not only which observations are influential but why: because of an extreme X value, because of an extreme Y value, or because of both.

Example 2 · Worked

You are analysing a regression model of company return on assets (ROA) estimated on 26 observations with three independent variables, and you are concerned about outliers and influential observations. Statistical software returns the following studentized residual t-statistic for each observation.

Studentized residuals by observation, ROA regression

Obs.	Studentized residual	Obs.	Studentized residual
1	0.182	14	−0.696
2	−0.956	15	0.297
3	2.915	16	−0.531
4	1.932	17	−0.212
5	−4.033	18	0.411
6	−0.328	19	−0.275
7	0.357	20	−0.597
8	−0.381	21	−0.463
9	−0.275	22	−0.616
10	−0.058	23	0.607
11	0.503	24	0.752
12	0.244	25	0.015
13	0.646	26	−0.018

1. Which observations, if any, count as outliers on the studentized residual criterion at the 5 percent level of significance?

Solution. First fix the degrees of freedom: n − k − 2 = 26 − 3 − 2 = 21. The two-sided critical values at the 5 percent level with 21 degrees of freedom are ±2.080. Now scan the table for absolute values above 2.080. Observation 3 has a studentized residual of 2.915 and Observation 5 has −4.033, an absolute value of 4.033. No other observation exceeds 2.080; the next largest in absolute terms is Observation 4 at 1.932, which falls short. Observations 3 and 5 are the outliers.

2. What action should you recommend on the influential observations?

Solution. The studentized residuals point to Observations 3 and 5, so those two should be investigated before anything is deleted. The first thing to establish is whether the values are genuine: check for data entry errors, misaligned fields and quality problems in the underlying source. Only after the data have been confirmed as accurate does the question of how to treat them in the model arise. Deleting them first and asking questions later would risk discarding real information.

Detection is the easy half. Once an observation has been flagged, the analyst has to work out why it is unusual and what, if anything, should be done. Automatically excluding one or several influential data points can materially change the coefficient estimates and the interpretation that rests on them, so the decision should never be automatic.

Case one: the data are wrong

Sometimes an influential point is simply an error: a figure keyed in with a misplaced decimal, a percentage recorded as a decimal fraction, a measurement taken on the wrong basis. The remedy is straightforward. Correct the entry if the true value can be established, or discard the observation if it cannot, then re-estimate the regression on the cleansed sample.

A softer alternative is winsorizing the dataset, which replaces extreme values at the tails with the nearest values inside a chosen cut-off rather than deleting the observations entirely. This limits the pull of an extreme figure while keeping the rest of the information that observation contains.

Case two: the data are right

The more interesting case is the influential point that turns out to be accurate. Then the model, not the data, is under suspicion. An accurate observation that the model badly misses is evidence of one of two things: an important explanatory variable has been left out, so the model has no way of accounting for what makes that observation special, or a regression assumption is being violated. The remedy is to identify and add the missing explanatory variables, and to re-check that the specification satisfies the regression assumptions.

Underneath all of this sits a judgement that no statistic can make for you. Whether an influential observation is a nuisance to be removed or a genuine feature of the population being studied depends on the purpose of the model. A credit model built to predict ordinary corporate defaults may reasonably set aside a single catastrophic failure; a model built to understand tail risk would be worthless if it did the same.

Why leverage and studentized residuals can disagree, and what to make of it

In the retailer example, leverage singles out Observation 7 while the studentized residuals single out Observations 2 and 3. This is not a contradiction. Observation 7 sits far out in the space of the independent variables but the model may still fit it well, in which case it has the capacity to influence without exercising it. Observations 2 and 3 sit at ordinary positions in the X space but the model predicts them badly. The useful reading is that leverage answers the question of how much influence an observation could exert, while the studentized residual answers the question of how far the model actually misses it. An observation that scores highly on both is the one that should worry you most, which is precisely why an influence plot puts the two measures on the same axes.

Many of the characteristics an analyst wants to bring into a regression are not measured on a numerical scale at all. Industry membership, whether an event has occurred, whether a company is regulated: these are qualitative facts. A dummy variable, also called an indicator variable, carries such a fact into the model by taking the value 1 when a stated condition holds and 0 when it does not. Its purpose is to let the regression distinguish between groups or categories within the data.

Where dummy variables come from

An inherent property of the data, such as the industry sector a company belongs to.
A characteristic naturally expressed as a true or false condition, such as whether an observation is dated before or after a defining market event.
A condition constructed by the analyst from a continuous characteristic, such as whether company sales fall below or above a chosen threshold.

The third route is worth pausing on. It converts continuous information into a binary flag, which discards detail, so it should be used when the threshold itself is economically meaningful rather than as a matter of convenience.

How many dummies

The counting rule is the single most examinable point in this material: to distinguish among n categories, use n − 1 dummy variables. Five industry sectors therefore require four dummies. The category left without a dummy of its own does not disappear from the analysis; it becomes the base or control group, and every dummy coefficient is then read as a difference relative to that base.

Four dummies representing membership of five industry sectors

Industry sector	Technology	Financial Services	Health Care	Energy
Technology	1	0	0	0
Financial Services	0	1	0	0
Health Care	0	0	1	0
Energy	0	0	0	1
Food & Beverage	0	0	0	0

Food & Beverage carries no dummy of its own and therefore serves as the base, or control, group.

The reason for n − 1 rather than n is not stylistic. Multiple regression assumes that no exact linear relation exists between two or more independent variables. If a dummy were included for every category, the full set of dummies would add to 1 for every single observation, which is exactly the constant column used to estimate the intercept. That perfect linear dependence makes the coefficients impossible to identify, and the regression fails outright rather than merely fitting badly.

Which category is chosen as the base is, strictly speaking, arbitrary: the fitted values, the residuals and the R-squared are identical whichever one is omitted. What changes is the presentation. Every dummy coefficient and every significance test is expressed relative to the omitted group, so a base chosen badly makes the output hard to read. A sensible choice is the largest category, which gives the most precisely estimated benchmark, or the category that is economically the natural reference point, such as unregulated companies in a study of regulation.

One practical consequence follows immediately. Because the base category is absorbed into the intercept, the intercept coefficient is not a general average. It is the fitted value for the base group when every other independent variable equals zero. Change which category is left out and the intercept changes with it, even though the fitted values for the sample do not move at all.

A dummy variable can enter a regression in two structurally different ways, and the choice determines what the model is allowed to say about the difference between groups.

The intercept dummy

The simpler form adds the dummy on its own alongside a continuous variable X. This is the intercept dummy.

One continuous regressor and one intercept dummy.

A single equation of this form is really two lines of best fit stacked into one model, selected by the value of the dummy:

When D = 0, the equation collapses to Y = b₀ + b₁X + ε, the base category.
When D = 1, it becomes Y = (b₀ + d₀) + b₁X + ε, the category the shifted intercept applies to.

Graphically the two lines are parallel and the vertical gap between them is d₀. That gap may be positive or negative. The essential restriction is that the effect of X on Y, the slope b₁, is forced to be identical across the two groups. The intercept dummy says the groups differ by a constant amount, no matter what X is.

The slope dummy

The alternative multiplies the dummy by the continuous variable. This is the slope dummy, and the product term it creates is an interaction term.

One continuous regressor and one slope dummy, which is the interaction of D with X.

When D = 0, the equation is Y = b₀ + b₁X + ε, once again the base category.
When D = 1, it becomes Y = b₀ + (b₁ + d₁)X + ε, so the slope for that category is the sum of the two coefficients.

Now the two lines share an intercept and fan apart. The difference between the groups is no longer constant: it is zero when X is zero and grows in proportion to X. As with the intercept shift, the difference in slopes may be positive or negative.

Both together

Nothing prevents a model from allowing both effects at once, which is done by combining the two specifications.

Intercept and slope both permitted to differ between the two groups.

When D = 0, Y = b₀ + b₁X + ε.
When D = 1, Y = (b₀ + d₀) + (b₁ + d₁)X + ε.

In this richer version the gap between the categories has two parts: a fixed component d₀ and a component d₁X that varies with the size of the independent variable. All three specifications extend directly to more categories and more independent variables, in which case the picture would contain one fitted line for each category rather than two.

Figure 3: What each kind of dummy does to the fitted line

An intercept dummy shifts the line without turning it. A slope dummy turns the line without shifting it. Including both allows the two categories to differ by an amount that changes with X.

Example 3 · Worked

An interviewer at a global macro hedge fund describes the following assignment. You must study the relation between stock market returns and GDP growth across a group of countries, and you must use a binary variable to classify each country by stock market type, coded 1 for emerging and 0 for developed.

1. Which construction does that assignment call for: a slope dummy that allows the slope to change, an intercept dummy that allows the intercept to change, or an interaction term?

Solution. An intercept dummy. The variable being introduced classifies countries by the development status of their stock market, and nothing in the brief asks for the sensitivity of returns to GDP growth to differ between the two groups. A dummy entered on its own shifts the intercept, which is exactly a constant difference in average returns between emerging and developed markets, holding GDP growth constant.

2. The interviewer now asks you to capture the effect on stock returns of the interaction between each country’s GDP growth and its stock market development status. Which specification should you use, writing GDPG for GDP growth, EM for emerging market status and DM for developed market status?

Solution. The correct model is

Stock return = b₀ + b₁GDPG + d₁EM + d₂(EM × GDPG) + ε.

It contains the continuous variable GDPG, one dummy for emerging market status with developed markets left as the base case, and the product term EM × GDPG that captures the interaction. A specification carrying dummies for both EM and DM would breach the n − 1 rule with two categories and two dummies, and a specification without the product term would not capture an interaction at all.

3. A second interviewer reports that an analyst on her team explained a cross-section of company returns on assets using a regulation flag (REG, set to 1 for a regulated company and 0 otherwise), market share (MKTSH), and REG_MKTSH, which multiplies those first two variables together. The fitted model is RET = 0.50 − 0.5REG + 0.4MKTSH − 0.2REG_MKTSH. Which of the following readings are correct? (A) Holding market share constant, the average return of a regulated firm is at least 0.5 percent lower than that of a non-regulated firm. (B) Non-regulated companies with larger market shares have lower returns on assets than regulated companies. (C) For each increase in market share, a regulated firm has an increasingly lower return on assets than an unregulated firm.

Solution. A and C are both correct; B is not.

A. The coefficient on REG is −0.5. As market share approaches zero, the regulated firm has 0.5 percent less return. Put differently, if the contribution of market share to return is the same for both, that is 0.2 × MKTSH for the regulated firm equals 0.4 × MKTSH for the non-regulated firm, the regulated firm still carries 0.5 percent less return.

C. The relevant sum of coefficients is −0.5 + 0.4 − 0.2 = −0.3. Consider a 1 percent increase in market share for both types of firm. The regulated firm gains 0.4 percent from MKTSH and loses 0.2 percent through the interaction term, a net 0.2 percent, while the non-regulated firm gains the full 0.4 percent. The regulated firm therefore ends up 0.2 percent worse off: 0.2(1%) − 0.4(1%) = −0.2%. The 0.5 percent level difference does not enter here, because the question concerns the change in return produced by the 1 percent increase in market share.

B. Incorrect. The coefficient on MKTSH is positive and the coefficient on REG is negative, so the statement runs against both signs.

Testing whether the regression function differs between groups is simple once dummies are in place: run the ordinary individual t-test on each dummy coefficient. A coefficient significantly different from zero means that category differs significantly from the base.

The following study makes the machinery concrete. An analyst is asked how the characteristics of a mutual fund affect its returns, using a large database of funds spanning three investment styles: blend, growth and value. The dependent variable is the five-year average annual return of the fund. Four continuous independent variables are used: the fund expense ratio (EXP), the portfolio cash ratio (CASH), the age of the fund (AGE) and the natural logarithm of fund size (SIZE).

Three style categories call for two dummies. BLEND takes the value 1 for a blend fund and 0 otherwise, GROWTH takes the value 1 for a growth fund and 0 otherwise, and value funds carry no dummy, making them the base category. The estimated model is

Two intercept dummies for style, with value funds as the omitted base.

Fit statistics and analysis of variance, style dummies model

Statistic	Value
R-squared	0.1230
Adjusted R-squared	0.1228
Standard error	4.224
Observations	23,025

Source	df	Sum of squares	Mean square	F	Significance
Regression	6	57,636.46	9,606	538	0
Residual	23,018	410,816.9	17.85
Total	23,024	468,453.3

Estimated coefficients, style dummies model

Term	Estimate	Std. error	t-Stat.
Intercept	−2.909	0.2990	−9.738
EXP	−0.586	0.0495	−11.824
CASH	−0.032	0.0029	−11.168
AGE	0.074	0.0033	22.605
SIZE	0.267	0.0141	18.924
BLEND	0.661	0.0678	9.749
GROWTH	2.498	0.0748	33.394

Reading the output

Every slope coefficient and the intercept are significantly different from zero. The R-squared of 0.1230 says the specification accounts for 12.30 percent of the variation in observed five-year average annual returns. The adjusted R-squared of 0.1228 is barely below it, and that near-equality is itself informative: adding variables that carry no explanatory weight would drive the adjusted figure noticeably below the unadjusted one, so the gap of two hundredths of a percentage point indicates that the specification is built on statistically meaningful variables.

The dummy coefficients are read as differences from the base. Blend funds deliver average annual returns exceeding the value category by 0.66 percent, and growth funds deliver 2.50 percent more than the value base. The intercept of −2.909 says that an average annual return of −2.91 percent is left unexplained by the independent variables in the model.

Adding slope dummies

The first model implies that each additional year of age is worth 0.07 percent of annual return, and it imposes that figure on all three styles alike. The analyst wonders whether the age effect really is common across styles, or whether growth and blend funds age differently from value funds. Answering that requires interaction terms. Two are constructed: AGE_BLEND, which equals AGE when BLEND is 1 and zero otherwise, and AGE_GROWTH, which equals AGE when GROWTH is 1 and zero otherwise.

Intercept dummies and slope dummies for style together.

Fit statistics and analysis of variance, intercept and slope dummies

Statistic	Value
R-squared	0.123
Adjusted R-squared	0.123
Standard error	4.224
Observations	23,025

Source	df	Sum of squares	Mean square	F	Significance
Regression	8	57,760.46	7,220	404.6	0.000
Residual	23,016	410,692.9	17.84
Total	23,024	468,453.3

Estimated coefficients, intercept and slope dummies

Term	Estimate	Std. error	t-Stat.	P-value
Intercept	−2.810	0.3060	−9.183	0.00
EXP	−0.587	0.0496	−11.839	0.00
CASH	−0.032	0.0029	−11.211	0.00
AGE	0.065	0.0059	11.012	0.00
SIZE	0.267	0.0141	18.906	0.00
BLEND	0.603	0.1088	5.546	0.00
GROWTH	2.262	0.1204	18.779	0.00
AGE_BLEND	0.005	0.0077	0.627	0.53
AGE_GROWTH	0.020	0.0081	2.478	0.01

The values and significance of the original slope coefficients are little changed, but the revised model says considerably more about age. For the base group, the value funds, the AGE coefficient of 0.065 means those funds earn an extra 0.065 percent of return as each year passes. That is the reading when BLEND and GROWTH are both zero.

AGE_BLEND is not significant, with a P-value of 0.53, so there is no evidence that blend funds age differently from value funds. AGE_GROWTH is significant, with a P-value of 0.01. For growth funds the extra annual return per year of age is therefore the sum of the AGE and AGE_GROWTH coefficients:

The effective age slope for growth funds is the base slope plus the interaction coefficient.

Putting the level effect and the age effect together, the overall result is that returns of growth funds exceed those of value funds by 2.347 percent, which is 2.262 percent from the GROWTH intercept dummy plus 0.085 percent from AGE together with AGE_GROWTH, for each year of the life of the fund since inception.

Why the R-squared barely moved when the interaction terms were added

Adding AGE_BLEND and AGE_GROWTH lifted the regression sum of squares from 57,636.46 to 57,760.46, an improvement of 124 out of a total sum of squares of 468,453.3. R-squared rounds to 0.123 either way. Yet the second model is clearly more informative: it shows that the common age slope of 0.074 in the first model was a blend of a 0.065 effect for value and blend funds and a 0.085 effect for growth funds. The lesson is that explanatory power and economic insight are not the same thing. A term can add almost nothing to R-squared, because the group it applies to is a modest share of a very large sample, and still answer the question the analyst actually asked.

So far the qualitative information has entered on the right-hand side of the equation. Turn it around: what happens when the variable to be explained is itself categorical? Predicting whether a company will go bankrupt is a question of this kind. The outcome is bankrupt or not bankrupt, while the explanatory data, return on equity, the debt-to-equity ratio, the debt rating, are ordinary continuous measures. A dependent variable of this type is called a qualitative dependent variable or categorical dependent variable. The binary case is the important one, although more than two categories is possible.

Why ordinary least squares fails here

Suppose the outcome is coded Y = 1 for bankrupt and Y = 0 otherwise, and the analyst simply runs a linear regression on three explanatory variables. The result is a linear probability model.

The linear probability model, which fits a straight line to a variable that can only take the values 0 and 1.

Two objections follow, and neither is a technicality.

The fitted values are not confined to the unit interval. Depending on the estimated coefficients and the observed values of the independent variables, the predicted value can exceed 1 or fall below 0. A predicted probability of 1.2, or of −0.1, has no meaning: the probability of bankruptcy, or of anything else, cannot lie outside the range from 0 to 1.
The linear form imposes a constant marginal effect. A straight line asserts that the probability of bankruptcy responds identically to a given change in the debt-to-equity ratio wherever that ratio happens to start. That is implausible at the extremes. A company already carrying very little debt, or already carrying a crushing amount, is unlikely to have its bankruptcy probability moved by the same amount as a company in the middle of the distribution.

The logistic transformation

The repair is to apply a non-linear transformation to the probability itself, and then relate the transformed probability linearly to the independent variables. The transformation used almost universally is the logistic one. Let P be the probability that the event occurs.

The logistic transformation, or logit, of the event probability.

Work outwards from the inside. The ratio P/(1 − P) divides the probability that the event happens by the probability that it does not, which is the odds of the event. If the probability that a company goes bankrupt is 0.75, then

Odds of 3 to 1: bankruptcy is three times as likely as survival.

Taking the natural logarithm of the odds gives the log odds, also known as the logit function. The transformation does the work that the linear probability model could not: odds run from 0 to infinity as the probability runs from 0 to 1, and their logarithm runs from minus infinity to plus infinity. A quantity with that range can safely be modelled as a linear function of the regressors without ever implying an impossible probability.

The logistic regression, or logit model: log odds are linear in the independent variables.

Once the log odds have been estimated, the probability itself is recovered by inverting the transformation.

Recovering the event probability from the fitted log odds.

It is worth seeing why this construction cannot misbehave. Whatever values the coefficients and regressors take, the linear combination inside the exponential is some finite real number. The exponential of any real number is strictly positive, so one plus that exponential is strictly greater than one, and the reciprocal must lie strictly between zero and one. The impossible predictions that defeated the linear probability model are ruled out by the algebra rather than by any restriction the analyst has to impose.

This expression traces an S-shaped, or sigmoidal, curve. It is approximately linear through the middle of its range and flattens out as the probability approaches zero or one, which is exactly the behaviour the linear probability model lacked. The model also assumes a logistic distribution for the error term, whose shape resembles the normal distribution but with fatter tails.

Figure 4: Straight line against S-curve

The straight line runs through the ceiling of 1 and would keep going. The logistic curve approaches the ceiling without crossing it, and approaches zero from above at the other end.

Because the logit model is not linear in the probability, least squares is not available. The coefficients are estimated instead by maximum likelihood estimation. The method chooses the coefficient values that make the pattern of outcomes actually observed in the sample as likely as possible, given the model. Since every outcome is binary, the probability distribution used to build the likelihood function is the binomial distribution. The estimates come out of statistical software; no candidate is expected to derive them.

What a candidate must be able to do is read the output. Three points govern interpretation.

A slope coefficient records how far the log odds of the event move when that independent variable rises by one unit, with every other independent variable held fixed. It is not a change in the probability and it is not a change in the odds.
The intercept is the log odds of the event when every independent variable equals zero.
Testing whether a logit coefficient differs from zero works much as it does in ordinary least squares, using the standard error reported alongside the estimate.

There is no direct counterpart to R-squared, because the model was never fitted by least squares. Software reports a pseudo-R-squared instead. It must be handled carefully: it is usable only for comparing different specifications of the same model on the same data, and it says nothing when compared across models built on different datasets.

Marginal effects are not constant

The most important practical consequence of non-linearity is that the effect of a one-unit change in an independent variable on the probability that Y equals 1 is not a fixed number. In the linear probability model the derivative of the probability with respect to a regressor is a constant, so the marginal effect is a constant. In the logistic model the derivative is the exponential function multiplied by the derivative of its own contents, so its value moves with the position on the curve.

Concretely, the impact of a one-unit change in X₁ depends on the value of the whole linear combination b₀ + b₁X₁ + b₂X₂ + b₃X₃, which means it depends on the levels of all the independent variables, not just the one being changed. When that combination is very small or very large, the fitted probability sits on one of the flat tails of the curve and the marginal effect is small. When the combination places the observation near the inflection point, where the probability is 0.5, the curve is at its steepest and the same one-unit change moves the probability much more.

Figure 5: The same change in X does different work at different points on the curve

The marginal effect of an independent variable in a logistic regression is the slope of this curve, so it depends on where the observation sits, which in turn depends on the values of every independent variable.

How the marginal effect is measured

The standard device is to evaluate the marginal effect for the average observation. Take the mean value of each independent variable in the dataset, compute the fitted probability at those means, then recompute the probability after raising the one variable of interest by one unit while leaving the others at their means. The difference between the two probabilities is the estimated marginal effect. Software supports this directly: in Python the marginal effects routine in the scikit-learn and statsmodels ecosystem serves the purpose, and in R the margins package provides the equivalent.

The same calculation can be run for a particular observation rather than the average one. Substitute the characteristics of that company or investment, then perturb the variable of interest by one unit. The output is a marginal probability specific to that case: how much the probability of the event would change if one of its characteristics changed. Used this way the method is a sensitivity analysis rather than a summary statistic.

Example 4 · Worked

You are asked to examine the propensity of companies to repurchase their own shares. For a sample of 500 companies you record which repurchased shares (Repurchase = 1) and which did not (Repurchase = 0), and you collect company data for the year before the repurchase: the cash-to-total-assets ratio (CASH), the debt-to-equity ratio (DE) and the net profit margin (NPM). The estimated model is

Logistic regression summary

Item	Value	Item	Value
Dependent variable	Repurchase	Observations	500
Model	Logit	Df residuals	496
Method	MLE	Df model	3
Pseudo-R-squared	0.0271	Log-likelihood	−304.20
Converged	TRUE	LL-Null	−312.68
Covariance type	Non-Robust	LLR P-value	0.0007

Estimated logit coefficients

Term	Coefficient	Std. error	z-Stat.	P-value	Lower 95%	Upper 95%
Intercept	−0.4738	0.196	−2.415	0.016	−0.858	−0.089
CASH	−0.9118	1.154	−0.790	0.430	−3.174	1.351
DE	−0.3186	0.133	−2.396	0.017	−0.579	−0.058
NPM	0.9407	0.417	2.255	0.024	0.123	1.758

Statistics of the independent variables

Independent variable	Median	Mean	Standard deviation
CASH	0.0515	0.083	0.0894
DE	0.8585	0.9182	0.7638
NPM	0.0775	−0.0535	1.4592

1. Interpret the intercept of the logit regression.

Solution. The intercept of −0.4738 is the log odds of being a share repurchaser when CASH, DE and NPM all equal zero. Convert it back into a probability in two steps. First take the odds:

e^−0.4738 = 0.6226.

Then convert odds into probability:

P = 0.6226 ÷ (1 + 0.6226) = 0.3837, or 38.37 percent.

This is the part of the propensity to repurchase that the three independent variables do not capture.

2. Estimate the marginal effect of each independent variable on the propensity to repurchase shares.

Solution. Start from the expression for the probability and evaluate it at the mean of each independent variable, using CASH = 0.0830, DE = 0.9182 and NPM = −0.0535:

P = 1 ÷ (1 + exp{−[−0.4738 + (−0.9118)(0.0830) + (0.9407)(−0.0535) + (−0.3186)(0.9182)]}) = 29.06 percent.

So the average company in this sample has a 29.06 percent probability of repurchasing shares. Now raise one variable at a time by 1 percent, that is by 0.01, holding the others at their means.

CASH. Raising CASH from 0.083 to 0.093 gives
P = 1 ÷ (1 + exp{−[−0.4738 + (−0.9118)(0.0930) + (0.9407)(−0.0535) + (−0.3186)(0.9182)]}) = 28.87 percent.
The marginal impact is 28.87% − 29.06% = −0.19 percent. A 1 percent increase in the cash ratio lowers the probability of a buyback by 0.19 percentage points.

NPM. Raising NPM from −0.0535 to −0.0435 gives
P = 1 ÷ (1 + exp{−[−0.4738 + (−0.9118)(0.0830) + (0.9407)(−0.0435) + (−0.3186)(0.9182)]}) = 29.26 percent.
The marginal impact is 29.26% − 29.06% = 0.20 percent, an increase in the probability of a buyback.

DE. Raising DE from 0.9182 to 0.9282 gives
P = 1 ÷ (1 + exp{−[−0.4738 + (−0.9118)(0.0830) + (0.9407)(−0.0535) + (−0.3186)(0.9282)]}) = 29.00 percent.
The marginal impact, rounded to two decimal places, is 29.00% − 29.06% = −0.07 percent, so more leverage reduces the probability of a buyback.

The signs line up with the coefficient signs, as they must: CASH and DE carry negative coefficients and reduce the probability, NPM carries a positive coefficient and raises it.

A note on rounding in the third calculation. The two probabilities displayed to two decimal places, 29.00 percent and 29.06 percent, differ by 0.06 percentage points, while the underlying unrounded difference is −0.0656 percentage points, which is the −0.07 percent quoted above. The CASH and NPM effects are quoted from the difference of the rounded probabilities. Reproduce the source figures as given, and be aware that a marginal effect of this size is sensitive to how many decimals are carried through.

Since a logit model is fitted by maximising a likelihood, the natural measure of fit is the value of that likelihood at the optimum. Software reports the log-likelihood, a chi-square distributed test statistic. The classification performance of the model can be judged by comparing the P-value of that statistic with the chosen significance level.

Two features of the log-likelihood must be kept straight.

It is always negative, so a higher value, meaning one closer to zero, indicates a better fit. A model with a log-likelihood of −450.40 fits better than one with −451.66.
Unlike adjusted R-squared, the number is meaningless on its own. It is useful only for comparing models estimated on the same dependent variable and the same data.

The same caution applies with more force to the pseudo-R-squared. It is constructed from the log-likelihood of the fitted model relative to the log-likelihood of the intercept-only model, so it behaves qualitatively like a proportion of explained variation, but it is not one. Values that would look poor for an ordinary least squares regression are entirely normal for a well specified logit, and comparing the pseudo-R-squared of a model fitted to one dataset with that of a model fitted to another says nothing at all. Its legitimate use is narrow: ranking competing specifications estimated on the same dependent variable and the same sample.

Alongside the log-likelihood of the estimated model, software normally reports the log-likelihood of the intercept-only model, usually labelled LL-Null. That figure is the benchmark against which the explanatory variables are judged.

The likelihood ratio test

The likelihood ratio test, or LR test, formalises the comparison. It is built from the log-likelihoods of a restricted and an unrestricted model.

The likelihood ratio statistic, distributed as chi-square with q degrees of freedom, where q is the number of restrictions.

The LR test is the logistic analogue of the joint F-test used with nested least squares models. Both compare a smaller model against a larger one that contains it; the difference is that the LR test works from log-likelihoods rather than sums of squares. Suppose the unrestricted Model A includes three regressors while the restricted Model B imposes b₂ = b₃ = 0 and keeps only the first. The null hypothesis is then that both restricted coefficients are zero, and the alternative is that at least one of them is not.

Note the direction of the test carefully. The null hypothesis states that the smaller, restricted model is the better model. Rejecting the null therefore rejects the restricted model in favour of the larger unrestricted one. The test performs best in large samples.

Example 5 · Worked

Using the share repurchase logit from Example 4, assess how well the model fits the data by comparing it with the intercept-only model as the restricted alternative.

Log-likelihood values for the two models

Model	Log-likelihood
Restricted: intercept only	−312.68
Unrestricted: intercept, CASH, DE, NPM	−304.20

Carry out the likelihood ratio test using the standard six-step hypothesis test procedure and state the conclusion.

Solution.

Step 1. State the hypotheses. H₀: b₁(CASH) = b₂(DE) = b₃(NPM) = 0. H_a: at least one b_j is not zero.

Step 2. Identify the test statistic. The likelihood ratio, LR = −2 × (log-likelihood of the restricted model − log-likelihood of the unrestricted model), with 3 degrees of freedom, one for each restriction.

Step 3. Specify the level of significance. α = 5 percent.

Step 4. State the decision rule. The critical chi-square value with 3 degrees of freedom at the 5 percent level is 7.8150. Reject the null if the calculated LR exceeds 7.8150.

Step 5. Calculate the statistic. LR = −2 × (−312.68 − (−304.20)) = −2 × (−8.48) = 16.960.

Step 6. Make the decision. 16.960 exceeds 7.8150, so reject the null hypothesis. The unrestricted model fits the data better than the restricted model.

The conclusion is that the three explanatory variables are jointly significant, and the reported LR P-value of 0.0007 confirms it. The individual z-statistics agree in part: DE and NPM are each significant at the 5 percent level, with P-values of 0.017 and 0.024, while CASH is not, at 0.430. Joint significance and individual significance are separate questions, and here the joint test is carried by two of the three variables.

Running the model

Estimation itself is a single instruction in either of the two languages an analyst is likely to use. In Python, the logit function in the statsmodels formula interface takes a formula such as the response regressed on CASH, DE and NPM together with the dataframe, fits by the Newton method and prints a summary. In R, the same job is done by the generalised linear model function with the binomial family and the logit link. The output in both cases contains the coefficient table, the log-likelihood, the null log-likelihood and the pseudo-R-squared discussed above.

Where logistic regression is used

Logistic regression is the workhorse of binary classification, which is why it appears throughout machine learning and as the output layer of neural networks. A representative application in fundamental analysis combines it with natural language processing to classify the sentiment of financial text. The dependent variable is the sentiment class, coded 1 for positive and 0 for negative, standing for good or bad corporate news. The independent variables are tokens, the key words and phrases drawn from annual reports, earnings releases and corporate announcements. The model is trained to sort those tokens into good news and bad news, then tested and deployed as an input to valuation work.

Check yourself

1. A regression uses 4 independent variables and 40 observations. One observation has a leverage of 0.32 and another has a leverage of 0.41. Which, if either, is flagged as potentially influential?

The threshold is 3(k + 1)/n = 3 × (4 + 1)/40 = 15/40 = 0.375. The observation with leverage 0.41 exceeds it and is flagged; the observation with leverage 0.32 does not. Note that being flagged means the observation has the capacity to influence the fit, not that it necessarily does. The next step is to re-estimate the regression without it and compare the coefficients.

2. Why can leverage never identify an outlier, and why can a studentized residual never identify a high-leverage point on its own?

Leverage is computed entirely from the independent variables. It measures the distance of an observation from the mean of the X data and does not use the dependent variable at all, so an observation with a wildly wrong Y value but an ordinary X value carries ordinary leverage. The studentized residual measures how badly the model predicts the Y value of a deleted observation, which is information about the dependent variable. The two measures are complementary, and an observation extreme on both is the most likely to change the fitted regression.

3. A regression with 30 observations and 2 independent variables produces a studentized residual of 2.35 for one observation. At the 5 percent level, how many degrees of freedom apply and what does the result imply?

The degrees of freedom are n − k − 2 = 30 − 2 − 2 = 26, not 27. A two-sided critical value at the 5 percent level with 26 degrees of freedom is approximately 2.056, so the absolute value of 2.35 exceeds it and the observation is flagged as a potentially influential outlier. The absolute value does not exceed 3, so on the separate rule of thumb it would not be labelled an outlier by that criterion alone. The correct response is investigation of the data, not automatic deletion.

4. An analyst wants to distinguish among four credit rating buckets in a regression and includes four dummy variables, one per bucket, alongside an intercept. What happens and why?

The regression fails. With one dummy per category, the four dummies sum to 1 for every observation, which reproduces exactly the constant column used to estimate the intercept. That is a perfect linear relation among the independent variables, which breaches the assumption that no exact linear relation exists between two or more regressors, and the coefficients cannot be identified. The fix is to use three dummies and let the fourth bucket serve as the base category, with each dummy coefficient then read as a difference from that base.

5. In the mutual fund model, the coefficient on GROWTH is 2.262 and the coefficient on AGE_GROWTH is 0.020, with AGE at 0.065. What is the age slope for growth funds, and how does the difference from value funds behave as funds age?

The age slope for growth funds is the base slope plus the interaction coefficient, 0.065% + 0.020% = 0.085% per year, against 0.065% per year for value funds. The gap between the two categories therefore has a fixed part of 2.262% from the intercept dummy and a part that grows with age, so returns of growth funds exceed those of value funds by 2.347% for each year since inception. The intercept dummy sets the level of the difference and the slope dummy sets how it widens.

6. Why is a slope coefficient of 0.75 in a logistic regression not a statement that the probability of the event rises by 75 percentage points?

The dependent variable in a logistic regression is the log odds, not the probability. A coefficient of 0.75 means the log odds rise by 0.75 for a one-unit increase in that variable, holding the others constant. Converting that into a change in probability requires the level of every other independent variable, because the probability curve is S-shaped: the same shift in log odds moves the probability a great deal near the halfway point of the curve and very little out on either tail.

7. Two logistic models are estimated on the same data. Model 1 has a log-likelihood of −451.66 and Model 2 has −450.40. Which fits better, and what would you need to test whether the difference is significant?

Model 2 fits better, because its log-likelihood is higher in the sense of being closer to zero. Log-likelihood is always negative, so the less negative value is the better fit. To test whether the improvement is statistically significant, the two models must be nested, and the likelihood ratio statistic LR = −2 × (restricted log-likelihood − unrestricted log-likelihood) is compared with a chi-square critical value with degrees of freedom equal to the number of restrictions imposed by the smaller model.

8. An influential observation is confirmed to be an accurate data point rather than an input error. What should the analyst consider next?

Accuracy shifts the suspicion from the data to the model. An accurate observation that the regression fits badly suggests either that an important explanatory variable has been omitted, so the model cannot account for what makes that observation distinctive, or that a regression assumption is being violated. The remedies are to identify and include the missing variables and to re-check the assumptions. Winsorizing is available where the aim is only to limit the pull of extreme values, and whether the point should be treated as a nuisance or as an integral feature of the data depends on the purpose of the model.

End of lesson.

Back to 01 Quantitative Methods

QM 4 – Extensions of Multiple Regression

Overview

Influential observations: high-leverage points and outliers

Unusual is not the same as influential

Why a scatterplot is not enough

Leverage: detecting extreme values of the independent variables

Studentized residuals: detecting extreme values of the dependent variable