QTA 15: Machine Learning and Prediction

Regularization stops a fitted model becoming too large or complex. It earns its place when a dataset holds many highly correlated features, because correlated inputs let the procedure buy a fine in-sample fit with estimates that offset one another: one feature draws a large positive coefficient and its near twin a large negative one, and the pair swings whenever the sample shifts.

The remedy is to charge the model for the size of the coefficients it uses. A penalty term is added to the objective function being minimised, called a loss function here, so fit must be weighed against the magnitude of the parameters delivering it. The features have to be normalized or standardized first.

The ridge regression loss function

Take n observations on each of m features plus an output variable y, and fit a linear regression whose estimates are written with hats. Ridge regression minimises the quantity below.

The first sum is the residual sum of squares. The second is the shrinkage term, charging for large slope parameters of either sign. The hyperparameter lambda decides how heavily that charge weighs against fit.

Hyperparameters and model parameters

A parameter that helps determine a model without being part of it is a hyperparameter, so lambda is the hyperparameter while the estimated intercept and slopes are the model parameters. Ridge regression has an analytic solution. Shrunken models are also easier to interpret and less prone to overfitting, since chasing noise now has a price.

The least absolute shrinkage and selection operator, shortened to LASSO, keeps the ridge idea and changes only the shape of the penalty, taking absolute values in place of squares.

Since the penalties are second order and first order, the two are also called L2 and L1 regularization. Ridge regression drives coefficients toward zero without arriving, so every feature stays in the specification. LASSO does arrive, setting less important coefficients to exactly zero and removing those features, which is why it is called a feature selection technique.

Elastic net

A third option puts both penalties in the loss function at once, each with its own hyperparameter, so that some coefficients shrink while others disappear.

Source: chapter example, highly correlated Treasury yield features.

Choosing the hyperparameter

The aim is a lambda yielding a model simple enough to interpret while still forecasting accurately. The training set produces coefficients for a given lambda, the validation set shows how that model generalises to unseen data, and the test set supplies a final accuracy measure.

Plenty of questions in finance admit exactly two answers. Will this individual default on a mortgage? Is this transaction fraudulent? Does this person have a private pension plan? Will this option expire in the money? What such classification problems model is the probability of the positive outcome, given the value one, against the negative outcome, given zero.

An ordinary linear model is the wrong instrument. Nothing in its design confines a fitted value between zero and one, so it will return a probability of 1.4 or of minus 0.2, and truncating those back piles observations up at the extremes.

The logistic function

Logistic regression, also called the logit model, passes the linear part through the cumulative logistic function, which has the sigmoid shape below and is bounded between zero and one by construction.

With m features the linear index for observation j is written below, and feeding it through the logistic function gives the fitted probability that the outcome equals one.

From a probability to a classification

A probability is not yet a decision. Converting one calls for a threshold Z: above it the observation is a predicted one, below it a predicted zero. Where both mistakes cost the same, Z equal to 0.5 is natural. Lending is not such a case, since predicting repayment on a loan that goes bad costs the lender the loan while the opposite error costs only the margin never earned. A threshold as low as 0.05 can be right.

The logistic transformation is nonlinear, which rules out ordinary least squares. Estimation relies on the maximum likelihood method, which searches for the coefficients making the observed training data as probable as the model can. For independent observations the likelihood is a product.

Where the observed outcome is one, the jth term collapses to the fitted probability; where it is zero, it collapses to one minus that probability. Products of many small numbers are awkward numerically, so the log-likelihood is maximised instead.

Reading a fitted logit model

A sample from the LendingClub database of peer-to-peer retail loans shows what a fitted logit looks like. The dependent variable takes the value 1 where the loan was charged off as irrecoverable and 0 where it was paid off, over 500 observations.

Source: chapter estimates. Two asterisks mark the 5% level, three the 1% level.

These coefficients cannot be read as least squares coefficients are, because the nonlinear transformation stands between the linear index and the probability. Signs and significance still speak. Longer loan terms and higher interest rates raise the probability of default significantly, while holding a mortgage lowers it. The other eight features fail to reach significance.

Applying regularization to a logit model

Maximising a likelihood is the same as minimising its negative, so both regularization methods carry across unaltered. Add lambda times the sum of squared coefficients, or of their absolute values, to the quantity being minimised.

How a model is judged depends on what it produces. Where the output is continuous, such as a forecast return or yield, the measure aggregates the gaps between predicted and realised values across the test sample. Writing the true value of observation i as y and its prediction with a hat, the mean squared forecast error over a test sample of size n test is below.

Replacing the squares with absolute values gives the mean absolute forecast error, which weighs a single very large miss less heavily.

The confusion matrix

A binary categorical output needs different machinery, and the standard device is a confusion matrix: a two by two table setting what the model predicted against what happened, once probabilities have been converted into categories using the threshold Z.

Source: the standard layout.

A true positive is a case called positive that turned out positive. A false negative is a case called negative that turned out positive, so the model missed it. A false positive is a false alarm, and a true negative a correct rejection.

Nothing in this structure depends on the number of features, the sample size, or which family of models produced the predictions. All that is needed is a binary outcome, so the same table serves a logistic regression, a decision tree and a neural network alike. The measures extend readily to several classes.

The four counts support several performance metrics. Accuracy is the share of all predictions that were correct, and the error rate is its complement.

Precision narrows the question to the cases the model flagged: of the observations predicted positive, what share really were positive. Recall turns it around: of those that really were positive, what share did the model find.

The threshold trade-off, ROC and AUC

Moving Z buys a gain on one error at the price of a loss on the other, and the trade-off between the true positive rate and the false positive rate parallels the one between type I and type II errors. The receiver operating curve, or ROC, traces that relationship as the threshold runs across its range.

The area under the ROC curve, the area under curve or AUC, condenses the picture into one number. Perfectly accurate predictions give an AUC of 1. An AUC of 0.5 corresponds to the dashed diagonal and means no predictive ability, and a lower value indicates negative predictive value.

A decision tree interrogates the input features one after another, and the name comes from the shape of the picture. At every node sits a question whose answer sends the observation to another node or to a leaf. Trees serve mainly for classification, though they also estimate continuous variables, which is why they are sometimes called classification and regression trees, or CARTs. Their interpretability earns them the label white-box models.

Measuring information gain

The feature placed at each node is whichever cuts uncertainty by the most, and that reduction is the information gain. Entropy, one of the two measures in use, records disorder and lies between 0 and 1 by construction.

The logarithm is taken to base 2 rather than the natural base. Changing the base multiplies every entropy figure by the same constant, so feature rankings are unaffected. The Gini measure is the alternative.

Gini and entropy usually produce very similar trees, so the choice between them rarely matters.

Continuous features and pure sets

A continuous feature such as an interest coverage ratio has no natural split point, so the algorithm searches candidate thresholds and keeps whichever maximises information gain at that node. The best threshold at one node need not be best at another. The ideal question leaves a branch in which every observation belongs to one class, described as a pure set.

Growing a tree by repeatedly picking the feature with the greatest information gain is the Iterative Dichotomizer algorithm. Growth stops when a leaf is a pure set, or when every feature has been used. Perfect classification is often unattainable, so some branches finish pure and others do not.

Overfitting, stopping rules and pruning

Given many features, a tree keeps splitting until it has memorised the training sample. A stopping rule set in advance, called pre-pruning, might cap the number of branches or refuse to split a node reached by fewer than a set number of training observations. Post-pruning builds a large tree first, then strips out the weakest nodes.

Why ensembles of learners help

An ensemble runs a range of models and combines their outputs into a single meta-model. Averaging many predictions improves fit through something like the wisdom of crowds, and the combination resists overfitting better than any one model. An ensemble pairing weak learners with the best available model frequently beats that model alone, and any mixture of types can be combined.

Bootstrap aggregation and pasting

Bootstrap aggregation, shortened to bagging, bootstraps repeatedly from the training sample to grow many trees whose outputs are pooled. Draw a subset of the training set: with 100,000 observations available, a subset of 10,000 is reasonable. Build a tree on it. Repeat many times, sampling with replacement. Then average the forecasts.

Sampling with replacement means some observations are never drawn in a given replication. Those out-of-bag observations took no part in fitting that tree, so they are available for assessing it. Pasting is the same procedure without replacement. With 100,000 items and subsamples of 10,000, pasting yields 10 subsamples.

Random forests and boosting

A random forest is an ensemble of decision trees created by sampling observations or features without replacement. Where features are sampled, the count is usually roughly the square root of the number available. Any single tree may be suboptimal, yet the aggregate is normally better.

Boosting trains each new model on the errors its predecessors made. Gradient boosting builds the next model on the residuals of the current one, so the training labels are replaced by those residuals, which proxy for the gradient. Adaptive boosting, known as AdaBoost, raises the weight on misclassified observations, pushing the next classifier toward the difficult cases. That sequential structure separates boosting from bagging.

K nearest neighbors, abbreviated to KNN, is about as direct as supervised learning gets. Select a value of K and a distance measure, usually the Euclidean or the Manhattan measure, then identify the K training observations lying closest in feature space to the point needing a prediction. For classification, a majority voting system assigns the class most of those neighbors belong to; for a continuous target, their average is the forecast. Nothing is fitted in advance, which is why KNN is a lazy learner.

The value of K is the critical choice, and it is the bias-variance tradeoff in miniature. A large K averages over many neighbors, giving high bias and low variance, while a small K tracks the training data closely and overfits more readily. A common rule sets K at approximately the square root of n, so n equal to 5,000 points gives K = 71.

Support vector machines

Support vector machines, or SVMs, suit classification problems with large numbers of features, and the idea is easiest to see with two. Imagine a bank holding a sample of 20 borrowers classified by whether they defaulted, with data on income and total savings. An SVM constructs the widest path it can, bounded by two parallel lines, that keeps the groups apart. The observations on the edges are the support vectors, and the centre of the path becomes the separation boundary. With more features that boundary is a hyperplane.

Source: chapter data for 12 prior car loan applicants, where 1 records a loan granted.

Perfect separation is available in this small example and not in general. Where the groups overlap, a tradeoff must be specified between the width of the path and the misclassifications it creates. Further extensions allow the path itself to be nonlinear.

Artificial neural networks, or ANNs, take loose inspiration from how the brain computes. The most common variety is the feedforward network trained with backpropagation, also known as a multi-layer perceptron, where backpropagation updates the weights and biases from one iteration to the next.

Consider a network with three features as inputs, a single hidden layer of three nodes, and one output. Values at the hidden nodes come from the input features, and the output comes from the hidden nodes.

The function applied at each node is the activation function, acting on a linear combination of the previous layer. Constants such as the b terms are the biases and the coefficients on the inputs are the weights. Activation functions introduce the nonlinearity: without them the output collapses to a linear regression. The logistic or sigmoid function met earlier is a popular choice.

Counting up, this network carries nine weights from the inputs to the hidden nodes, three biases at those nodes, three weights from the hidden nodes to the output and one bias at the output, giving 16 parameters. Practical networks hold several hidden layers and many more nodes.

Gradient descent and the learning rate

Parameters are chosen against the training set on familiar criteria: minimise mean squared errors where a continuous variable is predicted, or apply a maximum likelihood criterion for classification. No analytical formula gives the best values, so the gradient descent algorithm works numerically. It starts from trial values, finds the direction improving the objective function most, steps that way, then recalculates and steps again.

The length of each step is the learning rate, a hyperparameter rather than a parameter of the model. Too small a rate leaves the algorithm crawling toward the bottom. Too large a rate sends the walk oscillating across the valley without settling.

Stopping before the network overfits

Overfitting is held off by running the calculations for the validation data set alongside the training data set. Early on the objective function improves for both. At some stage further steps keep improving it for the training set while worsening it for the validation set, and that crossing point is where gradient descent should stop. Fitted weights can also be inspected for offsetting pairs, themselves a sign of overfitting.

Comparing two approaches is only fair when they see the same problem, so a neural network was fitted to the same ten features and the same output as the logistic regression above. It is a single hidden layer feedforward network with backpropagation, holding ten units in the hidden layer and a logistic activation function. Its loss function rests on an entropy measure, and the optimisation took 297 iterations to converge.

Source: chapter matrices for the LendingClub sample; a default is the positive outcome.

What the comparison shows

On the training sample the two approaches are indistinguishable on accuracy. The network is the more precise, while the logistic regression recovers more of the actual defaults. On the validation sample the ranking reverses on two measures of the three.

Both models fit the validation data less well than the training data, which points to slight overfitting. Removing the least useful features, or applying a regularization, would be a reasonable response. The matrices also show the classifiers diverging more than the summary measures suggest, since the true positive and true negative rates swap between the two datasets.

There is very little to choose between the two approaches, and the indicators point in contradictory directions. That is the ordinary case, and it is why the evaluation metric must match the decision the model supports. A lender worried about missing defaults reads recall first; one worried about turning away good business reads precision first.