QM 6 – Machine Learning

The volume of data available to investors keeps growing, the variety of that data keeps widening, and the economic value of the insights buried inside it keeps rising. Data science has grown up in response, borrowing from mathematics, computer science and business analytics, and adding something the older disciplines did not emphasise: learning. Learning here ranges from simple functions that map one variable onto another to elaborate networks that absorb information, order it and adapt to it.

For an investment professional the practical question is not whether to become a data scientist. It is whether you can source sensible inputs, read the outputs correctly, and turn those outputs into portfolio decisions. That requires an understanding of which investment problems machine learning can address, a working picture of how the main algorithms operate, and enough vocabulary to hold a serious conversation with a quantitative team.

Where the techniques are already being applied

The techniques now appear at every link of the value chain in asset and wealth management.

• Client-facing services. Chatbots field basic retirement savings questions and improve as they accumulate interactions with investors.
• Alpha generation. A signal for security selection can be produced in three ways: a non-linear forecast of one time series, a forecast assembled out of a set of factors defined in advance, or a design in which the algorithm itself picks which inputs to use, whether from existing sources or newly discovered ones. Work using textual analysis reports that when a company revises its annual and quarterly filings, the 10-K and the 10-Q, against the same document one year earlier, those revisions predict equity returns strongly. The predictive content concentrates in deterioration inside the sections dealing with management discussion and with risk.
• Portfolio construction. Target portfolio weights can be calculated subject to client restrictions and then reweighted dynamically to maximise a Sharpe ratio. A better estimate of the variance–covariance matrix can be obtained through principal components analysis, which shrinks how many variables are required before the variation present in the data is accounted for. Research points to machine learning solutions beating mean–variance optimisation at the construction stage.
• Trading and operations. Order flow management tools built on non-linear trading algorithms cut what it costs to put a portfolio decision into effect. The same forces have automated whole tools, processes and businesses, robo-advising being the obvious case.

The wider claim is more ambitious still. Large datasets and learning models may reshape what the profession believes about asset risk premiums, and may reconfigure the business processes of investment management itself.

How machine learning differs from statistics

Statistical modelling and machine learning both study observations in order to uncover the process that generated them. They part company on assumptions, vocabulary and method. A statistical approach starts from foundational assumptions and an explicit model of structure: the sample is assumed to be drawn from a specified probability distribution, and the analysis proceeds from there. Those restrictive assumptions, imposed before the data are examined, can simply be wrong.

Machine learning aims to pull knowledge out of very large quantities of data while imposing far fewer such restrictions. Its goal is to automate decision making by generalising from known examples until the underlying structure of the data emerges, with as little human help as possible. A compact way to remember the objective is that the algorithm finds a pattern and then applies that pattern.

Two practical consequences follow. First, machine learning copes better than a technique such as linear regression when there are very many variables, which is the problem of high dimensionality. Second, it copes better when the relationship is strongly non-linear. Algorithms of this kind are also unusually good at spotting change, because they can pick up the preconditions of a model breaking down or gauge the probability that a regime is about to switch.

The field divides into three broad classes of technique: supervised learning, unsupervised learning, and deep learning together with reinforcement learning. The rest of this lesson works through each of them, then through the individual algorithms, and finishes with a decision framework for choosing among them.

Supervised learning infers the pattern that links a set of inputs, the X variables, to a desired output, Y. Once inferred, that pattern maps any new input set into a predicted output. The defining requirement is a labeled dataset: one in which observed inputs are matched with the associated output. Running the algorithm over that dataset to extract the pattern is called training the algorithm. After training, the pattern can be applied to inputs that were never part of the training data.

Multiple regression is itself a case of supervised learning. Matched data on the X variables and Y are used to estimate parameters that characterise the relationship, and those parameters then predict Y for a fresh set of X values. The gap between predicted and actual Y measures how well the model performs out of sample, meaning on data it has not seen.

The vocabulary shift

The terminology differs from the regression terminology most candidates already know, and examiners exploit the difference.

The labeled data used for training carry both the target outcomes and the paired feature inputs. The test data supply feature inputs only, and the predicted targets are then compared against the actual targets that were held back.

A concrete case makes the vocabulary stick. Suppose the task is to separate legitimate credit card transactions from fraudulent ones. The target is binary, set to 1 where the transaction is fraudulent and to 0 where it is not. The features are the characteristics of each transaction. The algorithm trains on those elements so that it predicts the likelihood of fraud more accurately on new transactions. Learning from experience means something measurable here: as the underlying database grows and supplies more input, the share of transactions classified correctly should climb. One workable choice of algorithm here is a logistic regression, which returns an estimated probability that a given transaction is fraudulent.

Regression problems and classification problems

Supervised learning splits into two problem types, and the split is decided by the nature of the target variable, not by the technique used.

• Regression applies when the target is continuous. Note the trap in the vocabulary: the task is called regression even when the technique employed is not a regression at all.
• Classification applies when the target is categorical or ordinal, an ordinal target being a ranked category.

Regression tasks predict continuous targets. Ordinary least squares is the familiar case, but non-linear supervised techniques also exist, and they earn their place on large datasets carrying many features, a good number of which may be correlated with one another. Predicting how a stock price will perform from a history of market returns is a regression task, and so is estimating the chance that a bond defaults from a history of corporate ratios.

Classification tasks sort observations into distinct categories. When the target is categorical, the model that relates the outcome to the features is called a classifier, and logistic regression is a classifier you have already met. Many classifiers are binary, as with the credit card fraud problem. Multi-category classification is common too: sorting firms into several credit rating categories is a standard example, and there the outcome variable is ordinal because the categories carry a distinct ranking from low to high creditworthiness. On a scale of measurement, an ordinal variable occupies the ground between a purely categorical one and a fully continuous one.

Unsupervised learning does not use labeled data. There are inputs, the X variables, and no target is supplied at all. Because no labeled training data are provided, the algorithm has to discover structure inside the data themselves. That makes it a natural first pass over an unfamiliar dataset: it can hand a human expert insight into data that are too large or too complicated to visualise directly.

Two problem types suit unsupervised learning particularly well.

• Dimension reduction cuts the number of features while retaining variation across observations, so that the information carried by that variation survives. It may be used to squeeze a wide dataset down to a representation that fits on a computer screen, and it is used heavily in quantitative investment and risk management wherever the task is to identify the most predictive factors behind asset price movements.
• Clustering sorts observations into groups such that members of the same group resemble each other more than they resemble members of other groups. The criteria that define the groups may or may not be specified in advance, the number of groups being one such criterion. Asset managers have used clustering to sort companies into data-driven groupings built from financial statement data or corporate characteristics, in place of conventional groupings by sector or country.

Deep learning and reinforcement learning

Within artificial intelligence more broadly, two further categories are distinguished. Deep learning applies sophisticated algorithms to hard problems: recognising what an image contains, recognising a face, recognising speech, and processing natural language. Deep learning rests on neural networks, also called artificial neural networks, which are highly flexible algorithms that have succeeded across a wide span of supervised and unsupervised tasks, the common thread being very large datasets, relationships that are not linear, and features that interact with one another. Reinforcement learning has a computer learn by interacting with itself, or with data that the same algorithm generated. Combining the two sets of principles has produced efficient algorithms for very hard problems in robotics, health care and finance.

A map of the algorithms

It helps to see the whole territory before walking through it. The table below organises the algorithms by whether they are supervised or unsupervised and by the type of variables involved. Linear and logistic regression are assumed known from earlier quantitative methods work; everything else in the table is covered in this lesson.

Machine learning promises several advantages over a structured statistical approach when the task is to explore and analyse the structure of a very large dataset. The algorithms can uncover complex interactions between feature variables and the target, they process enormous quantities of data quickly, many of them capture non-linear relationships without effort, and some can recognise and predict structural change in the link between features and target. These advantages come mainly from the non-parametric and non-linear form of the models, which allows much more flexibility when a relationship is being inferred.

Flexibility is not free. The models produced can be so complex that the results resist interpretation, they may react to noise or to quirks of the particular dataset, and they may fit the training data too well. Where a model has been fitted to the training sample too closely, prediction on fresh material is typically poor. That failure carries the name overfitting: the fitted algorithm fails to generalize.

To generalize well is to keep explanatory power when predicting out of sample, on material never previously seen. What the overfitted model has done is take the noise, and the purely random fluctuation, present in its training sample and build it into the learned relationship. Neither survives into whatever data arrive next, so generalization suffers and predictive value drains away. Judging an algorithm therefore means looking at how badly it errs on fresh material, never at how neatly it fits the sample on which it was itself trained.

Underfitting, overfitting and robust fitting

Before going further, note how the dataset to which a model is applied is normally split. Three non-overlapping samples are carved out. The training sample fits the model. The validation sample validates and tunes it. The test sample establishes how well prediction holds up on material the model has not met. Of the three, the first two are labelled in-sample and the third out-of-sample.

A tailoring analogy makes the three fitting outcomes memorable. Overfitting is a custom suit cut so precisely that it fits exactly one person. Underfitting is a baggy suit that fits nobody. Robust fitting, the outcome wanted, is a suit that fits everyone of roughly similar dimensions.

• Underfitting means the model has not captured the relationships in the data at all. It shows up as errors on both sides of the boundary in a simple classification picture, and it is associated with high in-sample error.
• Overfitting means training has been pushed to a level of specificity at which quirks and spurious correlations in the sample begin to enter the model as though they were signal. Randomness is read as pattern. Memorisation has replaced learning, which gives the model flawless hindsight and no foresight whatever. Two things drive it: a lot of noise sitting in the data, and a model carrying more complexity than the problem needs.
• Good or robust fitting means the model fits the in-sample data well and also generalizes well out of sample, with both errors inside acceptable bounds.

Complexity has a specific meaning here. It counts the features, the terms and the branches a model carries, and it also asks whether the functional form is linear or not, the non-linear form being the more complex of the two. Overfitting risk climbs as complexity does.

Decomposing out-of-sample error

Calibrating fit means setting the two error rates side by side and watching how each responds to the data and to the algorithm. Write E_in for total error inside the sample: it is what the fitted relationship gets wrong when its predictions are set against the targets actually observed in the training set. Write E_out for total error outside it, measured on either the validation or the test material. Little or no error of the first kind sitting beside large error of the second kind is the signature of generalization failing.

The total out-of-sample error is decomposed into three sources.

1. Bias error, meaning how closely the model manages to fit its training data. Where the assumptions behind an algorithm are mistaken the approximation is poor, and the result is high bias, underfitting, and error inside the sample.
2. Variance error, meaning how far the results shift once new material arrives from the validation and test sets. An unstable model soaks up noise, which shows as high variance, overfitting, and error outside the sample.
3. Base error, which is due to randomness in the data and cannot be removed by any model.

Learning curves and fitting curves

A learning curve plots the accuracy rate, equal to one minus the error rate, in the validation or test samples against the amount of data in the training sample. It is therefore a diagnostic for under- and overfitting expressed through bias and variance.

• Where the model is robust, accuracy outside the sample improves as the training set grows. The two error rates close on one another and settle at the error rate wanted, which is another way of saying they settle at the base error.
• Where bias error dominates and the model is underfitted, the rates do converge, but they converge underneath the accuracy that was wanted. Feeding in further training observations will not lift performance to the level required.
• Where variance error dominates and the model is overfitted, the two rates never come together at all.

Complexity moves the two errors in opposite directions. Add complexity and E_in falls on the training material, taking bias error down with it. Add the same complexity and E_out climbs on the test material, taking variance error up. As a rule a linear form is the one exposed to bias and underfitting, while a form that is not linear is the one exposed to variance and overfitting. Somewhere between the two lies an optimal level of complexity, the level at which the bias and variance curves cross and the pair of error rates is jointly at its lowest. Plotting E_in and E_out against complexity produces the fitting curve, which shows that trade-off at a glance.

Locating that point, the last moment before rising variance drags the total error rate upward again, is what managing overfitting risk actually consists of, and it decides whether generalization succeeds. Practitioners restate the whole problem as a trade-off between cost and complexity, cost here being the gap between the two error rates. Framed that way, the trade-off can be plotted, which makes both failure modes visible and gives a route to an optimised model.

Overfitting impairs generalization, and the potential for it is built into supervised learning because noise is always present. Two methods are commonly used to reduce it.

1. Hold down complexity while the algorithm is being selected and trained, which means putting a number on an overfitting penalty and carrying it in the objective.
2. Sample properly, through cross-validation, which measures error on validation material and so estimates error outside the sample directly rather than by inference.

Behind the first sits Occam razor, the principle that among competing explanations the simplest is usually right. Translated into supervised learning it means capping how many features enter, and imposing a cost on any algorithm that is more complex or more flexible than it needs to be, so that a parameter survives only where it lowers error outside the sample.

The second strategy descends from the principle of avoiding sampling bias, which can enter in many ways. The underlying difficulty is having a dataset large enough that both training and testing can be done on representative samples. A sample that is unrepresentative, or a training sample that has been shrunk too far, can hide the true patterns and so increase bias.

The three-way partition

The technique for reducing sampling bias is careful partitioning of the dataset into three groups.

• Training sample: labeled observations, with Y known, on which the model is fitted.
• Validation sample: the material against which structural choices are made, how much complexity to allow, which of several candidate solutions to prefer, and how to tune whichever one is chosen. Validation is what that tuning amounts to.
• Test sample: material set aside untouched, so that predictive or classifying power can be confirmed on something entirely unseen.

The point of all of it is a tested model that can be turned loose on fresh observations from the same domain. A common experimental design splits 70% for training, 15% for hyperparameter tuning and 15% for testing.

K-fold cross-validation

Holdout samples, meaning data samples not used to train the model, solve one problem and create another: they shrink the training set. Cross-validation techniques address that. Under k-fold cross-validation everything other than the test sample and any fresh material is shuffled at random and cut into k sub-samples of equal size. All but one of them train the model; the remaining one, the kth, validates it. Values of 5 or 10 for k are the usual choices.

Run that k times over. Each observation then appears in training k minus 1 times and in validation exactly once, which is what holds bias and variance down together. Averaging the k validation errors gives mean E_val, and that average serves as a reasonable estimate of E_out.

One limitation matters for finance in particular: k-fold cross-validation cannot be used with time-series data, because only the most recent data can reasonably serve for model validation. Shuffling a return series destroys the ordering that makes the prediction problem meaningful.

In short, a supervised model carries over to fresh material drawn from the same distribution only to the extent that error outside the sample has been kept in check, and the two levers available during construction are cutting complexity, otherwise known as regularization, and cross-validation.

Supervised models are trained on labeled data, and the nature of the target decides the type: regression for a continuous target, classification for a categorical or ordinal one. This section and the three that follow work through the supervised algorithms in the order regression first, then classification, then the ensemble methods that combine them. Assume throughout a set of observations on some target Y together with n features taking real values, written X₁ through X_n, and a task of tying the vector of those features to Y at each observation.

Penalized regression is a computationally efficient technique for prediction problems. Two things recommend it in practice: it shrinks a long feature list into something manageable, and it predicts well on big datasets, most usefully those whose features move together, which is exactly where classical linear regression falls apart.

The motivation is the overfitting problem. In a large dataset there may be very many features that could in principle help explain Y. Fit a model to training data and it may reflect the characteristics of that specific sample so closely that it fails on new data. Features can enter that carry nothing except noise, or randomness peculiar to that one training set, and neither will be there in future. Penalized regression is therefore best described as a technique for avoiding overfitting. Since prediction is judged outside the sample, a parsimonious specification, one in which no variable is along for the ride, tends to do well for the simple reason that it has less room to overfit.

From ordinary least squares to a penalty

Standardize the data first, so that every feature carries a mean of zero and unit variance. That step is what makes the size of one coefficient comparable with the size of another. In ordinary least squares the coefficients are chosen to minimise the sum of the squared residuals, that is, the sum of squared differences between the actual values and the predicted values.

A penalized regression bolts a constraint on to that. Coefficients now minimise the squared residuals together with a penalty that rises as more features are admitted. Admitting a feature therefore has a price, and the improvement in fit has to be worth more than the price. Only features that genuinely help explain Y clear that hurdle.

One popular form of the penalty defines LASSO, whose name expands to least absolute shrinkage and selection operator. It is written as follows, with lambda strictly positive.

Minimising then applies to two things at once: the squared residuals, and the absolute sizes of the estimated coefficients.

Retention follows one test: does the fall in squared residuals exceed the rise in the penalty? Every penalized regression rests on a trade-off of that shape. What is distinctive about LASSO is that it pushes the weakest coefficients all the way to zero, dropping those features out of the specification, so a form of feature selection happens on its own.

Lambda as a hyperparameter

Lambda is a hyperparameter. That word denotes any parameter the researcher fixes in advance, before learning starts, rather than one the algorithm learns. Its job is to balance fit against parsimony. Fixing it in practice means running the model repeatedly at different settings and watching performance on the validation set, and that is precisely why a separate test set cannot be dispensed with: without one, the hyperparameters end up overfitted to the validation material.

Two mechanical consequences follow. Set lambda to zero and the penalty disappears, leaving an ordinary least squares regression exactly. And the penalty operates only while the model is being built on training material. After that it has served its purpose and drops away, with the finished model assessed on the squared residuals it produces against the test dataset.

Regularization in practice

Regularization is the general name for methods that cut statistical variability out of estimation problems carrying many dimensions. Here it works by pulling coefficient estimates toward zero, which keeps models simple and keeps overfitting risk down with them. Now that fast computation is cheap, analysts reach for LASSO and its relatives routinely, to strip away the less pertinent features and leave a parsimonious specification behind.

A documented case is the prediction of default probability within industrial sectors. Scores of candidate features, a great many of them collinear, were cut down to a set of under 10 variables. The reduction matters enormously in that setting because defaults themselves are scarce, with only about 100 observations available.

The same methods carry over to models that are not linear. Asset management has struggled for decades to obtain stable covariance matrixes, and therefore stable weights, when optimising a large portfolio on a mean–variance basis. Returns are strongly multi-collinear, so the estimated matrix reacts sharply to noise and to outliers, and the weights that come out of the optimisation are unstable in consequence. Regularization has been applied to that problem, and once again the parsimony delivered by a penalized method leaves less room for overfitting.

Support vector machine, or SVM, ranks among the most widely used algorithms of all. It is supervised, it is powerful, and it serves three purposes: classification, regression and the detection of outliers. The name sounds forbidding; the idea underneath is simple and is best carried by a picture.

Start with a simple dataset containing two features, an x coordinate and a y coordinate, with every observation labeled as belonging to one of two groups. Suppose those groups fall into two visibly separate regions, which might represent stocks with positive and with negative returns in a given year. An infinite number of straight lines could separate the two regions. Such data are called linearly separable, and every one of those candidate lines qualifies as a linear classifier: a binary classifier whose verdict comes from a linear combination of the features attached to each point.

Two features give a classifier that is a straight line. Attach n features instead and each observation sits in a space of n dimensions, with separability meaning that some linear boundary through that space carves the observations into two regions. Whatever its dimension, that boundary is called an n-dimensional hyperplane. At n equal to 1 the word for it is a line and at n equal to 2 it is a plane.

The maximum margin idea

Among all the hyperplanes that would work, SVM picks the one that splits the observations optimally into two sets. The intuition is that a correct prediction is most likely when the dividing surface sits as far as possible from every observation. So the algorithm separates by the maximum margin, a margin being the strip dividing the two groups, and the line down the centre of that strip is the discriminant boundary, or the boundary for short.

What fixes the width of the strip is the handful of observations nearest the boundary on each side, and those observations are the support vectors. Pile in further training data well away from them and the boundary does not budge. Introduce points near the hyperplane and it may well move, because the set of support vectors can change.

When the data are not separable

Real datasets are usually not separable in this clean way. Points end up on the wrong side of the boundary and are misclassified. The adaptation that copes with this is soft margin classification, under which every misclassified training point attracts a penalty inside the objective function. The boundary chosen is then the one balancing a wider strip against a smaller total penalty.

A second route exists. Rather than softening the margin, allow the separating surface itself to bend, giving a non-linear SVM. Fewer training points end up misclassified, but the algorithm is more complex and, for that reason, readier to overfit.

Investment applications

The natural home for SVM is a dataset of modest size but awkward shape, complex and carrying many dimensions, of which corporate financial statements and bankruptcy databases are the standard cases. Predicting corporate failure matters to investors deciding what to avoid or what to sell short, and a long list of fundamental and technical variables can be reduced to a binary verdict, bankruptcy likely or bankruptcy unlikely. Its virtue on such material is that many features can be carried without the result becoming hostage to outliers or to features that move together. The same algorithm sorts documents, news articles, company announcements and annual reports, into categories that matter to investors, sentiment being the obvious one.

K-nearest neighbor, or KNN, is a supervised technique whose usual job is classification, though it occasionally handles regression. Its logic is to place a new observation by measuring how near it stands to the observations already held.

Return to the scatter of two labeled groups used for SVM and drop in one new point that has to be assigned. With k set to 1, the new point simply inherits the label of whichever single point lies closest. With k set to 5, the five closest points are consulted instead. Suppose those five split three to two. The rule is majority among the k consulted, so the new point takes the label held by the three. Notice that k equal to 1 and k equal to 5 can return the same verdict even though the neighbors consulted were entirely different.

A corporate bond illustration

Take a database of rated corporate bonds that also records detailed characteristics for each one. Some of those characteristics belong to the issuer, its asset size, its industry, its leverage and cash flow ratios. Others belong to the issue, its tenor, whether the coupon floats or is fixed, and any options embedded in it. Now an unrated bond comes to market. Since issuers and issues that look alike ought to attract much the same rating, KNN can infer the implied rating for the new one by measuring how close it stands to what is already in the database.

Strengths, and the problem of defining similarity

Simplicity is the appeal, and it is not a weakness. The method is non-parametric, imposing nothing whatever about how the data are distributed, and it extends to problems with more than two classes without any modification.

What is hard is pinning down the word near. Feature choice matters, but the deeper decision is which distance metric will stand in for similarity, since a metric that suits the problem badly yields a model that performs badly. The judgement becomes still more subjective once the data are ordinal or categorical. Someone comparing how various equities have performed in the market might, for example, treat the correlation between their return histories as the similarity measure.

Defining similarity therefore leans on knowledge of the data and on a clear view of what the analysis is for commercially. Irrelevant or correlated features distort the answer, so features may have to be selected by hand. Stripping out the low-value material leaves what is pertinent, and done well it produces a distance measure that represents the problem more faithfully. Broadly, the fewer features carried, the better this algorithm performs.

Choosing k

The number k is the hyperparameter here, and it has to be chosen knowing that different values can point to different conclusions. Should the unrated bond be compared against the 3 closest bonds, the 15 closest, or the 50 closest? An even value invites ties, which leave no clear verdict at all. Choose too small a value and the error rate rises while local outliers start to dominate. Choose too large a value and the idea of a nearest neighbor is diluted, since far too many outcomes are being averaged. A number of techniques exist for settling on an optimal value, and they take account of how many categories there are and how those categories carve up the feature space.

In investment work the algorithm turns up in predicting bankruptcy, predicting stock prices, assigning credit ratings to corporate bonds, and building customised equity and bond indexes. Knowing which bonds resemble one another and which do not is precisely the input a custom, diversified bond index requires.

Classification and regression tree, or CART, is supervised and handles both kinds of target. Aim it at a categorical target and it returns a classification tree; aim it at a continuous one and it returns a regression tree. Binary classification and regression are its usual employments.

Work through a simplified model that sorts companies according to how likely they are to raise the dividend. What is needed is a binary tree, built out of three ingredients: one initial root node, some decision nodes, and the terminal nodes at which paths end. Every node other than a terminal one names one feature, call it f, together with the cutoff value c at which that feature is split.

Let the feature at the root be investment opportunities growth, written IOG and designated X₁, cut at 10%. Any new observation enters there. Below the root the observations are carved at each decision node into progressively smaller subgroups, and the process ends at terminal nodes, which are the only nodes carrying a predicted label. Here the two labels available are a dividend increase and no dividend increase.

Trace a single route. IOG above 10% answers Yes, sending the observation to a node splitting on free cash flow growth, written FCFG and designated X₂, cut at 20%. FCFG at or under that cutoff answers No, and the prediction returned is no dividend increase, at a terminal node. FCFG above the cutoff answers Yes, and the prediction returned is a dividend increase, at a different terminal node.

One structural feature deserves emphasis: a single feature may show up more than once in the same tree, paired with different features, and certain features earn relevance only after other conditions are satisfied. Go back to the root and take the other branch. Where IOG sits at or below 10% while FCFG exceeds 10%, IOG returns as a decision node further down, this time cut at 5% rather than 10%.

How the algorithm picks features and cutoffs

Training comes first, on labeled material. In the hypothetical case that material is 20 companies, half of which raised the dividend and half of which did not. The feature space here is simply the plane traced out by X₁ and X₂, and every split, at the root and at each decision node alike, cuts that plane into two rectangles, one holding values above the relevant cutoff and one holding values at or below it.

How does the algorithm decide what to split on? At every node it searches for the feature, and the cutoff on that feature, that pull the labeled data furthest apart, the objective being minimum classification error under some criterion, mean-squared error being one. Each split leaves smaller partitions, and within each of those the error is lower than it was before. Splitting stops at whatever level a further bifurcation would barely reduce classification error. The node is then terminal, and it takes the label of whichever category is in the majority inside it.

So a classification tree predicts, at each terminal node, whichever category holds most of the observations sitting in that partition. Suppose the rectangle at the top right, where IOG exceeds 10% and FCFG exceeds 20%, holds five members of the dividend increase group, more observations than any other partition. A company with those characteristics would be predicted to raise the dividend. Change the second condition so FCFG falls at or below 20% and the company lands in the rectangle at the lower right, which holds three members of the no increase group against two of the increase group, so the prediction flips. Where the objective is regression rather than classification, the terminal node returns the mean of the labeled values instead of a category.

Regularization and pruning

Nothing whatever is assumed about the shape of the training data, which means an unconstrained tree can go on splitting until it reproduces that data exactly. That is overfitting at its purest. Constraints are therefore imposed as regularization parameters, and three are common: a ceiling on how deep the tree may go, a floor on how many observations a node must contain, and a ceiling on how many decision nodes may exist. Growth halts as soon as whichever criterion applies is hit. Take the rectangle at the upper left, reached by IOG at or below 10%, then FCFG above 10%, then IOG at or below 5%, and holding three members of the increase group. A minimum population set at 3 would make that a terminal node.

The other route is to regularize after the fact by pruning: growing the tree first and then cutting back whichever sections contribute little to classifying anything.

Why the iterative structure matters

Because the structure is built one split at a time, dependencies between features emerge that other specifications simply cannot express. Take a stylized tree separating an attractive equity holding from a value trap, the trap being a holding that looks cheap and is nonetheless likely to lose money. Profitability turns out to be the decisive feature, but it only becomes decisive once the stock is already cheap. The hypothetical route is a price to earnings ratio under 15, then leverage that is high with debt above 50% of total capital, then sales expanding at more than 15%. Reverse that statement and the point is sharper: where the stock is not cheap, or leverage is not high, or sales are not expanding, profitability tells you nothing at all in this context. A multiple linear regression is at its weakest on relationships shaped like that one.

Part of the appeal of the method is that the tree explains itself: the prediction is a path a human can read. Set that against the algorithms treated as black boxes, whose reasoning cannot easily be followed and therefore cannot easily be trusted. Trees make a strong foundation for expert systems used in decision making, and they extract dependable rules even where the data are noisy and the interactions among many features are involved. Investment uses include sharpening the search for fraud in financial statements, imposing a consistent process on equity and fixed-income selection, and making an investment strategy easier to explain to a client.

Every algorithm so far has rested a prediction on one model. There is an alternative: poll a whole group of them. Each individual model errs at some rate and predicts noisily. Average enough predictions drawn from enough models and the noise ought to wash out, leaving an average that sits closer to the truth. That pooling of output across a collection of models is ensemble learning, and the practice of combining several learning algorithms goes by the name ensemble method. Accuracy and stability both improve relative to the best individual model, which is why these methods so often win the well-known competitions in the field.

Ensemble learning splits into two categories:

1. Aggregation of heterogeneous learners, meaning different types of algorithm combined through a voting classifier.
2. Aggregation of homogeneous learners, meaning the same algorithm applied to different training data, generated for example by bootstrap aggregating.

Voting classifiers

Picture a machine learning project that has run for a while, in which several algorithms have been trained and their results compared, say a support vector machine, a k-nearest neighbor routine and a tree. A majority-vote classifier hands each new observation whichever predicted label collected the most votes. If two of the three call a stock an outperformer and the third calls it an underperformer, outperformer is the answer returned.

Aggregate accuracy improves as more models join the panel, though not without limit. Past some optimal count, adding models is expected to make matters worse through overfitting. What makes the panel work is diversity: different algorithms, different modelling techniques, different hypotheses. The assumption underneath, in its extreme form, is that individual predictions are independent of one another, in which case the law of large numbers does the rest.

Bootstrap aggregating

Bootstrap aggregating, shortened to bagging, holds the algorithm fixed and varies the data instead. Out of the one training set it manufactures n further training sets, called bags, each drawn at random with replacement from the original. Training the same algorithm on those n independent sets yields n models. Each new observation is then run through all of them, and the n answers are pooled, by majority vote where the task is classification and by averaging where it is regression. What bagging buys is steadier predictions and less exposure to overfitting.

Random forest

A random forest classifier is nothing more than a great many decision trees trained through bagging. Run a tree algorithm across each of the n bags and the result is the crowd of differing trees that constitutes the forest.

Diversity can be pushed further still by withholding features at random during training. Where an observation carries n features, a subset of m of them, with m below n, is drawn at random and only those are available to the tree when it looks for a split at a given decision node. That subset size, the count of trees, the floor on how many observations a node or leaf may hold, and the ceiling on depth are all hyperparameters, and all can be tuned to lift accuracy. Once trained, every tree in the forest votes on each new observation and the majority carries it, which is the algorithmic counterpart of the wisdom of crowds.

Building a model this way lowers variance and shields the result from overfitting the training material. Signal also improves relative to noise, since the errors made by any one tree tend to be cancelled by the others. The cost is interpretability. A single tree can be read; a forest of them cannot, which places the method firmly among the black boxes.

A published illustration plots defaults among small and medium-sized businesses against two characteristics, profitability and leverage, and sets the observed pattern beside what two models predict. The linear specification cannot reproduce the shape of the relationship at all, because that shape is not linear. The forest, fed identical data, tracks the observed distribution closely.

A fraud detection application

The forest is the clearest illustration of what ensembles achieve: pooling output across many models yields classifications whose signal stands out from the noise better than any single classifier manages. Take a fraud detection exercise built on an openly available card transaction dataset, whose features were anonymized but might still explain which transactions were fraudulent. The awkwardness is rarity. Out of 284,807 transactions only 492 were fraudulent, which is 0.17%, so this is a needle in a haystack.

Pairing a forest with oversampling, which deliberately inflates how large a share of the training set the fraudulent cases occupy, copes with that imbalance remarkably well. Precision reaches 89%: of all transactions flagged as fraud, that share genuinely was fraud. Recall reaches 82%: of all the fraud actually present, that share was caught.

For all its relative simplicity the method is powerful and turns up widely. Factor-based strategies use it in asset allocation and in choosing individual investments. It has also been used to predict whether an initial public offering will go well, judged by how far the book was oversubscribed or by where the first trading day closed relative to the offer price, given what is known about the offering and about the issuer.

The unsupervised branch works without labels and therefore without any target, which leaves the algorithm to find whatever pattern lives inside the inputs. Two families of technique matter for the examination: shrinking the number of dimensions, done through principal components analysis, and grouping observations, done through k-means or through hierarchical methods. This section covers the first family and the next section covers the second.

Why shrink anything? Because a wide dataset is hard to picture and hard to model. Plots become impossible to read and fitted models start reflecting random influences peculiar to the one sample in hand, which is what the word noisy means in this context. Shrinking the dimensions means describing the same dataset with fewer features, where the features discarded were largely duplicating what the survivors already say.

Eigenvectors, eigenvalues and composite variables

Principal components analysis, or PCA, has been in the statistical toolkit for a very long time. It takes features that move closely together and condenses them into a handful of composite variables that do not move together at all. A composite variable is simply one built out of two or more variables whose statistical relationship with each other is strong. Loosely described, the procedure transforms the covariance matrix of the features, and it turns on two objects.

• An eigenvector defines one of the new composite variables. Each is a linear combination of the original features, each is uncorrelated with the others, and being a vector, each also points in a direction.
• An eigenvalue comes attached to every eigenvector, and it reports what share of the total variance present at the outset that eigenvector accounts for.

Ranking the eigenvectors by their eigenvalues, from largest downward, ranks them by how much of the original variance each one explains. Whichever eigenvector explains the most becomes the first principal component. Whatever variance the first one leaves behind is then attacked by the second, which explains the largest share of the remainder, and so on down through the third, the fourth and the rest. Since each one blends the whole feature set, a short list of them usually suffices to account for most of the variance sitting in the original covariance matrix.

Projection error and spread

Picture three features, so the data can be drawn in three dimensions with each observation carrying a reading on each axis. Standardize first, so every series has mean zero and standard deviation one. Now suppose the procedure has been run and the leading two components are known.

Drop a perpendicular from any observation onto the first component. The length of that perpendicular is the projection error attaching to that observation. Measure instead along the direction of the component itself, and the distances between observations give the spread of the data in that direction. The line selected as the first component is the one that makes the total of all the projection errors as small as it can be while making the spread as large as it can be. Those two criteria together are what guarantee that no other single direction captures more of the variance. The remainder is then best handled by the second component, which is set at right angles to the first and is therefore uncorrelated with it. Two numbers per observation now stand in for three, and that substitution is the dimension reduction.

Deciding how many components to keep

Keeping few components makes a complicated dataset manageable; keeping few also throws information away. Something has to arbitrate. That job falls to the scree plot, which charts what share of variance each component accounts for. The practical rule is to retain the smallest number of components that between them reach whatever share of the original variance the analyst has decided is enough, and in practice that target is commonly set somewhere between 85% and 95%.

An illustration uses two hypothetical equity indexes across the last 10-year period. One, a Diversified Large Cap 500, stands for a broad index of large companies spanning every economic sector. The other, a Very Large Cap 30, stands for a narrow index holding only the 30 biggest quoted companies. The raw material is index prices plus more than 2,000 features of a fundamental and technical kind. With that many candidates, overlap between features and combinations of features is inevitable, and multi-collinearity is the standard consequence. The procedure is applied to gather the information and the variance up into a much smaller set.

Twenty components were produced in all, and the researchers settle on three as sufficient for each index. Their charts flatten out after roughly the fifth component, meaning what remains adds very little to the account of how the variance is structured, so those later components can be dropped at negligible cost.

The contrast between the two rows repays attention. Concentrate an index into 30 very large names and a single dominant factor explains 55% of the variance on its own. Broaden it and that leading factor accounts for only 43%, so more components are needed before a comparable share is reached. That difference is diversification expressed in statistical language.

The investment committee of an endowment fund wants three names to buy for its large-cap equity portfolio. Eight constituents of a broad United States equity index are handed to an analyst, whose brief is to establish how closely the returns on those eight move together, which is to say how strongly they correlate. Since the object is diversification, the committee wants the three chosen names to correlate as little as possible. So the analyst groups the eight by resemblance and then takes one name out of each group.

The steps in the analysis

1. Assemble the panel of adjusted closing prices for the eight names.
2. Convert prices into daily log returns, so that each name is represented by a vector of 500 numbers.
3. Run the bottom-up algorithm. It begins by computing a Euclidean distance for every pair of return vectors, and those pairwise figures populate a distance matrix, otherwise called a dissimilarity matrix, whose diagonal is zero throughout. Every stock starts alone. The two nearest clusters are located and fused into one. Distances from that newly created cluster out to everything still unmerged are then recomputed, here through average linkage, which measures from the centre of the new cluster to the centre of each remaining one. Other linkage rules exist, and whichever is chosen the sequence is identical: fuse the closest pair, treat the result as one cluster, recompute the distances.
4. Keep repeating the recomputation until nothing is left unmerged.
5. Draw the dendrogram, then find the highest horizontal line that cuts exactly three dendrites, three being the number of recommendations wanted, and read off the grouping at that level.

Note that the returns, not the prices, are what get clustered. Every calculation below comes straight out of the matrix.

What the clustering reveals

Broadly the groups line up with sectors, but two anomalies surface. FB does not behave like the other two technology names at all once returns are compared for co-movement. And those other two behave more like the banks than like the pair of car makers, which end up isolated together.

The top-down algorithm begins from the opposite end, with all eight in one cluster, and splits recursively until each name stands alone. Working out the very first split means evaluating every way the eight could be divided, which is far too numerically intensive to set out in detail. The table below places the three-group outcome of all three algorithms side by side. Cluster numbers are arbitrary from one column to the next; what matters is which names sit together.

Reading the columns as groupings: bottom-up gives a five-name cluster of banks and two technology stocks, an auto pair, and FB alone. K-means gives a bank trio, a technology trio including FB, and the auto pair. Top-down gives a five-name cluster of the three banks with the two auto names, a technology pair, and FB alone.

One point commands unanimity: the three banking names belong together under every method. Two of the three also isolate the car makers as a group of their own. Where they part company is FB. K-means sorts strictly along sector lines and files it with the other technology names; both hierarchical methods treat it as an outlier and leave it by itself. As a rule the two hierarchical routes agree with each other more often than either agrees with k-means, though even a shared linkage rule does not guarantee identical output. K-means opens with three groups already in existence and shuffles points between them, a mechanism unlike anything the hierarchical methods do, so its answers are not expected to coincide with theirs.

Given the instruction that the three purchases should correlate as little as possible, the recommendation is to buy FB, to buy whichever of the two car makers looks better value, and to buy whichever of the three banks looks better value.

Removing the auto stocks

Now suppose the same exercise is repeated with the two car makers withdrawn, leaving six names. The panel of prices and daily log returns covers those six only, and the matrix is the old one with two rows, two columns and every distance involving them deleted.

Rerun on six names, the algorithm produces three groups: the bank trio, the pair of AAPL and GOOG, and FB alone. GS and JPM still fuse first and UBS still joins them, exactly as before, and the technology pair still forms. What no longer happens is the merger of that pair into the bank cluster, because three distinct groups already exist and FB stays out on its own, its return co-movement being so unlike either of the others. The revised recommendation is to buy FB, to buy whichever of the technology pair is cheaper, and to buy whichever of the three banks looks most attractive.

Everything covered so far collapses into a short series of questions. Work through them in sequence and the algorithm selects itself. One branch holds the supervised methods and the other holds the unsupervised ones, with each question acting as a junction.

1. Are the data complicated, carrying many features that move closely together? Where they are, principal components analysis should be run first to shrink the dimensions.
2. Is the problem one of classification or of numerical prediction?
3. If numerical prediction, do the data have non-linear characteristics? Penalized regression and LASSO suit linear data; CART, random forest or neural networks suit non-linear data.
4. If classification, are the data labeled? Labeled data lead to the classification algorithms; unlabeled data lead to the clustering algorithms.
5. If the data are labeled, do they have non-linear characteristics? K-nearest neighbor and support vector machine suit linear data; CART, random forest or neural networks, including deep neural networks, suit non-linear data.
6. If the data are unlabeled, do they have non-linear characteristics? Neural networks, including deep neural networks, suit non-linear data. For linear data, k-means applies when the number of categories is known and hierarchical clustering when it is not.

Note that the first question stands apart from the others. Dimension reduction is not an alternative to prediction or classification; it is a preparatory step that often precedes them.

Where this fits in a wider data project

Choosing an algorithm is one decision inside a longer sequence. Before any algorithm runs, the steps of preparing and wrangling the raw data have to be completed, because real data arrive incomplete, inconsistent and in formats no model can read. Data exploration follows, both to understand the distributions involved and to guide the work of engineering and selecting the features that will actually be fed to the model. Where the raw material is textual, exploring text-based sources such as filings, transcripts and news requires its own methods for extracting and constructing features before any financial forecasting can begin. The algorithms in this lesson are the engine; the surrounding project work determines whether the engine has anything worth running on.

Two habits of mind survive the whole subject. The first is that evaluation only counts when it happens outside the sample, since the point of any model is to say something about material it has never met. The second is that opacity has a price, and the price is commercial rather than aesthetic. A tree an investment committee can follow may be worth more than a forest that predicts a shade better, and choosing between them is a judgement about the business, not about statistics.